indices with large number of objects on same key

Telmo Menezes telmo at telmomenezes.com
Tue Mar 6 05:09:02 EST 2012


>> So I have the situation where Riak objects contain a list of keys
>> referencing other objects. There are cases where objects contain a
>> very high number of such references. At some point, it becomes
>> unreasonable to just store this entire list in the object.
>>
>> Is there some way of dealing with this problem in Riak that I'm missing?
>>
>> Secondary indicies seem like a possible solution: I could just tag the
>> referenced objets with the id of the referrer. The problem is that the
>> documentation is very unclear about what happens when a large number
>> of values have the same tag. If a query for this tag, is there a
>> reasonable way to get a high number of results? Any sort of pagination
>> or streaming?
>>
>
> Riak doesn't support pagination, but it does support streaming. You can perform streaming MR operations to consume the results of the last phase of the MapReduce job. I have no idea which clients support this functionality (outside of the C# client). Although I would suspect most would support it.
>
> From [1]:
>
> Q: Although streams weren't mentioned, do you have any recommendations on when to use streaming map/reduce versus normal map/reduce?
>
> Streaming MapReduce sends results back as they get produced from the last phase, in a multipart/mixedformat. To invoke this, add ?chunked=true to the URL when you submit the job. Streaming might be appropriate when you expect the result set to be very large and have constructed your application such that incomplete results are useful to it. For example, in an AJAX web application, it might make sense to send some results to the browser before the entire query is complete.
>
> [1]: http://basho.com/blog/technical/2010/07/27/webinar-recap---mapreduce-querying-in-riak/

Great, this should do it.
I'm using the Java driver with Scala and it does seem to support streaming.

Thanks!
Telmo.




More information about the riak-users mailing list