Unexpected issue w/ m/r Riak.reduceSlice

Jonathan Langevin jlangevin at loomlearning.com
Wed Nov 30 10:47:02 EST 2011


Based on a quite helpful tip from Sean via another thread, here's my
working solution :-)
It was indeed an issue with how batches were being processed, so running
the reduce phase as run-once, fixes the issue.

{
   "inputs":"demo_3_Course",
   "query":[
      {
         "map":{
            "keep":true,
            "language":"javascript",
            "arg":null,
            "source":"function(value){return [value.key];}"
         }
      },
      {
         "reduce":{
            "keep":true,
            "language":"javascript",
            "arg":{"sort":"function(a,b){
                    if (a < b) {
                        return -1;
                    } else if (a === b) {
                        return 0;
                    } else if (a > b) {
                        return 1;
                    }
                }","slice":[80,85],"reduce_phase_only_1":true},
            "source":"function(values,arg){return
Riak.reduceSlice(Riak.reduceSort(values,arg[\"sort\"]),arg[\"slice\"]);}"
         }
      }
   ]
}
*


 <http://www.loomlearning.com/>
 Jonathan Langevin
Manager, Information Technology
Loom Inc.
Wilmington, NC: (910) 241-0433 - jlangevin at loomlearning.com -
www.loomlearning.com - Skype: intel352
*


On Tue, Nov 29, 2011 at 6:07 PM, Jonathan Langevin <
jlangevin at loomlearning.com> wrote:

> I suppose I should instead stream the list of keys to client, slice keys
> in client, then fetch the objects, right?
> On Nov 29, 2011 5:21 PM, "Jonathan Langevin" <jlangevin at loomlearning.com>
> wrote:
>
>> When attempting to run m/r queries that execute Riak.reduceSlice to
>> create paginated result sets, I've found an unexpected result.
>>
>> For instance, if I call Riak.reduceSlice with start = 80, end = 85, which
>> you would expect to return 5 results (knowing that you have a total of 115
>> objects stored in Riak), you might instead get 16 objects returned back.
>> This is due to the logic of Riak.reduceSlice, which just returns the
>> results-thus-far when it finds that the end value (85) is greater than the
>> current length of value (16 objects).
>>
>> I suppose ultimately this is due to how Riak sends data in chunks between
>> phases (at least, that's my understanding)? So it happens to send a small
>> chunk of 16 results from the map phase to the reduce phase, and reduce
>> returns back the current "value" chunk since 16 results is less than 85
>> results...
>>
>>
>> How can I plan for this, so that I can ensure I'm getting the expected #
>> results back, and in fair context to the "page" of results that I should be
>> viewing?
>> And just a note, while I tested with & without sorted results, the final
>> code would be with sorted results that are then paginated.
>>
>>
>> Example data payload:
>> {
>>    "inputs":"demo_3_Course",
>>    "query":[
>>       {
>>          "map":{
>>             "keep":false,
>>             "language":"javascript",
>>             "arg":null,
>>             "source":"function(value){return [value.key];}"
>>          }
>>       },
>>       {
>>          "reduce":{
>>             "keep":true,
>>             "language":"javascript",
>>             "arg":[
>>                80,
>>                85
>>             ],
>>             "name":"Riak.reduceSlice"
>>          }
>>       }
>>    ]
>> }
>> *
>>
>>  <http://www.loomlearning.com/>
>>  Jonathan Langevin
>> Manager, Information Technology
>> Loom Inc.
>> Wilmington, NC: (910) 241-0433 - jlangevin at loomlearning.com -
>> www.loomlearning.com - Skype: intel352
>> *
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20111130/843a0807/attachment.html>


More information about the riak-users mailing list