Problems with weird errors running 0.13

Dan Reverri dan at basho.com
Mon Nov 29 14:09:12 EST 2010


Hi Jon,

If the map function is only returning '1' for each object there is no need
to distinguish between map inputs and reduce inputs.

For example, imagine we have 5 objects in our "test" bucket and the initial
batch of results to reduceSum only includes 3 of the objects:
[1,1,1] -> reduceSum -> [3]

The next time reduceSum is run it is given the remaining 2 map results as
well as the previously calculated reduceSum result:
[1,1,3] -> reduceSum -> [5]

Thanks,
Dan

Daniel Reverri
Developer Advocate
Basho Technologies, Inc.
dan at basho.com


On Mon, Nov 29, 2010 at 10:53 AM, Jon Brisbin <
jon.brisbin at npcinternational.com> wrote:

>
> On Nov 29, 2010, at 12:15 PM, Dan Reverri wrote:
>
> Note the following from the article:
> "The important thing to understand is that the function defining the reduce
> phase may be evaluated multiple times, and the input of later evaluations
> will include the input of earlier evaluations."
>
> Based on your defined reduce function it looks like you are trying to count
> the number of objects in the "test" bucket. Rather than return the length of
> the incoming list it would be better to sum the values of the incoming list
> using "reduceSum":
> https://github.com/basho/riak_kv/blob/master/priv/mapred_builtins.js#L66
>
>
> I logged the incoming data and since I'm getting a mixture of objects and
> sums (so far) in the reduce input, I did this to account for both kinds of
> data coming into the function in a single parameter:
>
> function(v){
>   var count=0;
>   for(i in v){
>     if(typeof v[i] === 'object'){
>       count += 1;
>     } else {
>       count += v[i];
>     }
>   }
>   ejsLog('/tmp/mapred.log','r: '+JSON.stringify(v));
>   return [count];
> }
>
> I'm not sure I understand how I can separate the objects from the reduce
> return values using reduceSum.
>
> jb
>
>
>
>
> Thanks,
> Dan
>
> Daniel Reverri
> Developer Advocate
> Basho Technologies, Inc.
> dan at basho.com
>
>
> On Mon, Nov 29, 2010 at 7:08 AM, Jon Brisbin <
> jon.brisbin at npcinternational.com> wrote:
>
>>
>> On Nov 29, 2010, at 8:59 AM, David Smith wrote:
>>
>> > On Mon, Nov 29, 2010 at 7:43 AM, Jon Brisbin
>> > <jon.brisbin at npcinternational.com> wrote:
>> >> I'm working on the Spring Data and Grails Gorm support for Riak and I'm
>> seeing some problems running my tests against Riak 0.13. I don't see these
>> problems when running 0.12.
>> >
>> > Can you elaborate on the problems you're seeing, please?
>>
>> There's two problems I'm seeing, one is with map/reduce, the other is that
>> the server will crash under the load the tests put on it (I'll send a
>> separate email with steps to reproduce).
>>
>> Here's an example of a problem I'm seeing in M/R (from an earlier
>> message):
>>
>> I'm seeing some weird behavior with map/reduce in 0.13 versus 0.12 and I'm
>> not sure what's going on here. I'm hanging out in the chat room right now
>> trying to find out, but basically, if I run this test case against 0.13 I
>> get a completely different result than what I get in 0.12:
>>
>> http://pastie.org/1320874
>>
>> In 0.12, I get [1] in 0.13 I get [2].
>>
>> The output I'm logging looks like this for 0.12:
>>
>> 11/23/2010 (13:39:47):
>> map={"bucket":"test","key":"test","vclock":"a85hYGBgzGDKBVIsTE+lb2UwJTLmsTLUTEk4xpcFAA==","values":[{"metadata":{"Links":[],"X-Riak-VTag":"5qaBBTXfdEQUyCBTDRRIQ7","content-type":"application/json","X-Riak-Last-Modified":"Tue,
>> 23 Nov 2010 19:39:40
>> GMT","X-Riak-Meta":[]},"data":"{\"test\":\"value\",\"integer\":12}\n"}]}
>> 11/23/2010 (13:39:47): reduce=[1]
>>
>> And this in 0.13:
>>
>> 11/23/2010 (13:46:52):
>> map={"bucket":"test","key":"test","vclock":"a85hYGDgzmDKBVIsjDI+7RlMiYx5rAz9ExKO8UGE2ZqTWGZL10AlRCfBJVgYLplNxhQGqmeMbn4EldBEUs/k7LodUxionils9l2oRBGy+cHJ7zGFgeqZk3faQCVOT0aoZzG1PohFmNWr8CxUWGUaijFv+lYiS2QBAA==","values":[{"metadata":{"Links":[],"X-Riak-VTag":"If4qIgutNTerbEmxiKLhR","content-type":"application/json","X-Riak-Last-Modified":"Tue,
>> 23 Nov 2010 19:46:44
>> GMT","X-Riak-Meta":[]},"data":"{\"test\":\"value\",\"integer\":12}\n"}]}
>> 11/23/2010 (13:46:52): reduce=[]
>> 11/23/2010 (13:46:52): reduce=[1,0]
>>
>> I tried Homebrew (OS X 10.6) and the OS X download packages of both
>> versions.
>>
>> Jon Brisbin
>> Portal Webmaster
>> NPC International, Inc.
>>
>>
>>
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
>
>
> Jon Brisbin
> Portal Webmaster
> NPC International, Inc.
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20101129/8f6d7626/attachment.html>


More information about the riak-users mailing list