Problems with weird errors running 0.13

Jon Brisbin jon.brisbin at npcinternational.com
Mon Nov 29 13:53:54 EST 2010


On Nov 29, 2010, at 12:15 PM, Dan Reverri wrote:

> Note the following from the article:
> "The important thing to understand is that the function defining the reduce phase may be evaluated multiple times, and the input of later evaluations will include the input of earlier evaluations."
> 
> Based on your defined reduce function it looks like you are trying to count the number of objects in the "test" bucket. Rather than return the length of the incoming list it would be better to sum the values of the incoming list using "reduceSum":
> https://github.com/basho/riak_kv/blob/master/priv/mapred_builtins.js#L66

I logged the incoming data and since I'm getting a mixture of objects and sums (so far) in the reduce input, I did this to account for both kinds of data coming into the function in a single parameter:

function(v){
  var count=0;
  for(i in v){ 
    if(typeof v[i] === 'object'){ 
      count += 1; 
    } else { 
      count += v[i]; 
    }
  } 
  ejsLog('/tmp/mapred.log','r: '+JSON.stringify(v)); 
  return [count]; 
}

I'm not sure I understand how I can separate the objects from the reduce return values using reduceSum. 

jb



> 
> Thanks,
> Dan
> 
> Daniel Reverri
> Developer Advocate
> Basho Technologies, Inc.
> dan at basho.com
> 
> 
> On Mon, Nov 29, 2010 at 7:08 AM, Jon Brisbin <jon.brisbin at npcinternational.com> wrote:
> 
> On Nov 29, 2010, at 8:59 AM, David Smith wrote:
> 
> > On Mon, Nov 29, 2010 at 7:43 AM, Jon Brisbin
> > <jon.brisbin at npcinternational.com> wrote:
> >> I'm working on the Spring Data and Grails Gorm support for Riak and I'm seeing some problems running my tests against Riak 0.13. I don't see these problems when running 0.12.
> >
> > Can you elaborate on the problems you're seeing, please?
> 
> There's two problems I'm seeing, one is with map/reduce, the other is that the server will crash under the load the tests put on it (I'll send a separate email with steps to reproduce).
> 
> Here's an example of a problem I'm seeing in M/R (from an earlier message):
> 
> I'm seeing some weird behavior with map/reduce in 0.13 versus 0.12 and I'm not sure what's going on here. I'm hanging out in the chat room right now trying to find out, but basically, if I run this test case against 0.13 I get a completely different result than what I get in 0.12:
> 
> http://pastie.org/1320874
> 
> In 0.12, I get [1] in 0.13 I get [2].
> 
> The output I'm logging looks like this for 0.12:
> 
> 11/23/2010 (13:39:47): map={"bucket":"test","key":"test","vclock":"a85hYGBgzGDKBVIsTE+lb2UwJTLmsTLUTEk4xpcFAA==","values":[{"metadata":{"Links":[],"X-Riak-VTag":"5qaBBTXfdEQUyCBTDRRIQ7","content-type":"application/json","X-Riak-Last-Modified":"Tue, 23 Nov 2010 19:39:40 GMT","X-Riak-Meta":[]},"data":"{\"test\":\"value\",\"integer\":12}\n"}]}
> 11/23/2010 (13:39:47): reduce=[1]
> 
> And this in 0.13:
> 
> 11/23/2010 (13:46:52): map={"bucket":"test","key":"test","vclock":"a85hYGDgzmDKBVIsjDI+7RlMiYx5rAz9ExKO8UGE2ZqTWGZL10AlRCfBJVgYLplNxhQGqmeMbn4EldBEUs/k7LodUxionils9l2oRBGy+cHJ7zGFgeqZk3faQCVOT0aoZzG1PohFmNWr8CxUWGUaijFv+lYiS2QBAA==","values":[{"metadata":{"Links":[],"X-Riak-VTag":"If4qIgutNTerbEmxiKLhR","content-type":"application/json","X-Riak-Last-Modified":"Tue, 23 Nov 2010 19:46:44 GMT","X-Riak-Meta":[]},"data":"{\"test\":\"value\",\"integer\":12}\n"}]}
> 11/23/2010 (13:46:52): reduce=[]
> 11/23/2010 (13:46:52): reduce=[1,0]
> 
> I tried Homebrew (OS X 10.6) and the OS X download packages of both versions.
> 
> Jon Brisbin
> Portal Webmaster
> NPC International, Inc.
> 
> 
> 
> 
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 


Jon Brisbin
Portal Webmaster
NPC International, Inc.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20101129/9d7a3d9b/attachment.html>


More information about the riak-users mailing list