map-reduce Problem ?

Sean Cribbs sean at basho.com
Mon Nov 15 11:46:43 EST 2010


This is possible with Riak's MapReduce but you will likely have increasing difficulty as your dataset grows, because of the impact of needing to list keys in a bucket and then eliminate data points you aren't interested in.  In the longer term, there will be improvements to MapReduce such that if your keys are meaningful, you will be able to filter them more easily (without examining the data first).  You might find Kevin Smith's overview enlightening: http://www.slideshare.net/hemulen/riak-mapred-preso

Sean Cribbs <sean at basho.com>
Developer Advocate
Basho Technologies, Inc.
http://basho.com/

On Nov 15, 2010, at 11:34 AM, Prometheus WillSurvive wrote:

> Hi ,
> 
> We have a huge database (around 4 billion record - 30 TB) storing the video watch infromation ie view count , comment , favorited etc. I want to produce daily report for all videos view counts. It means I need to look 2 day , today and yesterday so subtract yesterdey view count from today view count so I can find the daliy impression. Our Fat DB team doing this a few complex queries. I would like to ask you is this possible with Riak map-reduce way .  I want to make a demonstration to the team to show this ..
> 
> This is the scenario. We have similar data models for other thins. This could be a start. 
> 
> We have 30xHP DL380  x32 Gig Ram  Farm  to test this scenario. 
> 
> Any riak map-reduce experienced member can show some idea on this..  I guess.
> 
> Regards
> 
> Prometheus
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20101115/c255d3c2/attachment.html>


More information about the riak-users mailing list