map-reduce Problem ?

Greg Steffensen greg.steffensen at gmail.com
Mon Nov 15 11:57:08 EST 2010


Along these lines, are there any ideas floating around about how to speed up
the listing of keys in a bucket?  For the bitcask backend, it seems like an
index of keys-by-bucket ought to be the kind of thing that could be stored
in the hints files to speed this up without affecting performance for live
reads and writes.

Greg

On Mon, Nov 15, 2010 at 11:46 AM, Sean Cribbs <sean at basho.com> wrote:

> This is possible with Riak's MapReduce but you will likely have increasing
> difficulty as your dataset grows, because of the impact of needing to list
> keys in a bucket and then eliminate data points you aren't interested in.
>  In the longer term, there will be improvements to MapReduce such that if
> your keys are meaningful, you will be able to filter them more easily
> (without examining the data first).  You might find Kevin Smith's overview
> enlightening: http://www.slideshare.net/hemulen/riak-mapred-preso
>
> Sean Cribbs <sean at basho.com>
> Developer Advocate
> Basho Technologies, Inc.
> http://basho.com/
>
> On Nov 15, 2010, at 11:34 AM, Prometheus WillSurvive wrote:
>
> Hi ,
>
> We have a huge database (around 4 billion record - 30 TB) storing the video
> watch infromation ie view count , comment , favorited etc. I want to produce
> daily report for all videos view counts. It means I need to look 2 day ,
> today and yesterday so subtract yesterdey view count from today view count
> so I can find the daliy impression. Our Fat DB team doing this a few complex
> queries. I would like to ask you is this possible with Riak map-reduce way .
>  I want to make a demonstration to the team to show this ..
>
> This is the scenario. We have similar data models for other thins. This
> could be a start.
>
> We have 30xHP DL380  x32 Gig Ram  Farm  to test this scenario.
>
> Any riak map-reduce experienced member can show some idea on this..  I
> guess.
>
> Regards
>
> Prometheus
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20101115/c997bfb9/attachment.html>


More information about the riak-users mailing list