Schema Architecture, Map Reduce & Key Lists
jeremiah.peschka at gmail.com
Thu Feb 10 12:52:46 EST 2011
Riak 0.14 brings key filters - it's still going to take time to filter the keys in memory, but it's an in memory operation. Using 'smart keys' along the lines of UNIXTIMESTAMP:placement:campaign:customer you can rapidly filter your keys using meaningful criteria and perform MapReduce jobs on the results.
Nothing says you can't also store the same data in multiple buckets in multiple formats to make querying easier.
In response to number 2 - there's a way to set Riak to auto expire data from a bucket. It'll only be removed when compactions occur, but if you're storing clickstream data that should be happen often enough.
Microsoft SQL Server MVP
MCITP: Database Developer, DBA
On Thursday, February 10, 2011 at 9:35 AM, Mat Ellis wrote:
> We are converting a mysql based schema to Riak using Ripple. We're tracking a lot of clicks, and each click belongs to a cascade of other objects:
> click -> placement -> campaign -> customer
> i.e. we do a lot of operations on these clicks grouped by placement or sets of placements.
> Reading this http://lists.basho.com/pipermail/riak-users_lists.basho.com/2010-July/001591.html gave me pause for thought. I was hoping the time needed to crunch each day's data would be proportional to the volume of clicks on that day but it seems that it would be proportional to the total number of clicks ever.
> What's the best approach here? I can see a number of 'solutions' each of them complicated:
> (1) Maintain an index of clicks by day so that we can focus our operations on a time bound set of clicks
> (2) Delete or archive clicks once they have been processed or after a certain number of days
> (3) Add many links to each placement, one per click (millions potentially)
> On a related noob-note, what would be the best way of creating a set of the clicks for a given placement? Map Reduce or Riak Search or some other method?
> Thanks in advance.
> riak-users mailing list
> riak-users at lists.basho.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the riak-users