Schema Architecture, Map Reduce & Key Lists
mat at tecnh.com
Thu Feb 10 13:20:11 EST 2011
That's interesting. Unfortunately I think a smart key would simply shift the problem elsewhere as we also generate a UUID so clicks can be converted when a conversion event occurs. So we either have a UUID as the key (and can quickly look up a click to see if it's valid and then the associated data if it is) or we can have a Smart Key and can easily lookup the associated data. Either way the posting I linked to suggests that unless I supply long lists of keys I have to scan *all* keys in *all* buckets, no matter how much I segment my data into separate (i.e. smaller) "sub-" buckets.
The auto-expire feature will definitely be handy. We'll look at that.
Does anyone have:
(a) opinions on addressing this issue using Ripple's "many" and "one" associations? It seems I could end up with a placement record with millions of clicks attached
(b) any good examples of Ripple's associations being used in real code (as opposed to examples here http://seancribbs.github.com/ripple/Ripple/Associations.html) and/or a discussion about Ripple::Document vs Ripple::EmbeddedDocument?
On Feb 10, 2011, at 9:52 AM, Jeremiah Peschka wrote:
> Riak 0.14 brings key filters - it's still going to take time to filter the keys in memory, but it's an in memory operation. Using 'smart keys' along the lines of UNIXTIMESTAMP:placement:campaign:customer you can rapidly filter your keys using meaningful criteria and perform MapReduce jobs on the results.
> Nothing says you can't also store the same data in multiple buckets in multiple formats to make querying easier.
> In response to number 2 - there's a way to set Riak to auto expire data from a bucket. It'll only be removed when compactions occur, but if you're storing clickstream data that should be happen often enough.
> Jeremiah Peschka
> Microsoft SQL Server MVP
> MCITP: Database Developer, DBA
> On Thursday, February 10, 2011 at 9:35 AM, Mat Ellis wrote:
>> We are converting a mysql based schema to Riak using Ripple. We're tracking a lot of clicks, and each click belongs to a cascade of other objects:
>> click -> placement -> campaign -> customer
>> i.e. we do a lot of operations on these clicks grouped by placement or sets of placements.
>> Reading this http://lists.basho.com/pipermail/riak-users_lists.basho.com/2010-July/001591.html gave me pause for thought. I was hoping the time needed to crunch each day's data would be proportional to the volume of clicks on that day but it seems that it would be proportional to the total number of clicks ever.
>> What's the best approach here? I can see a number of 'solutions' each of them complicated:
>> (1) Maintain an index of clicks by day so that we can focus our operations on a time bound set of clicks
>> (2) Delete or archive clicks once they have been processed or after a certain number of days
>> (3) Add many links to each placement, one per click (millions potentially)
>> On a related noob-note, what would be the best way of creating a set of the clicks for a given placement? Map Reduce or Riak Search or some other method?
>> Thanks in advance.
>> riak-users mailing list
>> riak-users at lists.basho.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the riak-users