MapReduce filtering question

Sean Cribbs sean at basho.com
Sat Nov 20 08:38:34 EST 2010


On Nov 19, 2010, at 11:54 PM, Parker Thompson wrote:

> Thanks, a few questions inline...
> 

Responses inline.

> On Fri, Nov 19, 2010 at 2:43 PM, Sean Cribbs <sean at basho.com> wrote:
> 
> class Riak::Alternative
>  include Ripple::Document
>  many :visitors, :class_name => "Riak::Visitor"
>  property :alternative_id, Integer, :presence => true
>  key_on :alternative_id
> end
> 
> If I expect to be writing large numbers of visitor->alternatives links is it performant to be writing them all as links on one object, as opposed to creating many experience docs each with a link ?  Naïvely I would assume this might less evenly distribute write load or degrade as the size of the Link data grows.  Does this matter?

You either take the hit at write time or at query time. Personally, I think this is easier to understand and manage, but it may also depend on what other queries you want to run.  For example, if your other queries are more focused on the individual visitor, it might make sense to have a list of alternative_ids in the Visitor class, instead of having something (experience or alternative) link to it.  The key takeaway I'm trying to lead you to is that data types that are effectively "joins" are less efficient/useful in Riak.

>  
> ########
> 
> def visitors_who_shared
>  Riak::MapReduce.new(Ripple.client).
>    add("riak_alternatives", ar_id.to_s).
>    link(:bucket => 'riak_visitors').
>    map(link_to_events_forward_visitor).
>    map(map_share_events_to_visitor).
>    reduce(["riak_kv_mapreduce", "reduce_set_union"]).
>    map(map_identity, :keep => true).
>    run
> end
> 
> Ah, I was looking for a set_union.  Is there a full list of these functions hiding somewhere?
>  

Our community manager has some better documentation in the oven, but there are a number of them in these two files:

(JS) https://github.com/basho/riak_kv/blob/master/priv/mapred_builtins.js
(Erlang) https://github.com/basho/riak_kv/blob/d4c6930d97fb109f9dcb3c73a99747ca217275c7/src/riak_kv_mapreduce.erl

Cheers,

Sean Cribbs <sean at basho.com>
Developer Advocate
Basho Technologies, Inc.
http://basho.com/


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20101120/6bd41341/attachment.html>


More information about the riak-users mailing list