Structure of Erlang MR Arg

Bryan Fink bryan at basho.com
Tue Feb 26 09:16:22 EST 2013


Doh … sorry this thread ended up lost in the stack. Response below…

On Feb 14, 2013, at 10:58 AM, Jeremiah Peschka <jeremiah.peschka at gmail.com> wrote:
> In the MapReduce Implementation docs [1], it looks like the Arg parameter of an Erlang MR phase is not a single argument but is, in fact, a proplist of Erlang terms, am I correct in assuming that this is the case?
> 
> And, if this is the case, would the JSON submitted to Riak look something like this:
> 
> {"reduce":{"language":"erlang",
>            "module":"riak_kv_mapreduce",
>            "function":"reduce_count_inputs", 
>            "arg":{"do_prereduce":true,
>                   "reduce_phase_batch_size": 250
>                  }
>            }
> }

Your read of the doc is mostly correct. The arg is whatever you want it to be. However, the MR interface has recently begun to inspect the arg itself to find tuning parameters. What it's looking for is exactly what you suggested: a JSON object with those fields in it. The idea is that the object could also contain other fields, allowing other data to be passed to the function, as usual. Some of the built-in functions in riak_kv_mapreduce.erl need to be updated to support an arg in this format; sorry for the mixed message.

HTH,
Bryan





More information about the riak-users mailing list