riak2 erlang mapreduce counters

Russell Brown russell.brown at me.com
Fri Jan 24 02:59:18 EST 2014


Hi Bryce,

Sorry about this, and thanks for the detailed info. I do need to add MapReduce friendly functions.

On 23 Jan 2014, at 23:51, Bryce Verdier <bryceverdier at gmail.com> wrote:

> Thank you both Eric & Russeli for the answer, sadly it leads to more questions. Regardless of the type (though I can say in this case the counters were pushed from the python 2.0.2 client, so I assume its riak_dt_pncounter)

It is a riak_kv_pncounter, I don’t think any of the clients support the new API end points (bucket types) so you’ll be using the 1.4 counter.

> 
> I get this error:
> {"phase":0,"error":"badarg","input":"{ok,{r_object,<<\"ogir-fp\">>,<<\"682l2fp6\">>,[{r_content,{dict,4,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[[<<\"content-type\">>,97,112,112,108,105,99,97,116,105,111,110,47,114,105,97,107,95,99,111,117,110,116,101,114],[<<\"X-Riak-VTag\">>,52,103,66,88,71,122,56,55,111,105,66,112,103,65,75,99,54,72,55,69,110,79]],[[<<\"index\">>]],[],[[<<\"X-Riak-Last-Modified\">>|{1390,508268,984013}]],[],[]}}},<<69,1,71,1,0,0,0,29,70,1,131,108,0,0,0,1,104,2,109,...>>}],...},...}","type":"error","stack":"[{erlang,apply,[<<\"riak_kv_pncounter\">>,new,[]],[]},{riak_kv_crdt,crdt_value,2,[{file,\"src/riak_kv_crdt.erl\"},{line,94}]},{riak_kv_crdt,value,2,[{file,\"src/riak_kv_crdt.erl\"},{line,86}]},{mr_kv_counters,value,3,[{file,\"mr_kv_counters.erl\"},{line,38}]},{riak_kv_mrc_map,map,3,[{file,\"src/riak_kv_mrc_map.erl\"},{line,165}]},{riak_kv_mrc_map,process,3,[{file,\"src/riak_kv_mrc_map.erl\"},{line,141}]},{riak_pipe_vnode_worker,process_input,3,[{file,\"src/riak_pipe_vnode_worker.erl\"},{line,445}]},{riak_pipe_vnode_worker,...}]”}

That binary (<<69,1,71…>>) says CRDT(69), Version 1(1), riak_kv_pncounter (71). The error says that erlang is trying to apply the Module, Function, Arguments of <<“riak_kv_pncounter”>>, new, []. But modules need to be atoms.

> Just to help, this is my erlang MR code:
> value(RiakObject, _KeyData, _Arg) ->
>    Key   = riak_object:key(RiakObject),
>    Count = riak_kv_crdt:value(RiakObject, <<"riak_kv_pncounter">>),

Type needs to be an atom, as it is the module name (so riak_kv_pncounter). Also, the return value is now a tuple of

    {{Context :: binary(), Value :: term()}, Stats :: proplist()}. You probably only want the Value bit. So:

    {{_Ctx, Count}, _Stats} = riak_kv_crdt:value(RiakObject, riak_kv_pncounter),

Should be what you need, let me know if that works, please?

Cheers

Russell

>    [ {Key, Count} ].
> 
> What am I doing wrong? I can't seem to figure it out... I'm sure its something simple thing I'm just not seeing.
> 
> Thanks again,
> Bryce
> 
> On 01/23/2014 01:07 PM, Russell Brown wrote:
>> On 23 Jan 2014, at 20:51, Eric Redmond <eredmond at basho.com> wrote:
>> 
>>> For version 1.4 counters, riak_kv_pncounter. For 2.0 CRDT counters, riak_dt_pncounter.
>> As in, if the data was written in 1.4, or in 2.0 using the legacy, backwards compatible 1.4 API endpoints, the the type is risk_kv_pncounter. If the counter is 2.0, bucket types counter, then risk_dt_pncounter.
>> 
>> Really, we need to re-introduce the riak_kv_counter module for backwards compatibility, and add some friendly `value’ functions to risk_kv_crdt. I’m opening an issue for just this now.
>> 
>> The other option is to include the riak_kv_types.hrl and use the macros ?MAP_TYPE, ?SET_TYPE, ?V1_COUNTER_TYPE, ?COUNTER_TYPE for now, and assume that we’ll have some helper functions for MapReduce in before 2.0.
>> 
>> Cheers
>> 
>> Russell
>> 
>>> Eric
>>> 
>>> On Jan 23, 2014, at 3:44 PM, Bryce Verdier <bryceverdier at gmail.com> wrote:
>>> 
>>>> In 1.4 there was just the simple function riak_kv_counters:value. In 2.0 I found the riak_kv_crdt module, which has a value function in it. But I'm not sure what "type" to use for second value argument for a counter.
>>>> 
>>>> Can someone share that with me?
>>>> 
>>>> Thanks in advance,
>>>> Bryce
>>>> 
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> riak-users at lists.basho.com
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>> 
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> 
> 
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





More information about the riak-users mailing list