Map reduce timeout

Dan Reverri dan at basho.com
Tue Oct 5 13:11:54 EDT 2010


Hi Germain,

There is an issue in 0.12 that causes whole bucket map/reduce jobs to ignore
user supplied timeouts. This happens if the results are not streamed. You
can activate streaming by adding "chunked=true" to the end of your request:
curl -v -XPOST http://localhost:8098/mapred?chunked=true -d
"{\"timeout\":600000,\"query\":[{\"map\":{\"source\":\"function(v) { var
data = Riak.mapValuesJson(v)[0] ; if ( data.faved_what != undefined &&
data.faved_what.total != undefined ) { return  [ data.faved_what.total ] ; }
return [];
}\",\"language\":\"javascript\",\"arg\":[],\"keep\":false}},{\"reduce\":{\"source\":\"function(values,
arg) { return [ values.reduce( function (item, total) { return item +
total*1 ; }, 0 ) ]
}\",\"language\":\"javascript\",\"arg\":[],\"keep\":true}}],\"inputs\":\"people\"}"

This issue has been fixed in 0.13:
https://issues.basho.com/show_bug.cgi?id=523

Thanks,
Dan

Daniel Reverri
Developer Advocate
Basho Technologies, Inc.
dan at basho.com


On Tue, Oct 5, 2010 at 9:05 AM, Germain Maurice <
germain.maurice at linkfluence.net> wrote:

>  Hello everybody,
>
> I'm trying to execute some mapreduce jobs and i have a problem with
> timeout.
> I'm working with riak_kv_version : <<"0.12.1">> with a single node
> "cluster".
>
> My inputs are "people" bucket which contains more than 200k documents. So i
> have decided to set mapreduce timeout to 600000 milliseconds.
> Obviously, it doesn't take care of timeout.
> You'll find below the JSON object i sent to Riak/mapred and the Internal
> Server Error less than 60 seconds after. I added the error message i got
> in riak console.
>
> =====
> > date; perl riak.pl ; date
> mardi 5 octobre 2010, 17:47:58 (UTC+0200)
> --- "{\"timeout\":600000,\"query\":[{\"map\":{\"source\":\"function(v) {
> var data = Riak.mapValuesJson(v)[0] ; if ( data.faved_what != undefined &&
> data.faved_what.total != undefined ) { return  [ data.faved_what.total ] ; }
> return [];
> }\",\"language\":\"javascript\",\"arg\":[],\"keep\":false}},{\"reduce\":{\"source\":\"function(values,
> arg) { return [ values.reduce( function (item, total) { return item +
> total*1 ; }, 0 ) ]
> }\",\"language\":\"javascript\",\"arg\":[],\"keep\":true}}],\"inputs\":\"people\"}"
> --- 500 Internal Server Error
>
> mardi 5 octobre 2010, 17:48:53 (UTC+0200)
> =====
>
> ********
> =ERROR REPORT==== 5-Oct-2010::17:48:53 ===
> Failed reduce: {timeout,
>                   {gen_server,call,
>                       [<0.143.0>,
>                        {dispatch,
>                            {<0.143.0>,#Ref<0.0.5.43514>},
>                            {{jsanon,
> <<"function(values, arg) { return [ values.reduce( function (item, total) {
> return item + total*1 ; }, 0 ) ] }">>},
>                             [<<"2607">>,<<"261">>,<<"1753">>,<<"81">>,
> <<"18190">>,<<"581">>,<<"811">>,<<"732">>,
> <<"1831">>,<<"967">>,<<"2650">>,<<"222">>,
> <<"674">>,<<"2136">>,<<"1467">>,<<"362">>,
> <<"19965">>,<<"2025">>,<<"348">>,<<"4817">>,
> <<"286">>],
>                             []}},
>                        10000]}}
>
> =ERROR REPORT==== 5-Oct-2010::17:48:53 ===
> ** State machine <0.24966.6> terminating
> ** Last event in was {inputs,[<<"286">>]}
> ** When State == executing
> **      Data  == {state,1,riak_kv_reduce_phase,
>                     {state,
>                         {javascript,
>                             {reduce,
>                                 {jsanon,
> <<"function(values, arg) { return [ values.reduce( function (item, total) {
> return item + total*1 ; }, 0 ) ] }">>},
>                                 [],true}},
>                         [],[]},
>                     true,true,undefined,[],undefined,1,<0.24964.6>,660000}
> ** Reason for termination =
> ** {error,failed_reduce}
> ********
>
> Any idea about this ??
>
> Thank you,
> I'll go back to work on my slides for saturday :)
>
>
> --
> Germain Maurice
> Administrateur Système/Réseau
> Tel : +33.(0)1.42.43.54.33
>
> http://www.linkfluence.net
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20101005/1dc3ab26/attachment.html>


More information about the riak-users mailing list