Map reduce timeout

Germain Maurice germain.maurice at linkfluence.net
Tue Oct 5 15:20:14 EDT 2010


  Thank you Dan. :)
If i have searched in the previous emails on the mailing list, i would 
find the solution. :/


Le 05/10/10 19:11, Dan Reverri a écrit :
> Hi Germain,
>
> There is an issue in 0.12 that causes whole bucket map/reduce jobs to 
> ignore user supplied timeouts. This happens if the results are not 
> streamed. You can activate streaming by adding "chunked=true" to the 
> end of your request:
> curl -v -XPOST http://localhost:8098/mapred?chunked=true -d 
> "{\"timeout\":600000,\"query\":[{\"map\":{\"source\":\"function(v) { 
> var data = Riak.mapValuesJson(v)[0] ; if ( data.faved_what != 
> undefined && data.faved_what.total != undefined ) { return  [ 
> data.faved_what.total ] ; } return []; 
> }\",\"language\":\"javascript\",\"arg\":[],\"keep\":false}},{\"reduce\":{\"source\":\"function(values, 
> arg) { return [ values.reduce( function (item, total) { return item + 
> total*1 ; }, 0 ) ] 
> }\",\"language\":\"javascript\",\"arg\":[],\"keep\":true}}],\"inputs\":\"people\"}"
>
> This issue has been fixed in 0.13:
> https://issues.basho.com/show_bug.cgi?id=523
>
> Thanks,
> Dan
>
> Daniel Reverri
> Developer Advocate
> Basho Technologies, Inc.
> dan at basho.com <mailto:dan at basho.com>
>
>
> On Tue, Oct 5, 2010 at 9:05 AM, Germain Maurice 
> <germain.maurice at linkfluence.net 
> <mailto:germain.maurice at linkfluence.net>> wrote:
>
>      Hello everybody,
>
>     I'm trying to execute some mapreduce jobs and i have a problem
>     with timeout.
>     I'm working with riak_kv_version : <<"0.12.1">> with a single node
>     "cluster".
>
>     My inputs are "people" bucket which contains more than 200k
>     documents. So i have decided to set mapreduce timeout to 600000
>     milliseconds.
>     Obviously, it doesn't take care of timeout.
>     You'll find below the JSON object i sent to Riak/mapred and the
>     Internal Server Error less than 60 seconds after. I added the
>     error message i got
>     in riak console.
>
>     =====
>     > date; perl riak.pl <http://riak.pl> ; date
>     mardi 5 octobre 2010, 17:47:58 (UTC+0200)
>     ---
>     "{\"timeout\":600000,\"query\":[{\"map\":{\"source\":\"function(v)
>     { var data = Riak.mapValuesJson(v)[0] ; if ( data.faved_what !=
>     undefined && data.faved_what.total != undefined ) { return  [
>     data.faved_what.total ] ; } return [];
>     }\",\"language\":\"javascript\",\"arg\":[],\"keep\":false}},{\"reduce\":{\"source\":\"function(values,
>     arg) { return [ values.reduce( function (item, total) { return
>     item + total*1 ; }, 0 ) ]
>     }\",\"language\":\"javascript\",\"arg\":[],\"keep\":true}}],\"inputs\":\"people\"}"
>     --- 500 Internal Server Error
>
>     mardi 5 octobre 2010, 17:48:53 (UTC+0200)
>     =====
>
>     ********
>     =ERROR REPORT==== 5-Oct-2010::17:48:53 ===
>     Failed reduce: {timeout,
>                       {gen_server,call,
>                           [<0.143.0>,
>                            {dispatch,
>                                {<0.143.0>,#Ref<0.0.5.43514>},
>                                {{jsanon,
>     <<"function(values, arg) { return [ values.reduce( function (item,
>     total) { return item + total*1 ; }, 0 ) ] }">>},
>                                 [<<"2607">>,<<"261">>,<<"1753">>,<<"81">>,
>     <<"18190">>,<<"581">>,<<"811">>,<<"732">>,
>     <<"1831">>,<<"967">>,<<"2650">>,<<"222">>,
>     <<"674">>,<<"2136">>,<<"1467">>,<<"362">>,
>     <<"19965">>,<<"2025">>,<<"348">>,<<"4817">>,
>     <<"286">>],
>                                 []}},
>                            10000]}}
>
>     =ERROR REPORT==== 5-Oct-2010::17:48:53 ===
>     ** State machine <0.24966.6> terminating
>     ** Last event in was {inputs,[<<"286">>]}
>     ** When State == executing
>     **      Data  == {state,1,riak_kv_reduce_phase,
>                         {state,
>                             {javascript,
>                                 {reduce,
>                                     {jsanon,
>     <<"function(values, arg) { return [ values.reduce( function (item,
>     total) { return item + total*1 ; }, 0 ) ] }">>},
>                                     [],true}},
>                             [],[]},
>                        
>     true,true,undefined,[],undefined,1,<0.24964.6>,660000}
>     ** Reason for termination =
>     ** {error,failed_reduce}
>     ********
>
>     Any idea about this ??
>
>     Thank you,
>     I'll go back to work on my slides for saturday :)
>
>
>     -- 
>     Germain Maurice
>     Administrateur Système/Réseau
>     Tel : +33.(0)1.42.43.54.33
>
>     http://www.linkfluence.net
>
>
>     _______________________________________________
>     riak-users mailing list
>     riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>     http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20101005/f7160437/attachment.html>


More information about the riak-users mailing list