Riak shutdown during heavy load testing

Wilson Tuladhar mailwilson007 at gmail.com
Mon May 2 03:44:57 EDT 2011


Hi Dan,
And thanks for the reply. But if we want to get all the keys from the bucket
then is there any other way to do that?? I am using Erlang client.

//Wilson

On Mon, May 2, 2011 at 6:42 AM, Dan Reverri <dan at basho.com> wrote:

> The error message you posted indicates the list keys operation is timing
> out. The list keys operation is an expensive operation and should only be
> used during development or for diagnostic purposes. It should not be used
> during a load test.
>
> Thanks,
> Dan
>
> Daniel Reverri
> Developer Advocate
> Basho Technologies, Inc.
> dan at basho.com
>
>
>
> On Fri, Apr 29, 2011 at 5:57 AM, Wilson Tuladhar <mailwilson007 at gmail.com>wrote:
>
>> Hi,
>>
>> Here the error that get logged into sasl-error.log file.
>>
>> =ERROR REPORT==== 29-Apr-2011::14:45:37 ===
>> ** State machine <0.3117.0> terminating
>> ** Last event in was timeout
>> ** When State == initialize
>> **      Data  ==
>> {state,<0.3101.0>,plain,<<>>,undefined,undefined,undefined,
>>                   undefined,<<"Games">>,
>>                   {<<"Games">>,[]},
>>                   60000,69622590,
>>                   {chstate,'riaksearch at 127.0.0.1',
>>                     [{'riaksearch at 127.0.0.1',{4,63470357906}}],
>>                    {64,
>>                     [{0,'riaksearch at 127.0.0.1'},
>>                      {22835963083295358096932575511191922182123945984,
>>                       'riaksearch at 127.0.0.1'},
>>                      {45671926166590716193865151022383844364247891968,
>>                       'riaksearch at 127.0.0.1'},
>>                      {68507889249886074290797726533575766546371837952,
>>                       'riaksearch at 127.0.0.1'},
>>                      {91343852333181432387730302044767688728495783936,
>>                       'riaksearch at 127.0.0.1'},
>>                      {114179815416476790484662877555959610910619729920,
>>                       'riaksearch at 127.0.0.1'},
>>                      {137015778499772148581595453067151533092743675904,
>>                       'riaksearch at 127.0.0.1'},
>>                      {159851741583067506678528028578343455274867621888,
>>                       'riaksearch at 127.0.0.1'},
>>                      {182687704666362864775460604089535377456991567872,
>>                       'riaksearch at 127.0.0.1'},
>>                      {205523667749658222872393179600727299639115513856,
>>                       'riaksearch at 127.0.0.1'},
>>                      {228359630832953580969325755111919221821239459840,
>>                       'riaksearch at 127.0.0.1'},
>>                      {251195593916248939066258330623111144003363405824,
>>                       'riaksearch at 127.0.0.1'},
>>                      {274031556999544297163190906134303066185487351808,
>>                       'riaksearch at 127.0.0.1'},
>>                      {296867520082839655260123481645494988367611297792,
>>                       'riaksearch at 127.0.0.1'},
>>                      {319703483166135013357056057156686910549735243776,
>>                       'riaksearch at 127.0.0.1'},
>>                      {342539446249430371453988632667878832731859189760,
>>                       'riaksearch at 127.0.0.1'},
>>                      {365375409332725729550921208179070754913983135744,
>>                       'riaksearch at 127.0.0.1'},
>>                      {388211372416021087647853783690262677096107081728,
>>                       'riaksearch at 127.0.0.1'},
>>                      {411047335499316445744786359201454599278231027712,
>>                       'riaksearch at 127.0.0.1'},
>>                      {433883298582611803841718934712646521460354973696,
>>                       'riaksearch at 127.0.0.1'},
>>                      {456719261665907161938651510223838443642478919680,
>>                       'riaksearch at 127.0.0.1'},
>>                      {479555224749202520035584085735030365824602865664,
>>                       'riaksearch at 127.0.0.1'},
>>                      {502391187832497878132516661246222288006726811648,
>>                       'riaksearch at 127.0.0.1'},
>>                      {525227150915793236229449236757414210188850757632,
>>                       'riaksearch at 127.0.0.1'},
>>                      {548063113999088594326381812268606132370974703616,
>>                       'riaksearch at 127.0.0.1'},
>>                      {570899077082383952423314387779798054553098649600,
>>                       'riaksearch at 127.0.0.1'},
>>                      {593735040165679310520246963290989976735222595584,
>>                       'riaksearch at 127.0.0.1'},
>>                      {616571003248974668617179538802181898917346541568,
>>                       'riaksearch at 127.0.0.1'},
>>                      {639406966332270026714112114313373821099470487552,
>>                       'riaksearch at 127.0.0.1'},
>>                      {662242929415565384811044689824565743281594433536,
>>                       'riaksearch at 127.0.0.1'},
>>                      {685078892498860742907977265335757665463718379520,
>>                       'riaksearch at 127.0.0.1'},
>>                      {707914855582156101004909840846949587645842325504,
>>                       'riaksearch at 127.0.0.1'},
>>                      {730750818665451459101842416358141509827966271488,
>>                       'riaksearch at 127.0.0.1'},
>>                      {753586781748746817198774991869333432010090217472,
>>                       'riaksearch at 127.0.0.1'},
>>                      {776422744832042175295707567380525354192214163456,
>>                       'riaksearch at 127.0.0.1'},
>>                      {799258707915337533392640142891717276374338109440,
>>                       'riaksearch at 127.0.0.1'},
>>                      {822094670998632891489572718402909198556462055424,
>>                       'riaksearch at 127.0.0.1'},
>>                      {844930634081928249586505293914101120738586001408,
>>                       'riaksearch at 127.0.0.1'},
>>                      {867766597165223607683437869425293042920709947392,
>>                       'riaksearch at 127.0.0.1'},
>>                      {890602560248518965780370444936484965102833893376,
>>                       'riaksearch at 127.0.0.1'},
>>                      {913438523331814323877303020447676887284957839360,
>>                       'riaksearch at 127.0.0.1'},
>>                      {936274486415109681974235595958868809467081785344,
>>                       'riaksearch at 127.0.0.1'},
>>                      {959110449498405040071168171470060731649205731328,
>>                       'riaksearch at 127.0.0.1'},
>>                      {981946412581700398168100746981252653831329677312,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1004782375664995756265033322492444576013453623296,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1027618338748291114361965898003636498195577569280,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1050454301831586472458898473514828420377701515264,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1073290264914881830555831049026020342559825461248,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1096126227998177188652763624537212264741949407232,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1118962191081472546749696200048404186924073353216,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1141798154164767904846628775559596109106197299200,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1164634117248063262943561351070788031288321245184,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1187470080331358621040493926581979953470445191168,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1210306043414653979137426502093171875652569137152,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1233142006497949337234359077604363797834693083136,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1255977969581244695331291653115555720016817029120,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1278813932664540053428224228626747642198940975104,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1301649895747835411525156804137939564381064921088,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1324485858831130769622089379649131486563188867072,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1347321821914426127719021955160323408745312813056,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1370157784997721485815954530671515330927436759040,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1392993748081016843912887106182707253109560705024,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1415829711164312202009819681693899175291684651008,
>>                       'riaksearch at 127.0.0.1'},
>>                      {1438665674247607560106752257205091097473808596992,
>>                       'riaksearch at 127.0.0.1'}]},
>>                    {dict,4,16,16,8,80,48,
>>                     {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>>                     {{[],[],[],[],
>>                       [[{bucket,<<"Games">>}|
>>                         {meta_entry,
>>                          [{name,<<"Games">>},
>>                           {n_val,3},
>>                           {allow_mult,false},
>>                           {last_write_wins,false},
>>                           {precommit,
>>                            [{struct,
>>                              [{<<"mod">>,<<"riak_search_kv_hook">>},
>>                               {<<"fun">>,<<"precommit">>}]}]},
>>                           {postcommit,[]},
>>
>>  {chash_keyfun,{riak_core_util,chash_std_keyfun}},
>>                           {linkfun,
>>
>> {modfun,riak_kv_wm_link_walker,mapreduce_linkfun}},
>>                           {old_vclock,86400},
>>                           {young_vclock,20},
>>                           {big_vclock,50},
>>                           {small_vclock,10},
>>                           {r,quorum},
>>                           {w,quorum},
>>                           {dw,quorum},
>>                           {rw,quorum}],
>>                          63470357892}]],
>>                       [],[],
>>                       [[{bucket,<<"User">>}|
>>                         {meta_entry,
>>                          [{name,<<"User">>},
>>                           {n_val,3},
>>                           {allow_mult,false},
>>                           {last_write_wins,false},
>>                           {precommit,
>>                            [{struct,
>>                              [{<<"mod">>,<<"riak_search_kv_hook">>},
>>                               {<<"fun">>,<<"precommit">>}]}]},
>>                           {postcommit,[]},
>>
>>  {chash_keyfun,{riak_core_util,chash_std_keyfun}},
>>                           {linkfun,
>>
>> {modfun,riak_kv_wm_link_walker,mapreduce_linkfun}},
>>                           {old_vclock,86400},
>>                           {young_vclock,20},
>>                           {big_vclock,50},
>>                           {small_vclock,10},
>>                           {r,quorum},
>>                           {w,quorum},
>>                           {dw,quorum},
>>                           {rw,quorum}],
>>                          63470357885}],
>>                        [{bucket,<<"Game_instances">>}|
>>                         {meta_entry,
>>                          [{name,<<"Game_instances">>},
>>                           {n_val,3},
>>                           {allow_mult,false},
>>                           {last_write_wins,false},
>>                           {precommit,
>>                            [{struct,
>>                              [{<<"mod">>,<<"riak_search_kv_hook">>},
>>                               {<<"fun">>,<<"precommit">>}]}]},
>>                           {postcommit,[]},
>>
>>  {chash_keyfun,{riak_core_util,chash_std_keyfun}},
>>                           {linkfun,
>>
>> {modfun,riak_kv_wm_link_walker,mapreduce_linkfun}},
>>                           {old_vclock,86400},
>>                           {young_vclock,20},
>>                           {big_vclock,50},
>>                           {small_vclock,10},
>>                           {r,quorum},
>>                           {w,quorum},
>>                           {dw,quorum},
>>                           {rw,quorum}],
>>                          63470357900}]],
>>                       [],[],[],[],[],[],[],
>>                       [[{bucket,<<"Game_instances_run">>}|
>>                         {meta_entry,
>>                          [{name,<<"Game_instances_run">>},
>>                           {n_val,3},
>>                           {allow_mult,false},
>>                           {last_write_wins,false},
>>                           {precommit,
>>                            [{struct,
>>                              [{<<"mod">>,<<"riak_search_kv_hook">>},
>>                               {<<"fun">>,<<"precommit">>}]}]},
>>                           {postcommit,[]},
>>
>>  {chash_keyfun,{riak_core_util,chash_std_keyfun}},
>>                           {linkfun,
>>
>> {modfun,riak_kv_wm_link_walker,mapreduce_linkfun}},
>>                           {old_vclock,86400},
>>                           {young_vclock,20},
>>                           {big_vclock,50},
>>                           {small_vclock,10},
>>                           {r,quorum},
>>                           {w,quorum},
>>                           {dw,quorum},
>>                           {rw,quorum}],
>>                          63470357906}]]}}}},
>>                   undefined}
>> ** Reason for termination =
>> ** {timeout,{gen_server,call,
>>                         [{riak_kv_keylister_master,'riaksearch at 127.0.0.1
>> '},
>>
>> {start_kl,69622590,<0.3117.0>,{<<"Games">>,[]}}]}}
>>
>> =CRASH REPORT==== 29-Apr-2011::14:45:38 ===
>>   crasher:
>>     initial call: riak_kv_keys_fsm:init/1
>>     pid: <0.3117.0>
>>     registered_name: []
>>     exception exit: {timeout,{gen_server,call,[{riak_kv_keylister_master,
>> riaksearch at 127.0.0.1},{start_kl,69622590,<0.3117.0>,{<<"Games">>,[]}}]}}
>>       in function  gen_fsm:terminate/7
>>       in call from proc_lib:init_p_do_apply/3
>>     ancestors: [<0.3110.0>]
>>     messages: []
>>     links: []
>>     dictionary: []
>>     trap_exit: true
>>     status: running
>>     heap_size: 610
>>     stack_size: 24
>>     reductions: 9906
>>   neighbours:
>>
>>
>> And the functions that i have made are
>> %% gets all the keys in the bucket
>> get_bucket_keys(Bucket) ->
>>     {ok, Pid} = riakc_pb_socket:start_link("127.0.0.1", 8087),
>>     {ok, List} = riakc_pb_socket:list_keys(Pid, Bucket,infinity),
>>     riakc_pb_socket:stop(Pid),
>>     {ok,lists:foldr(fun(Key, Acc) ->
>>     [binary_to_list(Key)|Acc]
>>     end, [], List)}.
>>
>> %% select the Key and get the values to return
>> select(DB, Key, Values_to_return) ->
>>     {ok, Pid} = riakc_pb_socket:start_link("127.0.0.1", 8087),
>>     Ret =
>> case riakc_pb_socket:get(Pid, list_to_binary(DB),
>>  list_to_binary(Key)) of
>>     {ok, FromDBObject} ->
>> Val = to_list(binary_to_term(riakc_obj:get_value(FromDBObject))),
>>  riakc_pb_socket:stop(Pid),
>> Return =
>>     lists:foldr(
>>       fun(Key_for_return, Acc)->
>>       case proplists:get_value(Key_for_return, Val) of
>>   undefined -> Acc;
>>   Sth -> [{Key_for_return,Sth} | Acc]
>>       end
>>       end, [], Values_to_return),
>> {ok, Return};
>>     {error, Msg} ->
>>  {error, Msg}
>> end,
>>     Ret.
>>
>> And I do the following to get all the keys in the bucket and then find
>> some field from that key and return the value.
>> {ok, Key_list} = db_api:get_bucket_keys("Games"),
>>     lists:foldl(
>>       fun(Key, Acc) ->
>>       case db_api:select("Games", Key, ["game_name"]) of
>>   {ok, []} ->
>>       Acc;
>>   {ok, List} -> [[{"game_id", Key} | List]|Acc]
>>       end
>>       end,[], Key_list).
>>
>>
>> It works for like around 2-3 seconds then it starts to timeout and
>> eventually crash..
>>
>> //Wilson
>>
>>
>> On Fri, Apr 29, 2011 at 2:01 PM, Rusty Klophaus <rusty at basho.com> wrote:
>>
>>> Hi Wilson,
>>>
>>> Do you see any errors or otherwise abnormal looking messages in the logs?
>>> That will help diagnose the issue.
>>>
>>> The max requests per second depends very much on your hardware, size of
>>> data, and read/write patterns.
>>>
>>> Thanks,
>>> Rusty
>>>
>>> On Fri, Apr 29, 2011 at 6:44 AM, Wilson Tuladhar <
>>> mailwilson007 at gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am using riaksearch with erlang client and i have made an API that
>>>> does for the select, insert and other such stuffs. I have built an web
>>>> application on top of it and everything works fine when i do the manual
>>>> testing.
>>>>
>>>> But now when i am using the automated load testing through Tsung by
>>>> sending 100 req/sec, the riak shuts down after few seconds. Does anybody
>>>> know why this is happening or how much requests can riak handle in a second.
>>>> I am using only single node currently.
>>>>
>>>> //Wilson
>>>>
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> riak-users at lists.basho.com
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>
>>>>
>>>
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20110502/b8c5af7f/attachment.html>


More information about the riak-users mailing list