Riak shutdown during heavy load testing

Wilson Tuladhar mailwilson007 at gmail.com
Fri Apr 29 08:57:58 EDT 2011


Hi,

Here the error that get logged into sasl-error.log file.

=ERROR REPORT==== 29-Apr-2011::14:45:37 ===
** State machine <0.3117.0> terminating
** Last event in was timeout
** When State == initialize
**      Data  == {state,<0.3101.0>,plain,<<>>,undefined,undefined,undefined,
                  undefined,<<"Games">>,
                  {<<"Games">>,[]},
                  60000,69622590,
                  {chstate,'riaksearch at 127.0.0.1',
                   [{'riaksearch at 127.0.0.1',{4,63470357906}}],
                   {64,
                    [{0,'riaksearch at 127.0.0.1'},
                     {22835963083295358096932575511191922182123945984,
                      'riaksearch at 127.0.0.1'},
                     {45671926166590716193865151022383844364247891968,
                      'riaksearch at 127.0.0.1'},
                     {68507889249886074290797726533575766546371837952,
                      'riaksearch at 127.0.0.1'},
                     {91343852333181432387730302044767688728495783936,
                      'riaksearch at 127.0.0.1'},
                     {114179815416476790484662877555959610910619729920,
                      'riaksearch at 127.0.0.1'},
                     {137015778499772148581595453067151533092743675904,
                      'riaksearch at 127.0.0.1'},
                     {159851741583067506678528028578343455274867621888,
                      'riaksearch at 127.0.0.1'},
                     {182687704666362864775460604089535377456991567872,
                      'riaksearch at 127.0.0.1'},
                     {205523667749658222872393179600727299639115513856,
                      'riaksearch at 127.0.0.1'},
                     {228359630832953580969325755111919221821239459840,
                      'riaksearch at 127.0.0.1'},
                     {251195593916248939066258330623111144003363405824,
                      'riaksearch at 127.0.0.1'},
                     {274031556999544297163190906134303066185487351808,
                      'riaksearch at 127.0.0.1'},
                     {296867520082839655260123481645494988367611297792,
                      'riaksearch at 127.0.0.1'},
                     {319703483166135013357056057156686910549735243776,
                      'riaksearch at 127.0.0.1'},
                     {342539446249430371453988632667878832731859189760,
                      'riaksearch at 127.0.0.1'},
                     {365375409332725729550921208179070754913983135744,
                      'riaksearch at 127.0.0.1'},
                     {388211372416021087647853783690262677096107081728,
                      'riaksearch at 127.0.0.1'},
                     {411047335499316445744786359201454599278231027712,
                      'riaksearch at 127.0.0.1'},
                     {433883298582611803841718934712646521460354973696,
                      'riaksearch at 127.0.0.1'},
                     {456719261665907161938651510223838443642478919680,
                      'riaksearch at 127.0.0.1'},
                     {479555224749202520035584085735030365824602865664,
                      'riaksearch at 127.0.0.1'},
                     {502391187832497878132516661246222288006726811648,
                      'riaksearch at 127.0.0.1'},
                     {525227150915793236229449236757414210188850757632,
                      'riaksearch at 127.0.0.1'},
                     {548063113999088594326381812268606132370974703616,
                      'riaksearch at 127.0.0.1'},
                     {570899077082383952423314387779798054553098649600,
                      'riaksearch at 127.0.0.1'},
                     {593735040165679310520246963290989976735222595584,
                      'riaksearch at 127.0.0.1'},
                     {616571003248974668617179538802181898917346541568,
                      'riaksearch at 127.0.0.1'},
                     {639406966332270026714112114313373821099470487552,
                      'riaksearch at 127.0.0.1'},
                     {662242929415565384811044689824565743281594433536,
                      'riaksearch at 127.0.0.1'},
                     {685078892498860742907977265335757665463718379520,
                      'riaksearch at 127.0.0.1'},
                     {707914855582156101004909840846949587645842325504,
                      'riaksearch at 127.0.0.1'},
                     {730750818665451459101842416358141509827966271488,
                      'riaksearch at 127.0.0.1'},
                     {753586781748746817198774991869333432010090217472,
                      'riaksearch at 127.0.0.1'},
                     {776422744832042175295707567380525354192214163456,
                      'riaksearch at 127.0.0.1'},
                     {799258707915337533392640142891717276374338109440,
                      'riaksearch at 127.0.0.1'},
                     {822094670998632891489572718402909198556462055424,
                      'riaksearch at 127.0.0.1'},
                     {844930634081928249586505293914101120738586001408,
                      'riaksearch at 127.0.0.1'},
                     {867766597165223607683437869425293042920709947392,
                      'riaksearch at 127.0.0.1'},
                     {890602560248518965780370444936484965102833893376,
                      'riaksearch at 127.0.0.1'},
                     {913438523331814323877303020447676887284957839360,
                      'riaksearch at 127.0.0.1'},
                     {936274486415109681974235595958868809467081785344,
                      'riaksearch at 127.0.0.1'},
                     {959110449498405040071168171470060731649205731328,
                      'riaksearch at 127.0.0.1'},
                     {981946412581700398168100746981252653831329677312,
                      'riaksearch at 127.0.0.1'},
                     {1004782375664995756265033322492444576013453623296,
                      'riaksearch at 127.0.0.1'},
                     {1027618338748291114361965898003636498195577569280,
                      'riaksearch at 127.0.0.1'},
                     {1050454301831586472458898473514828420377701515264,
                      'riaksearch at 127.0.0.1'},
                     {1073290264914881830555831049026020342559825461248,
                      'riaksearch at 127.0.0.1'},
                     {1096126227998177188652763624537212264741949407232,
                      'riaksearch at 127.0.0.1'},
                     {1118962191081472546749696200048404186924073353216,
                      'riaksearch at 127.0.0.1'},
                     {1141798154164767904846628775559596109106197299200,
                      'riaksearch at 127.0.0.1'},
                     {1164634117248063262943561351070788031288321245184,
                      'riaksearch at 127.0.0.1'},
                     {1187470080331358621040493926581979953470445191168,
                      'riaksearch at 127.0.0.1'},
                     {1210306043414653979137426502093171875652569137152,
                      'riaksearch at 127.0.0.1'},
                     {1233142006497949337234359077604363797834693083136,
                      'riaksearch at 127.0.0.1'},
                     {1255977969581244695331291653115555720016817029120,
                      'riaksearch at 127.0.0.1'},
                     {1278813932664540053428224228626747642198940975104,
                      'riaksearch at 127.0.0.1'},
                     {1301649895747835411525156804137939564381064921088,
                      'riaksearch at 127.0.0.1'},
                     {1324485858831130769622089379649131486563188867072,
                      'riaksearch at 127.0.0.1'},
                     {1347321821914426127719021955160323408745312813056,
                      'riaksearch at 127.0.0.1'},
                     {1370157784997721485815954530671515330927436759040,
                      'riaksearch at 127.0.0.1'},
                     {1392993748081016843912887106182707253109560705024,
                      'riaksearch at 127.0.0.1'},
                     {1415829711164312202009819681693899175291684651008,
                      'riaksearch at 127.0.0.1'},
                     {1438665674247607560106752257205091097473808596992,
                      'riaksearch at 127.0.0.1'}]},
                   {dict,4,16,16,8,80,48,
                    {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                    {{[],[],[],[],
                      [[{bucket,<<"Games">>}|
                        {meta_entry,
                         [{name,<<"Games">>},
                          {n_val,3},
                          {allow_mult,false},
                          {last_write_wins,false},
                          {precommit,
                           [{struct,
                             [{<<"mod">>,<<"riak_search_kv_hook">>},
                              {<<"fun">>,<<"precommit">>}]}]},
                          {postcommit,[]},
                          {chash_keyfun,{riak_core_util,chash_std_keyfun}},
                          {linkfun,

{modfun,riak_kv_wm_link_walker,mapreduce_linkfun}},
                          {old_vclock,86400},
                          {young_vclock,20},
                          {big_vclock,50},
                          {small_vclock,10},
                          {r,quorum},
                          {w,quorum},
                          {dw,quorum},
                          {rw,quorum}],
                         63470357892}]],
                      [],[],
                      [[{bucket,<<"User">>}|
                        {meta_entry,
                         [{name,<<"User">>},
                          {n_val,3},
                          {allow_mult,false},
                          {last_write_wins,false},
                          {precommit,
                           [{struct,
                             [{<<"mod">>,<<"riak_search_kv_hook">>},
                              {<<"fun">>,<<"precommit">>}]}]},
                          {postcommit,[]},
                          {chash_keyfun,{riak_core_util,chash_std_keyfun}},
                          {linkfun,

{modfun,riak_kv_wm_link_walker,mapreduce_linkfun}},
                          {old_vclock,86400},
                          {young_vclock,20},
                          {big_vclock,50},
                          {small_vclock,10},
                          {r,quorum},
                          {w,quorum},
                          {dw,quorum},
                          {rw,quorum}],
                         63470357885}],
                       [{bucket,<<"Game_instances">>}|
                        {meta_entry,
                         [{name,<<"Game_instances">>},
                          {n_val,3},
                          {allow_mult,false},
                          {last_write_wins,false},
                          {precommit,
                           [{struct,
                             [{<<"mod">>,<<"riak_search_kv_hook">>},
                              {<<"fun">>,<<"precommit">>}]}]},
                          {postcommit,[]},
                          {chash_keyfun,{riak_core_util,chash_std_keyfun}},
                          {linkfun,

{modfun,riak_kv_wm_link_walker,mapreduce_linkfun}},
                          {old_vclock,86400},
                          {young_vclock,20},
                          {big_vclock,50},
                          {small_vclock,10},
                          {r,quorum},
                          {w,quorum},
                          {dw,quorum},
                          {rw,quorum}],
                         63470357900}]],
                      [],[],[],[],[],[],[],
                      [[{bucket,<<"Game_instances_run">>}|
                        {meta_entry,
                         [{name,<<"Game_instances_run">>},
                          {n_val,3},
                          {allow_mult,false},
                          {last_write_wins,false},
                          {precommit,
                           [{struct,
                             [{<<"mod">>,<<"riak_search_kv_hook">>},
                              {<<"fun">>,<<"precommit">>}]}]},
                          {postcommit,[]},
                          {chash_keyfun,{riak_core_util,chash_std_keyfun}},
                          {linkfun,

{modfun,riak_kv_wm_link_walker,mapreduce_linkfun}},
                          {old_vclock,86400},
                          {young_vclock,20},
                          {big_vclock,50},
                          {small_vclock,10},
                          {r,quorum},
                          {w,quorum},
                          {dw,quorum},
                          {rw,quorum}],
                         63470357906}]]}}}},
                  undefined}
** Reason for termination =
** {timeout,{gen_server,call,
                        [{riak_kv_keylister_master,'riaksearch at 127.0.0.1'},
                         {start_kl,69622590,<0.3117.0>,{<<"Games">>,[]}}]}}

=CRASH REPORT==== 29-Apr-2011::14:45:38 ===
  crasher:
    initial call: riak_kv_keys_fsm:init/1
    pid: <0.3117.0>
    registered_name: []
    exception exit: {timeout,{gen_server,call,[{riak_kv_keylister_master,
riaksearch at 127.0.0.1},{start_kl,69622590,<0.3117.0>,{<<"Games">>,[]}}]}}
      in function  gen_fsm:terminate/7
      in call from proc_lib:init_p_do_apply/3
    ancestors: [<0.3110.0>]
    messages: []
    links: []
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 610
    stack_size: 24
    reductions: 9906
  neighbours:


And the functions that i have made are
%% gets all the keys in the bucket
get_bucket_keys(Bucket) ->
    {ok, Pid} = riakc_pb_socket:start_link("127.0.0.1", 8087),
    {ok, List} = riakc_pb_socket:list_keys(Pid, Bucket,infinity),
    riakc_pb_socket:stop(Pid),
    {ok,lists:foldr(fun(Key, Acc) ->
    [binary_to_list(Key)|Acc]
    end, [], List)}.

%% select the Key and get the values to return
select(DB, Key, Values_to_return) ->
    {ok, Pid} = riakc_pb_socket:start_link("127.0.0.1", 8087),
    Ret =
case riakc_pb_socket:get(Pid, list_to_binary(DB),
 list_to_binary(Key)) of
    {ok, FromDBObject} ->
Val = to_list(binary_to_term(riakc_obj:get_value(FromDBObject))),
 riakc_pb_socket:stop(Pid),
Return =
    lists:foldr(
      fun(Key_for_return, Acc)->
      case proplists:get_value(Key_for_return, Val) of
  undefined -> Acc;
  Sth -> [{Key_for_return,Sth} | Acc]
      end
      end, [], Values_to_return),
{ok, Return};
    {error, Msg} ->
 {error, Msg}
end,
    Ret.

And I do the following to get all the keys in the bucket and then find some
field from that key and return the value.
{ok, Key_list} = db_api:get_bucket_keys("Games"),
    lists:foldl(
      fun(Key, Acc) ->
      case db_api:select("Games", Key, ["game_name"]) of
  {ok, []} ->
      Acc;
  {ok, List} -> [[{"game_id", Key} | List]|Acc]
      end
      end,[], Key_list).


It works for like around 2-3 seconds then it starts to timeout and
eventually crash..

//Wilson


On Fri, Apr 29, 2011 at 2:01 PM, Rusty Klophaus <rusty at basho.com> wrote:

> Hi Wilson,
>
> Do you see any errors or otherwise abnormal looking messages in the logs?
> That will help diagnose the issue.
>
> The max requests per second depends very much on your hardware, size of
> data, and read/write patterns.
>
> Thanks,
> Rusty
>
> On Fri, Apr 29, 2011 at 6:44 AM, Wilson Tuladhar <mailwilson007 at gmail.com>wrote:
>
>> Hi,
>>
>> I am using riaksearch with erlang client and i have made an API that does
>> for the select, insert and other such stuffs. I have built an web
>> application on top of it and everything works fine when i do the manual
>> testing.
>>
>> But now when i am using the automated load testing through Tsung by
>> sending 100 req/sec, the riak shuts down after few seconds. Does anybody
>> know why this is happening or how much requests can riak handle in a second.
>> I am using only single node currently.
>>
>> //Wilson
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20110429/7426e97a/attachment-0001.html>


More information about the riak-users mailing list