Riak 0.7.1 list_buckets times out on Mac OS X but not on Ubuntu

Rusty Klophaus rusty at basho.com
Wed Feb 10 12:08:51 EST 2010


Hi Zubair,

Yes, the current implementation of list_buckets expects all nodes to be
present, otherwise it has no way of finding all of the buckets, so this is
caused by 192.168.1.5 being down.

We've thought about different options, such as adding a boolean flag to
list_buckets that tells the system whether to proceed on error or timeout.

The upshot is that, list_buckets (just like list_keys) is not generally
something you want to do often, as it is resource intensive when you have a
large number of objects.

Best,
Rusty

On Wed, Feb 10, 2010 at 12:02 PM, Zubair Quraishi <zubairq at gmail.com> wrote:

> *Yes, of course. Some extra info I forgot earlier is that I have two
> nodes, 192.168.1.3 and 192.168.1.5, and 192.168.1.5 is down.
> *
> *- I'm using the default backend*
>
> *- just 1 bucket I think (groceries :)
> *
> *-  {ok, C} = riak:client_connect('riak at 192.168.1.3').
> *
> * *{ok,{riak_client,'riak at 192.168.1.3',<<0,38,76,114>>}}
> *(riaktest at users-Mac-Pro.local)**2> C:list_buckets().
> *
> =ERROR REPORT==== 10-Feb-2010::17:49:58 ===
> ** State machine <5066.1052.0> terminating
> ** Last event in was timeout
> ** When State == waiting_kl
> **      Data  == {state,<0.46.0>,plain,
>                      {sbf,3.0e-5,0.85,1,0,
>                          [{bloom,4.500000000000001e-6,11784498,24,0,
>                               [{array,621379,0,0,1000000},
>                                {array,621379,0,0,1000000},
>                                {array,621379,0,0,1000000},
> ...
>                                {array,621379,0,0,1000000},
>                                {array,621379,0,0,1000000},
>                                {array,621379,0,0,1000000}]}]},
>                      [{45671926166590716193865151022383844364247891968,
>                        'riak at 192.168.1.3'},
>                       {68507889249886074290797726533575766546371837952,
>                        'riak at 192.168.1.5'},
>                       {91343852333181432387730302044767688728495783936,
>                        'riak at 192.168.1.3'},
>                       {114179815416476790484662877555959610910619729920,
>                        'riak at 192.168.1.5'},
>                       {137015778499772148581595453067151533092743675904,
>                        'riak at 192.168.1.3'},
>                       {159851741583067506678528028578343455274867621888,
>                        'riak at 192.168.1.5'},
>                       {182687704666362864775460604089535377456991567872,
>
> ...
>                       {1050454301831586472458898473514828420377701515264,
>
>                        'riak at 192.168.1.3'},
>                       {1347321821914426127719021955160323408745312813056,
>                        'riak at 192.168.1.5'},
>                       {1370157784997721485815954530671515330927436759040,
>                        'riak at 192.168.1.3'},
>                       {1392993748081016843912887106182707253109560705024,
>                        'riak at 192.168.1.5'},
>                       {1415829711164312202009819681693899175291684651008,
>                        'riak at 192.168.1.3'},
>                       {1438665674247607560106752257205091097473808596992,
>                        'riak at 192.168.1.5'}],
>                      '_',120000,112621262,
>                      {chstate,'riak at 192.168.1.3',
>                          [{'riak at 192.168.1.5',{32,63432664825}}],
>                          {64,
>                           [{0,'riak at 192.168.1.3'},
>
> {22835963083295358096932575511191922182123945984,
>                             'riak at 192.168.1.5'},
>
> {45671926166590716193865151022383844364247891968,
>                             'riak at 192.168.1.3'},
>
> {68507889249886074290797726533575766546371837952,
>                             'riak at 192.168.1.5'},
>
> {91343852333181432387730302044767688728495783936,
>                             'riak at 192.168.1.3'},
>
> {114179815416476790484662877555959610910619729920,
>                             'riak at 192.168.1.5'},
>
> {137015778499772148581595453067151533092743675904,
>                             'riak at 192.168.1.3'},
>
> {159851741583067506678528028578343455274867621888,
>                             'riak at 192.168.1.5'},
> ...
>                             'riak at 192.168.1.5'},
>
> {1415829711164312202009819681693899175291684651008,
>                             'riak at 192.168.1.3'},
>
> {1438665674247607560106752257205091097473808596992,
>                             'riak at 192.168.1.5'}]},
>                          {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                                []}}}}}
> ** Reason for termination =
> ** {function_clause,
>        [{riak_keys_fsm,waiting_kl,
>             [timeout,
>              {state,<0.46.0>,plain,
>                  {sbf,3.0e-5,0.85,1,0,
>                      [{bloom,4.500000000000001e-6,11784498,24,0,
>                           [{array,621379,0,0,1000000},
>                            {array,621379,0,0,1000000},
>                            {array,621379,0,0,1000000},
> ...
>                            {array,621379,0,0,1000000},
>                            {array,621379,0,0,1000000}]}]},
>                  [{45671926166590716193865151022383844364247891968,
>                    'riak at 192.168.1.3'},
>                   {68507889249886074290797726533575766546371837952,
>                    'riak at 192.168.1.5'},
> ...
>                   {1233142006497949337234359077604363797834693083136,
>                    'riak at 192.168.1.3'},
>                   {1255977969581244695331291653115555720016817029120,
>                    'riak at 192.168.1.5'},
>                   {1278813932664540053428224228626747642198940975104,
>                    'riak at 192.168.1.3'},
>                   {1438665674247607560106752257205091097473808596992,
>                    'riak at 192.168.1.5'}],
>                  '_',120000,112621262,
>                  {chstate,'riak at 192.168.1.3',
>                      [{'riak at 192.168.1.5',{32,63432664825}}],
>                      {64,
>                       [{0,'riak at 192.168.1.3'},
>                        {22835963083295358096932575511191922182123945984,
>                         'riak at 192.168.1.5'},
> .......
>                         'riak at 192.168.1.3'},
>                        {1392993748081016843912887106182707253109560705024,
>                         'riak at 192.168.1.5'},
>                        {1415829711164312202009819681693899175291684651008,
>                         'riak at 192.168.1.3'},
>                        {1438665674247607560106752257205091097473808596992,
>                         'riak at 192.168.1.5'}]},
>                      {dict,0,16,16,8,80,48,
>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                          {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                            []}}}}}]},
>         {gen_fsm,handle_msg,7},
>         {proc_lib,init_p_do_apply,3}]}
> ** at node riak at 192.168.1.3 **
>
>
> *- Yes, deleting data/ring did work, so I guess when the second Riak node
> dropped it caused some problem?
> *
>
> *- multi-node, but one node down
> *
> *- users-Mac-Pro:riak root# ./bin/riak console*
> Exec: /opt/local/bin/riak/rel/riak/erts-5.7.4/bin/erlexec -boot
> /opt/local/bin/riak/rel/riak/releases/0.8/riak -embedded -config
> /opt/local/bin/riak/rel/riak/etc/app.config -args_file
> /opt/local/bin/riak/rel/riak/etc/vm.args -- console
> Root: /opt/local/bin/riak/rel/riak
> Erlang R13B03 (erts-5.7.4) [source] [64-bit] [smp:8:8] [rq:8]
> [async-threads:5] [hipe] [kernel-poll:true]
>
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/kernel-2.13.4.ez/kernel-2.13.4/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/stdlib-1.16.4.ez/stdlib-1.16.4/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/sasl-2.1.8.ez/sasl-2.1.8/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/crypto-1.6.3.ez/crypto-1.6.3/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/runtime_tools-1.8.2.ez/runtime_tools-1.8.2/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/mochiweb-0.01.ez/mochiweb-0.01/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/webmachine-1.5.2.ez/webmachine-1.5.2/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/riak-0.7.1.ez/riak-0.7.1/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/xmerl-1.2.3.ez/xmerl-1.2.3/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/webtool-0.8.5.ez/webtool-0.8.5/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/tools-2.6.5.ez/tools-2.6.5/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/syntax_tools-1.6.4.ez/syntax_tools-1.6.4/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/ssl-3.10.7.ez/ssl-3.10.7/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/snmp-4.15.ez/snmp-4.15/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/public_key-0.4.ez/public_key-0.4/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/protobuffs-4.ez/protobuffs-4/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/otp_mibs-1.0.6.ez/otp_mibs-1.0.6/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/os_mon-2.2.4.ez/os_mon-2.2.4/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/observer-0.9.8.1.ez/observer-0.9.8.1/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/mnesia-4.4.12.ez/mnesia-4.4.12/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/inets-5.2.ez/inets-5.2/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/hipe-3.7.4.ez/hipe-3.7.4/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/gs-1.5.11.ez/gs-1.5.11/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/eunit-2.1.4.ez/eunit-2.1.4/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/et-1.3.3.ez/et-1.3.3/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/erts-5.7.4.ez/erts-5.7.4/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/edoc-0.7.6.5.ez/edoc-0.7.6.5/ebin
> ** Bad path can't read
> /opt/local/bin/riak/rel/riak/lib/compiler-4.6.4.ez/compiler-4.6.4/ebin
> ** Found 0 name clashes in code paths
> Eshell V5.7.4  (abort with ^G)
> (riak at 192.168.1.3)1>
>
>
> *(riak at 192.168.1.3)1> io:format("~p", [riak_ring_manager:get_my_ring()]).*
> {ok,{chstate,'riak at 192.168.1.3',[],
>
>              {64,
>               [{0,'riak at 192.168.1.3'},
>                {22835963083295358096932575511191922182123945984,
>                 'riak at 192.168.1.3'},
>
>                {45671926166590716193865151022383844364247891968,
>                 'riak at 192.168.1.3'},
>                {68507889249886074290797726533575766546371837952,
>                 'riak at 192.168.1.3'},
>
>                {91343852333181432387730302044767688728495783936,
>                 'riak at 192.168.1.3'},
>                {114179815416476790484662877555959610910619729920,
>                 'riak at 192.168.1.3'},
>
>                {137015778499772148581595453067151533092743675904,
>                 'riak at 192.168.1.3'},
>                {159851741583067506678528028578343455274867621888,
>                 'riak at 192.168.1.3'},
>
>                {182687704666362864775460604089535377456991567872,
>                 'riak at 192.168.1.3'},
>                {205523667749658222872393179600727299639115513856,
>                 'riak at 192.168.1.3'},
>
>                {228359630832953580969325755111919221821239459840,
>                 'riak at 192.168.1.3'},
>                {251195593916248939066258330623111144003363405824,
>                 'riak at 192.168.1.3'},
>
>                {274031556999544297163190906134303066185487351808,
>                 'riak at 192.168.1.3'},
>                {296867520082839655260123481645494988367611297792,
>                 'riak at 192.168.1.3'},
>
>                {319703483166135013357056057156686910549735243776,
>                 'riak at 192.168.1.3'},
>                {342539446249430371453988632667878832731859189760,
>                 'riak at 192.168.1.3'},
>
>                {365375409332725729550921208179070754913983135744,
>                 'riak at 192.168.1.3'},
>                {388211372416021087647853783690262677096107081728,
>                 'riak at 192.168.1.3'},
>
>                {411047335499316445744786359201454599278231027712,
>                 'riak at 192.168.1.3'},
>                {433883298582611803841718934712646521460354973696,
>                 'riak at 192.168.1.3'},
>                {456719261665907161938651510223838443642478919680,
>                 'riak at 192.168.1.3'},
>                {479555224749202520035584085735030365824602865664,
>                 'riak at 192.168.1.3'},
>                {502391187832497878132516661246222288006726811648,
>                 'riak at 192.168.1.3'},
>                {525227150915793236229449236757414210188850757632,
>                 'riak at 192.168.1.3'},
>                {548063113999088594326381812268606132370974703616,
>                 'riak at 192.168.1.3'},
>                {570899077082383952423314387779798054553098649600,
>                 'riak at 192.168.1.3'},
>                {593735040165679310520246963290989976735222595584,
>                 'riak at 192.168.1.3'},
>                {616571003248974668617179538802181898917346541568,
>                 'riak at 192.168.1.3'},
>                {639406966332270026714112114313373821099470487552,
>                 'riak at 192.168.1.3'},
>                {662242929415565384811044689824565743281594433536,
>                 'riak at 192.168.1.3'},
>                {685078892498860742907977265335757665463718379520,
>                 'riak at 192.168.1.3'},
>                {707914855582156101004909840846949587645842325504,
>                 'riak at 192.168.1.3'},
>                {730750818665451459101842416358141509827966271488,
>                 'riak at 192.168.1.3'},
>                {753586781748746817198774991869333432010090217472,
>                 'riak at 192.168.1.3'},
>                {776422744832042175295707567380525354192214163456,
>                 'riak at 192.168.1.3'},
>                {799258707915337533392640142891717276374338109440,
>                 'riak at 192.168.1.3'},
>                {822094670998632891489572718402909198556462055424,
>                 'riak at 192.168.1.3'},
>                {844930634081928249586505293914101120738586001408,
>                 'riak at 192.168.1.3'},
>                {867766597165223607683437869425293042920709947392,
>                 'riak at 192.168.1.3'},
>                {890602560248518965780370444936484965102833893376,
>                 'riak at 192.168.1.3'},
>                {913438523331814323877303020447676887284957839360,
>                 'riak at 192.168.1.3'},
>                {936274486415109681974235595958868809467081785344,
>                 'riak at 192.168.1.3'},
>                {959110449498405040071168171470060731649205731328,
>                 'riak at 192.168.1.3'},
>                {981946412581700398168100746981252653831329677312,
>                 'riak at 192.168.1.3'},
>                {1004782375664995756265033322492444576013453623296,
>                 'riak at 192.168.1.3'},
>                {1027618338748291114361965898003636498195577569280,
>                 'riak at 192.168.1.3'},
>
>                {1050454301831586472458898473514828420377701515264,
>                 'riak at 192.168.1.3'},
>                {1073290264914881830555831049026020342559825461248,
>                 'riak at 192.168.1.3'},
>                {1096126227998177188652763624537212264741949407232,
>                 'riak at 192.168.1.3'},
>                {1118962191081472546749696200048404186924073353216,
>                 'riak at 192.168.1.3'},
>                {1141798154164767904846628775559596109106197299200,
>                 'riak at 192.168.1.3'},
>                {1164634117248063262943561351070788031288321245184,
>                 'riak at 192.168.1.3'},
>                {1187470080331358621040493926581979953470445191168,
>                 'riak at 192.168.1.3'},
>                {1210306043414653979137426502093171875652569137152,
>                 'riak at 192.168.1.3'},
>
>                {1233142006497949337234359077604363797834693083136,
>                 'riak at 192.168.1.3'},
>                {1255977969581244695331291653115555720016817029120,
>                 'riak at 192.168.1.3'},
>
>                {1278813932664540053428224228626747642198940975104,
>                 'riak at 192.168.1.3'},
>                {1301649895747835411525156804137939564381064921088,
>                 'riak at 192.168.1.3'},
>                {1324485858831130769622089379649131486563188867072,
>
>                 'riak at 192.168.1.3'},
>                {1347321821914426127719021955160323408745312813056,
>                 'riak at 192.168.1.3'},
>
>                {1370157784997721485815954530671515330927436759040,
>                 'riak at 192.168.1.3'},
>                {1392993748081016843912887106182707253109560705024,
>                 'riak at 192.168.1.3'},
>
>                {1415829711164312202009819681693899175291684651008,
>                 'riak at 192.168.1.3'},
>                {1438665674247607560106752257205091097473808596992,
>                 'riak at 192.168.1.3'}]},
>
>              {dict,0,16,16,8,80,48,
>                    {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                    {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}}}ok
> (riak at 192.168.1.3)2>
>
>
> *Zubair*
>
>
> On Wed, Feb 10, 2010 at 5:48 PM, Rusty Klophaus <rusty at basho.com> wrote:
>
>> Hi Zubair,
>>
>> Two more questions:
>>
>> - Is this on a single-node or multi-node cluster?
>> - Can you get to the Riak console (start Riak with "./bin/riak console",
>> run "io:format("~p", [riak_ring_manager:get_my_ring()])." and send me the
>> output?
>>
>> Finally, to clarify my instructions on deleting the "data" directory, you
>> will need to shut down Riak, then rename or delete the ./rel/riak/data
>> directory, and then restart Riak.
>>
>> Thanks,
>> Rusty
>>
>>
>> On Wed, Feb 10, 2010 at 11:43 AM, Rusty Klophaus <rusty at basho.com> wrote:
>>
>>> Hi Zubair,
>>>
>>> Hmmm... that is strange. Can you provide the following information to
>>> help me troubleshoot?:
>>>
>>> - Can you post the actual error message that you see?
>>> - Are you using the same backend on each system? Which backend are you
>>> using?
>>> - Approximately how many buckets have you created, and how many objects
>>> are stored in total?
>>> - If you delete or rename the ./rel/riak/data directory and write more
>>> objects, can you get the timeout to repeat? If so, when do you start to see
>>> the problem?
>>>
>>> Thanks,
>>> Rusty
>>>
>>>
>>> On Wed, Feb 10, 2010 at 11:30 AM, Zubair Quraishi <zubairq at gmail.com>wrote:
>>>
>>>> Hi, I am using Riak 0.7.1 on both Mac OS X and Ubuntu 8 and 9. I can put
>>>> objects into Riak fine on both platforms, but when I do list_buckets( ) it
>>>> doesn't work on Mac OS X (it times out). It works fine on Ubuntu. Has anyone
>>>> else experienced this?
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> riak-users at lists.basho.com
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20100210/043e06f2/attachment.html>


More information about the riak-users mailing list