Listing buckets causes nodes to misbehave

Armon Dadgar armon.dadgar at gmail.com
Fri Mar 16 15:48:08 EDT 2012


Interesting, so it's not just us. Based on the behavior,
I'm guessing that listing the buckets requires traversing the
key space?


> Message: 2
> Date: Thu, 15 Mar 2012 11:19:15 -0700
> From: Aphyr <aphyr at aphyr.com>
> To: riak-users at lists.basho.com
> Subject: Re: Listing buckets causes nodes to misbehave
> Message-ID: <4F6232A3.5050006 at aphyr.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
> 
> Yup, list-keys and list-buckets does this for us too, since Riak 0.14. 
> Bitcask, 6 nodes, physical hardware, 1024 partitions, 100-300 million 
> keys with n_val 3.
> 
> --Kyle
> 
> On 03/15/2012 11:17 AM, Armon Dadgar wrote:
>> We are currently running Riak 1.1 in production, using LevelDB
>> with snappy compression and yesterday we ran into a strange
>> issue where listing the buckets via:
>> 
>> $ curl "http://east-riak-001:8098/riak?buckets=true"
>> 
>> Doing this caused curl to hang indefinitely, and the target node to
>> freak out.
>> Symptoms included a massive increase in system load (2-3x), FSM timings
>> exploding
>> from ~30-50 msec to 500 - 5000 msec, and most of the APIs becoming
>> totally unresponsive.
>> 
>> We ran this on a few nodes on the cluster, which basically brought
>> everything to a halt. Our solution was to just reboot all the nodes, at
>> which point things returned to normal.
>> 
>> Curious if anybody else has experienced this.
>> 
>> Best Regards,
>> 
>> Armon Dadgar
>> 
>> 
>> 
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





More information about the riak-users mailing list