Listing buckets causes nodes to misbehave
armon.dadgar at gmail.com
Fri Mar 16 15:48:08 EDT 2012
Interesting, so it's not just us. Based on the behavior,
I'm guessing that listing the buckets requires traversing the
> Message: 2
> Date: Thu, 15 Mar 2012 11:19:15 -0700
> From: Aphyr <aphyr at aphyr.com>
> To: riak-users at lists.basho.com
> Subject: Re: Listing buckets causes nodes to misbehave
> Message-ID: <4F6232A3.5050006 at aphyr.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
> Yup, list-keys and list-buckets does this for us too, since Riak 0.14.
> Bitcask, 6 nodes, physical hardware, 1024 partitions, 100-300 million
> keys with n_val 3.
> On 03/15/2012 11:17 AM, Armon Dadgar wrote:
>> We are currently running Riak 1.1 in production, using LevelDB
>> with snappy compression and yesterday we ran into a strange
>> issue where listing the buckets via:
>> $ curl "http://east-riak-001:8098/riak?buckets=true"
>> Doing this caused curl to hang indefinitely, and the target node to
>> freak out.
>> Symptoms included a massive increase in system load (2-3x), FSM timings
>> from ~30-50 msec to 500 - 5000 msec, and most of the APIs becoming
>> totally unresponsive.
>> We ran this on a few nodes on the cluster, which basically brought
>> everything to a halt. Our solution was to just reboot all the nodes, at
>> which point things returned to normal.
>> Curious if anybody else has experienced this.
>> Best Regards,
>> Armon Dadgar
>> riak-users mailing list
>> riak-users at lists.basho.com
More information about the riak-users