Listing buckets causes nodes to misbehave
jared at basho.com
Fri Mar 16 16:30:17 EDT 2012
Of course, I miss the obvious and a kind colleague pointed me to the exact same note on the more relevant page (list buckets) http://wiki.basho.com/HTTP-List-Buckets.html
On Mar 16, 2012, at 1:48 PM, Armon Dadgar wrote:
> Interesting, so it's not just us. Based on the behavior,
> I'm guessing that listing the buckets requires traversing the
> key space?
>> Message: 2
>> Date: Thu, 15 Mar 2012 11:19:15 -0700
>> From: Aphyr <aphyr at aphyr.com>
>> To: riak-users at lists.basho.com
>> Subject: Re: Listing buckets causes nodes to misbehave
>> Message-ID: <4F6232A3.5050006 at aphyr.com>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>> Yup, list-keys and list-buckets does this for us too, since Riak 0.14.
>> Bitcask, 6 nodes, physical hardware, 1024 partitions, 100-300 million
>> keys with n_val 3.
>> On 03/15/2012 11:17 AM, Armon Dadgar wrote:
>>> We are currently running Riak 1.1 in production, using LevelDB
>>> with snappy compression and yesterday we ran into a strange
>>> issue where listing the buckets via:
>>> $ curl "http://east-riak-001:8098/riak?buckets=true"
>>> Doing this caused curl to hang indefinitely, and the target node to
>>> freak out.
>>> Symptoms included a massive increase in system load (2-3x), FSM timings
>>> from ~30-50 msec to 500 - 5000 msec, and most of the APIs becoming
>>> totally unresponsive.
>>> We ran this on a few nodes on the cluster, which basically brought
>>> everything to a halt. Our solution was to just reboot all the nodes, at
>>> which point things returned to normal.
>>> Curious if anybody else has experienced this.
>>> Best Regards,
>>> Armon Dadgar
>>> riak-users mailing list
>>> riak-users at lists.basho.com
> riak-users mailing list
> riak-users at lists.basho.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the riak-users