Listing buckets causes nodes to misbehave

Jared Morrow jared at basho.com
Fri Mar 16 16:30:17 EDT 2012


Of course, I miss the obvious and a kind colleague pointed me to the exact same note on the more relevant page (list buckets) http://wiki.basho.com/HTTP-List-Buckets.html

-Jared


On Mar 16, 2012, at 1:48 PM, Armon Dadgar wrote:

> Interesting, so it's not just us. Based on the behavior,
> I'm guessing that listing the buckets requires traversing the
> key space?
> 
> 
>> Message: 2
>> Date: Thu, 15 Mar 2012 11:19:15 -0700
>> From: Aphyr <aphyr at aphyr.com>
>> To: riak-users at lists.basho.com
>> Subject: Re: Listing buckets causes nodes to misbehave
>> Message-ID: <4F6232A3.5050006 at aphyr.com>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>> 
>> Yup, list-keys and list-buckets does this for us too, since Riak 0.14. 
>> Bitcask, 6 nodes, physical hardware, 1024 partitions, 100-300 million 
>> keys with n_val 3.
>> 
>> --Kyle
>> 
>> On 03/15/2012 11:17 AM, Armon Dadgar wrote:
>>> We are currently running Riak 1.1 in production, using LevelDB
>>> with snappy compression and yesterday we ran into a strange
>>> issue where listing the buckets via:
>>> 
>>> $ curl "http://east-riak-001:8098/riak?buckets=true"
>>> 
>>> Doing this caused curl to hang indefinitely, and the target node to
>>> freak out.
>>> Symptoms included a massive increase in system load (2-3x), FSM timings
>>> exploding
>>> from ~30-50 msec to 500 - 5000 msec, and most of the APIs becoming
>>> totally unresponsive.
>>> 
>>> We ran this on a few nodes on the cluster, which basically brought
>>> everything to a halt. Our solution was to just reboot all the nodes, at
>>> which point things returned to normal.
>>> 
>>> Curious if anybody else has experienced this.
>>> 
>>> Best Regards,
>>> 
>>> Armon Dadgar
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> 
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120316/c7384c05/attachment.html>


More information about the riak-users mailing list