Listing buckets causes nodes to misbehave

Armon Dadgar armon.dadgar at gmail.com
Fri Mar 16 16:36:50 EDT 2012


Got it. Definitely a case of RTFM. I know list keys would do a traversal,
but I thought listing buckets was just a metadata lookup. Good to know.

Best Regards,

Armon Dadgar

On Mar 16, 2012, at 1:30 PM, Jared Morrow wrote:

> Of course, I miss the obvious and a kind colleague pointed me to the exact same note on the more relevant page (list buckets) http://wiki.basho.com/HTTP-List-Buckets.html
> 
> -Jared
> 
> 
> On Mar 16, 2012, at 1:48 PM, Armon Dadgar wrote:
> 
>> Interesting, so it's not just us. Based on the behavior,
>> I'm guessing that listing the buckets requires traversing the
>> key space?
>> 
>> 
>>> Message: 2
>>> Date: Thu, 15 Mar 2012 11:19:15 -0700
>>> From: Aphyr <aphyr at aphyr.com>
>>> To: riak-users at lists.basho.com
>>> Subject: Re: Listing buckets causes nodes to misbehave
>>> Message-ID: <4F6232A3.5050006 at aphyr.com>
>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>> 
>>> Yup, list-keys and list-buckets does this for us too, since Riak 0.14. 
>>> Bitcask, 6 nodes, physical hardware, 1024 partitions, 100-300 million 
>>> keys with n_val 3.
>>> 
>>> --Kyle
>>> 
>>> On 03/15/2012 11:17 AM, Armon Dadgar wrote:
>>>> We are currently running Riak 1.1 in production, using LevelDB
>>>> with snappy compression and yesterday we ran into a strange
>>>> issue where listing the buckets via:
>>>> 
>>>> $ curl "http://east-riak-001:8098/riak?buckets=true"
>>>> 
>>>> Doing this caused curl to hang indefinitely, and the target node to
>>>> freak out.
>>>> Symptoms included a massive increase in system load (2-3x), FSM timings
>>>> exploding
>>>> from ~30-50 msec to 500 - 5000 msec, and most of the APIs becoming
>>>> totally unresponsive.
>>>> 
>>>> We ran this on a few nodes on the cluster, which basically brought
>>>> everything to a halt. Our solution was to just reboot all the nodes, at
>>>> which point things returned to normal.
>>>> 
>>>> Curious if anybody else has experienced this.
>>>> 
>>>> Best Regards,
>>>> 
>>>> Armon Dadgar
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> riak-users at lists.basho.com
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> 
>> 
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120316/5ae8d4de/attachment.html>


More information about the riak-users mailing list