sbenton at contractor.basho.com
Thu May 31 14:24:13 EDT 2012
Apologies if this is the wrong place for this, but I just updated the
eLevelDB wiki page to mention randomization of the write buffer length (via
setting write_buffer_size_min and write_buffer_size_max). Before there was
no mention of these config parameters. Perhaps people were just using
levelDB's 4MB default buffer size, causing all the vnodes to compact at the
same time? Or are there default write_buffer_size_min and
parameters under the hood?
P.S. Mathew V is getting back to me shortly on changes to this page due to
changes in 1.2.
On Thu, May 31, 2012 at 9:26 AM, Nam Nguyen <nam at tinyco.com> wrote:
> Hi Sean,
> You are right. At first I thought it was localized to that one particular
> node. Now others are also exhibiting the same symptom.
> I am putting in another node.
> On May 30, 2012, at 11:23 PM, Sean Cribbs wrote:
> The LevelDB storage backend has a known issue where compaction can stall a
> heavily-loaded node for a long time (we've seen 60 seconds or more in
> production clusters). We're very sorry about this, but an improvement will
> be available in the next release. In the meantime, DO NOT make the node
> leave the cluster - this will only make things worse! It might be worth
> adding another node to the cluster, but I suggest you wait until the node
> finishes compaction.
> On Wed, May 30, 2012 at 10:43 PM, Nam Nguyen <nam at tinyco.com> wrote:
>> My 5-node cluster exhibits a strange spike on one particular node.
>> Overall, the mean get time is about 1ms. This node occasionally shoots up
>> to 40ms.
>> During those times, %iowait is still the same as it is before the spike.
>> No error. Console log shows many lines like the below, which I don't think
>> relevant to the spike.
>> 2012-05-30 21:29:50.591 [info]
>> <0.72.0>@riak_core_sysmon_handler:handle_event:85 monitor long_gc <0.938.0>
>> The cluster is set up uniformly. Ubuntu 64bit, m2.2xlarge instance. Riak
>> 1.1.2 with LevelDB backend.
>> What would be the best course of actions for me?
>> I plan to:
>> - riak-admin leave on that node
>> - set up new instance
>> - riak-admin reip the new instance
>> - riak-admin join it to the cluster
>> riak-users mailing list
>> riak-users at lists.basho.com
> Sean Cribbs <sean at basho.com>
> Software Engineer
> Basho Technologies, Inc.
> riak-users mailing list
> riak-users at lists.basho.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the riak-users