high 99.9% latencies with leveldb backend

Matthew Von-Maszewski matthewv at basho.com
Tue Oct 30 15:44:29 EDT 2012


D,

Go here:  http://basho.com/blog/technical/2012/10/30/leveldb-in-riak-1p2/

leveldb has code that will explicitly stall write operations if it gets too far behind in compaction.  3 to 4 seconds is nothing.  1.1 can stall into the minutes range if you push it hard enough.

You either need to throttle the input rate or update to 1.2.1.  That is the best hope I can offer.

Matthew



On Oct 30, 2012, at 3:31 PM, Dietrich Featherston wrote:

> Seeing 99th percentile put latencies at around 30-40 ms with 99.9th percentile jumping all the way up to 3-4s. This is riak 1.1 with the eleveldb backend on a 9-node cluster, N = 2, W = 1. Lots of free iops, but CPU is consistently burning 30-40% across all 8 cores.
> 
> Wondering if this could be caused by leveldb compactions which, to my knowledge, will wedge the scheduler for the duration of the compaction in riak 1.1.
> 
> Any other thoughts for troubleshooting? Not a platform or erlang version with dtrace support so that's out.
> 
> Thanks,
> D
> 
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20121030/a27de313/attachment.html>


More information about the riak-users mailing list