Riak stalls, leveldb backend

Michael Lewis lewismj at me.com
Tue Jan 6 06:01:42 EST 2015


I'd be interested in the answer to this too, isn't it inevitable that in the underlying storage you will get compaction delays, you see the same in any key value store, eg if you were pushing lots of small packets of data into HBase/Cassandra etc... Eventually the system has to perform the compaction ? Not sure how you tune it so that you lower the throughput but remove some of the compaction times.


> On 5 Jan 2015, at 17:07, Andy Pellett <andy at embed.ly> wrote:
> Hi all,
> I've been experiencing stalls where riak won't return any data (queries time out) with my riak cluster. Here are some basic details:
> - 8 nodes
> - riak 1.4.10 (upgraded from 1.4.6 -> 1.4.8 -> 1.4.10)
> - leveldb backend
> - n_val is 2
> - allow_mult is false
> - ec2 i2.2xlarge boxes (8 cores, 61gb ram, 800gb disk space)
> - about 33% disk space utilization per node
> The riak cluster will stall for as long as a few minutes at a time, but will otherwise work as expected for hours. There doesn't seem to be an obvious pattern as to when the stalls happen.
> My first thought was that the stalls may be related to AAE, but I've disabled that via 'riak attach' and the settings file. Sidenote, I still see messages like:
> 2015-01-05 12:24:04.666 [info] <0.574.0>@riak_kv_entropy_manager:perhaps_log_throttle_change:826 Changing AAE throttle from 0 -> 10 msec/key, based on maximum vnode mailbox size 209 from 'riak-user at riak-host'
> which makes me question whether AAE is actually turned off.
> Now I'm leaning towards leveldb compactions being the issue. What can I do to verify this is the issue, and how can I fix it?
> I see log messages about large objects:
> 2015-01-05 16:11:28.046 [warning] <0.6398.0>@riak_kv_vnode:encode_and_put_no_sib_check:1830 Writing very large object (11307735 bytes) to <<"BucketName">>/<<"keys_1420466400">>
> Could these be causing longer-running compactions, or more frequent compactions?
> Thanks for reading,
> Andy
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

More information about the riak-users mailing list