get and put operations are slow on Riak KV cluster

Luke Bakken lbakken at basho.com
Tue Jun 14 13:53:45 EDT 2016


Hello Abhinav,

Have you followed all of our suggested tuning in this document?

http://docs.basho.com/riak/kv/2.1.4/using/performance/

Specifically, these settings may help a lot in your environment:

http://docs.basho.com/riak/kv/2.1.4/using/performance/#optional-i-o-settings


--
Luke Bakken
Engineer
lbakken at basho.com


On Mon, Jun 13, 2016 at 12:00 AM, Abhinav Tripathi
<abhi.160382 at gmail.com> wrote:
> Hi,
> I am seeing quite slow get and put operations on our Riak KV cluster. We are
> new to Riak and have tried to follow the instructions in the documentation.
> It would be great if you could take a look and let us know what are we doing
> wrong.
>
> We are using Riak as a distributed cache for our application. The 95th
> percentile (and above) get operations are >10ms. Similarly, 99th and above
> percentile for put operations is getting close to 10ms. Ideally, we want the
> 99% gets to stay around 1ms and 99% puts to stay <5ms.
>
> I have attached the configurations for all 3 nodes in our cluster. Also,
> attached are recent console.log files. I have attached a recent one-minute
> stats file as well from which the below values are.
>
> GETs
>     "node_get_fsm_time_100": 71429,
>     "node_get_fsm_time_95": 16551,
>     "node_get_fsm_time_99": 27513,
>     "node_get_fsm_time_mean": 4393,
>     "node_get_fsm_time_median": 989
>
> PUTs
>     "node_put_fsm_time_100": 10785,
>     "node_put_fsm_time_95": 2679,
>     "node_put_fsm_time_99": 7800,
>     "node_put_fsm_time_mean": 1752,
>     "node_put_fsm_time_median": 1608
>
> I have recorded many one-minute stats for our cluster. Most of them for the
> past couple of days show high values like these.
>
> All three Riak nodes are 4-core, 16GB RAM, 128 GB SSD machines ... basically
> m3.xlarge AWS instances.
>
> Also, I can see many such lines in the console.log,
>
> 2016-06-13 00:03:32.545 [info] <0.23701.3269> Merged
> {["/apps/riak/lib/bitcask/502391187832497878132516661246222288006726811648/75.bitcask.data","/apps/riak/lib/bitcask/502391187832497878132516661246222288006726811648/76.bitcask.data","/apps/riak/lib/bitcask/502391187832497878132516661246222288006726811648/77.bitcask.data","/apps/riak/lib/bitcask/502391187832497878132516661246222288006726811648/78.bitcask.data"],[]}
> in 596.530484 seconds.
>
> These merge operations are usually taking close to 600 seconds. Could that
> be a problem as well?
>
> Thanks,
> Abhinav.
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>




More information about the riak-users mailing list