Interesting sawtooth increasing CPU usage on lightly-used Riak cluster on EC2 micro instances. Is this expected?

Mark Phillips mark at basho.com
Fri Jun 3 17:10:55 EDT 2011


Hi Eamonn,

Without having the chance to think too hard about this, here is what
might be going on:

There was an issue with 0.14.0 related to the "/stats" resource
leading to increased CPU usage over time.[1] This is fixed in the
latest release of version.

Couple that with the fact that you're running EC2 Micros on unbuntu
10.04, and that might just explain the increased CPU usage.

Your best bet is to probably upgrade to 0.14.2 (if you haven't
already) and see if that doesn't mollify or even eliminate the issue.

Hope that helps.

Mark

1 - https://issues.basho.com/show_bug.cgi?id=971


On Mon, May 30, 2011 at 10:52 AM, Eamonn <eamonn.obrien-strain at hp.com> wrote:
> I have a cluster of six Riak nodes that has been operating for a few months
> on Amazon EC2.   Because this is a development deployment with very light
> usage currently, I have used cheap "micro" instances.
>
> I am using  riak_0.14.0-1_amd64.deb with no changes to the default
> app.config except to modify the IP addresses.
>
> If you look at the graph of CPU usage for these instances over the last two
> weeks
>  http://eamonn.org/riak/riak-cluster-cpu.png
> you see an interesting pattern.
>
> Each node gradually increases its CPU over about two days and then suddenly
> drops down slightly, forming a saw-tooth pattern.  From an initial low
> average CPU several months ago, the CPU usage has now slowly risen.   It now
> seems to have reached an equilibrium, with three of the nodes at 50% and
> three at 60%.
>
> Most of this usage happens when the cluster is not being used externally.
>  The little spikes you see on the graph are probably the actual external
> access via the REST API.
>
> I assume this activity is caused by the continuous "gossip" between the
> nodes.  Perhaps the different equilibrium CPU percentages are related to
> which share of the data items each node has.  Is the sawtooth pattern
> showing some kind of garbage collection?
>
> The Riak cluster does seem to work correctly with reasonable latency, at
> least under low load.  (I have not yet done load-testing).
>
> Is this pattern expected?   Is it a sign of some problem with my
> configuration?  Any suggestions for how to tune the cluster to run better on
> EC2 micro instances?  Any suggestions of what metrics to use to decide when
> to dynamically scale the cluster by spinning up nodes or spinning them down?
>
> Thanks,
> __
> Eamonn O'Brien-Strain
> HP Labs
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>



More information about the riak-users mailing list