High CPU on a single node in production

Josh Yudaken josh at smyte.com
Thu Jan 7 21:06:50 EST 2016


Hi Fred,

That makes a lot of sense. Yes the server was under high indexing load
as the handoffs have all caused re-indexing. I'm going to drop
transfer-limit to 1 [from 2] across the board which will hopefully
reduce further issues.

At the moment our cluster wants to perform a ton of handoffs, almost
moving every single vnode. See cluster status/handoff summary. I
attempted to reduce this but couldn't get the `cluster plan` lower so
eventually just resorted to committing it and letting riak slowly
churn through the transfers. I'll set up statistics on the vnode queue
lengths and see if it reveals anything interesting.

At this point in time we're primarily using Riak as a search engine,
and it doesn't feel production ready. Is there any time frame on being
" a better solr citizen"? We originally chose Riak over ElasticSearch
as we needed a key-value database and the "search for free" aspect was
great.. but it's becoming far more problematic than we believe simply
running two databases would be.

If you notice `riak33` in status/summary is down. Bringing it back up
causes the issues described, and ends up taking down our site within
five hours.

Regards,
Josh

===
Cluster status:
+------------------------------------------+------+-------+-----+-------+
|                   node                   |status| avail |ring |pending|
+------------------------------------------+------+-------+-----+-------+
| (C) riak at riak25-2.c.authbox-api.internal |valid |  up   |  8.3|  10.7 |
|     riak at riak26-2.c.authbox-api.internal |valid |  up   |  8.6|   9.3 |
|     riak at riak27-2.c.authbox-api.internal |valid |  up   | 10.4|  11.2 |
|     riak at riak28-2.c.authbox-api.internal |valid |  up   | 10.1|  10.0 |
|     riak at riak29-2.c.authbox-api.internal |valid |  up   | 11.3|  10.0 |
|     riak at riak30-2.c.authbox-api.internal |valid |  up   | 11.7|  10.1 |
|     riak at riak31-2.c.authbox-api.internal |valid |  up   | 11.5|   9.9 |
|     riak at riak32-2.c.authbox-api.internal |valid |  up   | 11.2|   9.8 |
|      riak at riak33.c.authbox-api.internal  |valid | down! | 10.1|  10.1 |
|      riak at riak35.c.authbox-api.internal  |valid |  up   |  6.8|   9.1 |
+------------------------------------------+------+-------+-----+-------+
===
Handoff summary:
+----------------------------------------+-----+---------+------+------+------+
|                  Node                  |Total|Ownership|Resize|Hinted|Repair|
+----------------------------------------+-----+---------+------+------+------+
|  riak at riak25-2.c.authbox-api.internal  |  0  | 0 (95)  |      |      |      |
|  riak at riak26-2.c.authbox-api.internal  |  1  | 1 (111) |      |      |      |
|  riak at riak27-2.c.authbox-api.internal  |  0  | 0 (118) |      |      |      |
|  riak at riak28-2.c.authbox-api.internal  |  0  | 0 (119) |      |      |      |
|  riak at riak29-2.c.authbox-api.internal  |  0  | 0 (122) |      |      |      |
|  riak at riak30-2.c.authbox-api.internal  |  1  | 1 (131) |      |      |      |
|  riak at riak31-2.c.authbox-api.internal  |  0  | 0 (138) |      |      |      |
|  riak at riak32-2.c.authbox-api.internal  |  0  | 0 (133) |      |      |      |
|   riak at riak35.c.authbox-api.internal   |  0  | 0 (97)  |      |      |      |
+----------------------------------------+-----+---------+------+------+------+

(unreachable): riak at riak33.c.authbox-api.internal
===



> Hi Josh,
>
> Sorry for not getting back sooner.
>
> I am not entirely sure what is going on with your handoffs.  It could be that you have overloaded Solr with handoff activity, and that is causing vnodes to become unresponsive.  We are actively working on a fix for this, which allows vnodes to continue their work, even if Solr is taking its time with ingest.  This also includes batching, with aggregates insertion (and deletion) operations into Solr, to smooth out some of the bumps, while at the same time being a better Solr citizen.
>
> The hundreds of entries you see on close trees seem to be due to the fact that your YZ AAE trees are in need of being rebuilt.  Can it be that you have hit the magic 7 day grace period on AAE tree expiry?  The index failures you see in the logs seem to be because the yz_entropy_mgr has been shut down.  You are seeing this during a riak stop, correct?  Is the system under high indexing load, at the time?  That could account for the log messages, as index operations may be coming in while yokozuna is being shut down.
>
> Regarding ring resize, please have a look at https://github.com/basho/yokozuna/issues/279 <https://github.com/basho/yokozuna/issues/279>.  I do not believe these issues have been rectified, so the official line is what you see in the documentation.  You can, of course, reindex your data after a ring resize, but that is not acceptable in a production scenario, if you have an SLA around search availability.
>
> Hope that helps, and let us know if you have any more information about what might be consuming CPU on your nodes.  I would keep a close eye on the vnode queue lengths in the Riak stats (riak_kv_vnodeq_(min|max|avg|mean|etc)).  If your vnode queues start getting deep, then vnodes are likely being blocked by Solr.
>
> -Fred

On Wed, Jan 6, 2016 at 2:11 PM, Luke Bakken <lbakken at basho.com> wrote:
>> We're planning on having a rather large cluster rather soon, which was
>> the reason for the large ring size. Your documentation indicates ring
>> resize is *not* possible with search 2.0 [1], although an issue I
>> found on github indicated it might be now? [2]
>
> Yeah, resize not applicable in your situation. Are you planning on
> having 20 or more nodes eventually?
>
>> I've been through the tuning list multiple times, and haven't seen any
>> changes
>
> So I'll assume the tunings have all been applied.
>
>> I migrated the machine seeing issues to a new host, and now
>> the new host is seeing similar problems. Heres a screenshot of `htop`
>> just before I stopped the node in order to bring our site back up [3].
>
> Just so I'm clear, that "htop" output (https://goo.gl/k8Wtcw) is from
> one node, correct? There shouldn't be more than one beam.smp process
> running. Is this a per-thread view? Doesn't seem to be, with all the
> different PIDs listed, but I've only ever used top or the varios "sar"
> utilites.
>
> Thanks,
> Luke




More information about the riak-users mailing list