Issues with high node load and very slow response

Jordan West jwest at basho.com
Sun Jul 27 22:30:40 EDT 2014


On Sat, Jul 26, 2014 at 8:54 PM, Chaim Solomon <chaim at itcentralstation.com>
wrote:

>
> And on almost all other nodex I am getting a lot of these:
> 2014-07-26 23:50:58.796 [info] <0.12268.221>@yz_kv:should_handoff:157
> waiting for bucket types prefix and indexes to agree between '
> riak at 10.128.138.25' and 'riak at 10.128.137.185'
>
>
This error is indicating that 'riak at 10.128.138.25' is unable to handoff
data to 'riak at 10.128.137.185'. The former is delaying handoff until the
latter has met some requirements necessary to index the data sent to it.

Unfortunately, there are several conditions that could cause this. To rule
out the obvious ones can you verify that `riak at 10.128.137.185` has search
enabled. If it is the case that search is enabled we can try a few
different things to debug further.

Attaching to 'riak at 10.128.138.25' (via "riak attach") and using the shell,
if you could provide the output of:

```
rpc:multicall([node() | nodes()], app_helper, get_env, [yokozuna, enabled,
false], 5000).
```

As well as:

```
[riak_core_node_watcher:services(N) || N <- [node() | nodes()]].
```

And finally:

```
yz_kv:is_metadata_consistent('riak at 10.127.137.185').
```

Based on the output of those we can debug further.

Additionally, its entirely possible that when we get rid of that error it
will not resolve the primary issue (CPU usage/request slowness). However,
the check that is the source of that log message can block in the request
path, so something funky could certainly be going on.

Jordan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140727/e6d00d0b/attachment.html>


More information about the riak-users mailing list