Issues with high node load and very slow response

Bryan Hunt bhunt at basho.com
Sun Jul 27 05:12:09 EDT 2014


Hello Chaim,

How big is the object you are trying to index? Are siblings (allow_mult)
enabled? What type of object is it? CRDT, binary blob, text? Can you paste
the output from riak-admin member-status and ring-status. Also riak status.
Thanks,

Bryan
On 27 Jul 2014 04:55, "Chaim Solomon" <chaim at itcentralstation.com> wrote:

> Hi,
>
> I've been having issues for some days with the cluster I am running (or
> now - trying to run). The last major operation was going through all items
> of a bucket and doing a read/update operation on them from Python code -
> nothing fancy, just modifying some data.
>
> I got issues with one node being VERY busy - 2 CPUs fully loaded - and the
> requests were timing out or at least VERY slow.
> I took the node out of the cluster and added another node - then after
> clearing the node brought it back. After some days of waiting for AAE and
> the migration of data to finish, I am still having one node at 2 CPUs at
> 100% and one node with one CPU at 100% and the requests are still VERY
> slow. I am getting a lot of these in the console.log:
>
> 2014-07-26 23:37:33.459 [error] <0.7427.266>@yz_kv:index:206 failed to
> index object {<<"linkmeta">>,<<"...">>} with error {"Failed to index
> docs",{error,req_timedout}} because
> [{yz_solr,index,3,[{file,"src/yz_solr.erl"},{line,192}]},{yz_kv,index,7,[{file,"src/yz_kv.erl"},{line,258}]},{yz_kv,index,3,[{file,"src/yz_kv.erl"},{line,193}]},{riak_kv_vnode,actual_put,6,[{file,"src/riak_kv_vnode.erl"},{line,1416}]},{riak_kv_vnode,perform_put,3,[{file,"src/riak_kv_vnode.erl"},{line,1404}]},{riak_kv_vnode,do_put,7,[{file,"src/riak_kv_vnode.erl"},{line,1199}]},{riak_kv_vnode,handle_command,3,[{file,"src/riak_kv_vnode.erl"},{line,485}]},{riak_core_vnode,vnode_command,3,[{file,"src/riak_core_vnode.erl"},{line,345}]}]
>
> And on almost all other nodex I am getting a lot of these:
> 2014-07-26 23:50:58.796 [info] <0.12268.221>@yz_kv:should_handoff:157
> waiting for bucket types prefix and indexes to agree between '
> riak at 10.128.138.25' and 'riak at 10.128.137.185'
>
> How can I get the cluster to work again?
>
> It is a 2.0.0rc1 cluster.
>
> Chaim Solomon
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140727/d95eac74/attachment.html>


More information about the riak-users mailing list