Did a force-remove of two nodes, now system is unresponsive

Ciprian Manea ciprian at basho.com
Tue Aug 19 02:57:43 EDT 2014


Hi Marcel,

What is the configured ring size for this cluster?

You can slow down the transfers by running $ riak-admin transfer-limit 1 in
one of your riak nodes. iowait should decrease as well once transfer-limit
is lowered, unless one of your disks is failing or is about to fail.


Regards,
Ciprian


On Mon, Aug 18, 2014 at 9:18 PM, marcel.koopman <marcel.koopman at gmail.com>
wrote:

> We have a 5 node riak cluster.
> Two nodes in this cluster, had to be removed because they are no longer
> available (since a half year).
> So a force remove was done.
>
> After this, the 3 remaining nodes began to transfer all data. So we ended
> up
> with a complete unresponsive system. The iowait is blocking us now.
> So we are hoping that this will settle today, the next transaction was
> actually adding two new nodes.
>
> And yes this is production, Is there any chance we lost data?
>
>
>
> --
> View this message in context:
> http://riak-users.197444.n3.nabble.com/Did-a-force-remove-of-two-nodes-now-system-is-unresponsive-tp4031603.html
> Sent from the Riak Users mailing list archive at Nabble.com.
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140819/3515d3be/attachment.html>


More information about the riak-users mailing list