Did a force-remove of two nodes, now system is unresponsive

marcel.koopman marcel.koopman at gmail.com
Tue Aug 19 04:38:03 EDT 2014


Hi,

The configured ring size is 64 (default i believe).
Is this something we can change now?
Also still riak is doing transfers, making some of the nodes unresponsive.
We've limited the transfers to 1.
But how do we handle this?

Kind regards,
Marcel


2014-08-19 8:58 GMT+02:00 Ciprian Manea [via Riak Users] <
ml-node+s197444n4031634h21 at n3.nabble.com>:

> Hi Marcel,
>
> What is the configured ring size for this cluster?
>
> You can slow down the transfers by running $ riak-admin transfer-limit 1
> in one of your riak nodes. iowait should decrease as well once
> transfer-limit is lowered, unless one of your disks is failing or is about
> to fail.
>
>
> Regards,
> Ciprian
>
>
> On Mon, Aug 18, 2014 at 9:18 PM, marcel.koopman <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=4031634&i=0>> wrote:
>
>> We have a 5 node riak cluster.
>> Two nodes in this cluster, had to be removed because they are no longer
>> available (since a half year).
>> So a force remove was done.
>>
>> After this, the 3 remaining nodes began to transfer all data. So we ended
>> up
>> with a complete unresponsive system. The iowait is blocking us now.
>> So we are hoping that this will settle today, the next transaction was
>> actually adding two new nodes.
>>
>> And yes this is production, Is there any chance we lost data?
>>
>>
>>
>> --
>> View this message in context:
>> http://riak-users.197444.n3.nabble.com/Did-a-force-remove-of-two-nodes-now-system-is-unresponsive-tp4031603.html
>> Sent from the Riak Users mailing list archive at Nabble.com.
>>
>> _______________________________________________
>> riak-users mailing list
>> [hidden email] <http://user/SendEmail.jtp?type=node&node=4031634&i=1>
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email] <http://user/SendEmail.jtp?type=node&node=4031634&i=2>
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://riak-users.197444.n3.nabble.com/Did-a-force-remove-of-two-nodes-now-system-is-unresponsive-tp4031603p4031634.html
>  To unsubscribe from Did a force-remove of two nodes, now system is
> unresponsive, click here
> <http://riak-users.197444.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4031603&code=bWFyY2VsLmtvb3BtYW5AZ21haWwuY29tfDQwMzE2MDN8LTE3MDk3MTY2NDg=>
> .
> NAML
> <http://riak-users.197444.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>



-- 
Met vriendelijke groet,

Marcel Koopman




--
View this message in context: http://riak-users.197444.n3.nabble.com/Did-a-force-remove-of-two-nodes-now-system-is-unresponsive-tp4031603p4031635.html
Sent from the Riak Users mailing list archive at Nabble.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140819/accc3498/attachment.html>


More information about the riak-users mailing list