Nodes not leaving in Riak 0.14.2

Samuel Elliott sam at lenary.co.uk
Wed Apr 4 13:46:42 EDT 2012


Hi Anthony,

1) Which version of riak are you running? I think there are more
messages in the mailing list about people who have had nodes get stuck
leaving (it's worth also searching for "handoff" as that's the part of
the vnode lifecycle which is the important one in a node leaving
manoeuvre).

2) I think to play it safe, it's probably best to have the nodes leave
one by one. I don't know exactly how riak keeps track of network
partitions, but I could imagine it seeing 3 nodes dropping off as some
kind of network partition rather than those 3 nodes wanting to leave.

Hopefully this will get the ball rolling with getting your issue fixed.

Sam

On Wed, Apr 4, 2012 at 8:02 AM, Anthony Molinaro
<anthonym at alumni.caltech.edu> wrote:
> Ping
>
> On Mon, Apr 02, 2012 at 06:28:00PM -0700, Anthony Molinaro wrote:
>> Hi,
>>
>>   So I found one of my riak clusters misbehaving and found out it was
>> running on some inferior hardware.  After getting some new machines from
>> my ops group.  I went ahead and added several nodes.  Everything went
>> relatively smoothly until I tried to have the old nodes leave the ring.
>>
>> I need to remove 4 nodes, and I assumed since I had been successful
>> adding a few nodes at once, I would also be able to have multiple nodes
>> leave.
>>
>> Unfortunately, that does not appear to be the case.  One node (the first
>> removed) completely left, the others did not.  Then no matter how many
>> times I did a 'riak-admin leave' they would not leave.  I re-added the
>> one that had left successfully to see if a ring change would unstick
>> something.  It does appear to, as even with all the original nodes back
>> the 3 I attempted to have leave are still not leaving.
>>
>> I'm going through now and just doing a rolling restart hoping that
>> might fix things, but I'm thinking I might be stuck.
>>
>> Is there anyway to remove these nodes?
>>
>> Thanks,
>>
>> -Anthony
>>
>> --
>> ------------------------------------------------------------------------
>> Anthony Molinaro                           <anthonym at alumni.caltech.edu>
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro                           <anthonym at alumni.caltech.edu>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



-- 
Samuel Elliott
sam at lenary.co.uk
http://lenary.co.uk/
+44 (0)7891 993 664




More information about the riak-users mailing list