Node will not leave cluster

David Greenstein dave at kibits.com
Thu Jun 7 13:21:19 EDT 2012


As it turns out, this is a known issue with 1.0.x. It is documented with a workaround in the release note named riak-1.0.org under "Ownership Handoff Stall".

Dave

____
David Greenstein : Founder & CTO : Kibits Labs : dave at kibits.com : 617-901-9473 : www.kibits.com
Get the iPhone or Android app now:  http://www.kibits.com/app



On Jun 6, 2012, at 12:50 PM, David Greenstein wrote:

> 
> Thanks Austin.
> 
> Interesting… after running the ring_status command it seems there are several handoffs that are either hung or not completing. The database is fairly small and running the ring_status command over several minutes reveals the same output.
> 
> The member_status is 25% for ring and pending for the node leaving.
> 
> So, I think something is wrong with the handoff, but, as far as I can tell, there's no indication as to what is wrong in the logs or in the output of these commands.
> 
> ============================== Ownership Handoff ==============================
> Owner:      riak at 10.0.1.14
> Next Owner: riak at 10.0.1.15
> 
> Index: 479555224749202520035584085735030365824602865664
>  Waiting on: []
>  Complete:   [riak_kv_vnode,riak_pipe_vnode]
> 
> Index: 1027618338748291114361965898003636498195577569280
>  Waiting on: []
>  Complete:   [riak_kv_vnode,riak_pipe_vnode]
> 
> Index: 1301649895747835411525156804137939564381064921088
>  Waiting on: []
>  Complete:   [riak_kv_vnode,riak_pipe_vnode]
> 
> -------------------------------------------------------------------------------
> Owner:      riak at 10.0.1.15
> Next Owner: riak at 10.0.1.14
> 
> Index: 913438523331814323877303020447676887284957839360
>  Waiting on: []
>  Complete:   [riak_kv_vnode,riak_pipe_vnode]
> 
> -------------------------------------------------------------------------------
> Owner:      riak at 10.0.1.16
> Next Owner: riak at 10.0.1.15
> 
> Index: 936274486415109681974235595958868809467081785344
>  Waiting on: []
>  Complete:   [riak_kv_vnode,riak_pipe_vnode]
> 
> -------------------------------------------------------------------------------
> 
> 
> ============================== Unreachable Nodes ==============================
> All nodes are up and reachable
> 
> [root at ip-10-0-1-20 ec2-user]# /db/riak/bin/riak-admin member_status
> ================================= Membership ==================================
> Status     Ring    Pending    Node
> -------------------------------------------------------------------------------
> leaving    25.0%     25.0%    'riak at 10.0.1.20'
> valid      28.1%     25.0%    'riak at 10.0.1.14'
> valid      20.3%     25.0%    'riak at 10.0.1.15'
> valid      26.6%     25.0%    'riak at 10.0.1.16'
> -------------------------------------------------------------------------------
> Valid:3 / Leaving:1 / Exiting:0 / Joining:0 / Down:0
> 
> 
> 
> 
> 
> Any other clues would be appreciated. Thank you!
> 
> Dave
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20120607/e6518cbe/attachment.html>


More information about the riak-users mailing list