Node will not leave cluster

David Greenstein dave at kibits.com
Wed Jun 6 12:50:53 EDT 2012


Thanks Austin.

Interesting… after running the ring_status command it seems there are several handoffs that are either hung or not completing. The database is fairly small and running the ring_status command over several minutes reveals the same output.

The member_status is 25% for ring and pending for the node leaving.

So, I think something is wrong with the handoff, but, as far as I can tell, there's no indication as to what is wrong in the logs or in the output of these commands.

============================== Ownership Handoff ==============================
Owner:      riak at 10.0.1.14
Next Owner: riak at 10.0.1.15

Index: 479555224749202520035584085735030365824602865664
  Waiting on: []
  Complete:   [riak_kv_vnode,riak_pipe_vnode]

Index: 1027618338748291114361965898003636498195577569280
  Waiting on: []
  Complete:   [riak_kv_vnode,riak_pipe_vnode]

Index: 1301649895747835411525156804137939564381064921088
  Waiting on: []
  Complete:   [riak_kv_vnode,riak_pipe_vnode]

-------------------------------------------------------------------------------
Owner:      riak at 10.0.1.15
Next Owner: riak at 10.0.1.14

Index: 913438523331814323877303020447676887284957839360
  Waiting on: []
  Complete:   [riak_kv_vnode,riak_pipe_vnode]

-------------------------------------------------------------------------------
Owner:      riak at 10.0.1.16
Next Owner: riak at 10.0.1.15

Index: 936274486415109681974235595958868809467081785344
  Waiting on: []
  Complete:   [riak_kv_vnode,riak_pipe_vnode]

-------------------------------------------------------------------------------


============================== Unreachable Nodes ==============================
All nodes are up and reachable

[root at ip-10-0-1-20 ec2-user]# /db/riak/bin/riak-admin member_status
================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
leaving    25.0%     25.0%    'riak at 10.0.1.20'
valid      28.1%     25.0%    'riak at 10.0.1.14'
valid      20.3%     25.0%    'riak at 10.0.1.15'
valid      26.6%     25.0%    'riak at 10.0.1.16'
-------------------------------------------------------------------------------
Valid:3 / Leaving:1 / Exiting:0 / Joining:0 / Down:0





Any other clues would be appreciated. Thank you!

Dave
 




More information about the riak-users mailing list