Can't handoff 1 partition, errors

fenix.serega at gmail.com fenix.serega at gmail.com
Thu Jul 4 09:26:20 EDT 2013


Hi

Cluster 5 nodes, 1.3.1

Can't handofff 1 hinted parttion -
17126972312471518572699431633393941636592959488

root at de5:/mnt/operational/riak# /opt/riak/bin/riak-admin transfers
'riak at de5' waiting to handoff 1 partitions
'riak at de2' waiting to handoff 1 partitions
'riak at de1' waiting to handoff 1 partitions

Active Transfers:

transfer type: hinted_handoff
vnode type: riak_kv_vnode
partition: 17126972312471518572699431633393941636592959488
started: 2013-07-04 13:12:31 [952.22 ms ago]
last update: no updates seen

                         unknown
riak at de1.nntp.ge =======================> riak at de4.nntp.ge
                         unknown



Errors:

013-07-04 15:14:32.252 [info]
<0.2761.4>@riak_core_handoff_sender:start_fold:130 Starting hinted_handoff
transfer of riak_kv_vnode from 'riak at de1'
17126972312471518572699431633393941636592959488 to 'riak at de4'
17126972312471518572699431633393941636592959488

2013-07-04 15:15:32.452 [error]
<0.2761.4>@riak_core_handoff_sender:start_fold:219 hinted_handoff transfer
of riak_kv_vnode from 'riak at de1'
17126972312471518572699431633393941636592959488 to 'riak at de4'
17126972312471518572699431633393941636592959488 failed because of closed

2013-07-04 15:15:32.499 [error]
<0.160.0>@riak_core_handoff_manager:handle_info:282 An outbound handoff of
partition riak_kv_vnode 17126972312471518572699431633393941636592959488 was
terminated for reason: {shutdown,{error,closed}}


2013-07-04 15:10:51.578 [info]
<0.10780.1>@riak_core_handoff_receiver:process_message:99 Receiving handoff
data for partition
riak_kv_vnode:17126972312471518572699431633393941636592959488
2013-07-04 15:11:25.389 [info]
<0.11061.1>@riak_core_handoff_receiver:process_message:99 Receiving handoff
data for partition
riak_kv_vnode:17126972312471518572699431633393941636592959488
2013-07-04 15:11:51.581 [error]
<0.10780.1>@riak_core_handoff_receiver:handle_info:80 Handoff receiver for
partition 17126972312471518572699431633393941636592959488 exited abnormally
after processing 0 objects:
{timeout,{gen_fsm,sync_send_all_state_event,[<0.1291.0>,{handoff_data,<<141,87,121,56,212,219,27,31,198,58,34,107,106,8,89,210,54,162,76,217,34,132,136,36,235,88,134,239,48,51,102,48,24,227,26,84,68,174,44,41,74,178,148,107,228,86,246,125,29,220,146,48,246,68,215,218,165,25,219,216,178,164,108,253,190,186,61,191,219,115,255,249,253,206,121,158,115,222,243,158,243,121,191,239,121,207,231,156,231,253,194,184,72,33,152,0,81,69,85,85,85,44,0,156,82,67,32,79,99,145,8,53,156,186,38,66,29,131,81,69,156,69,34,207,104,170,105,170,169,35,213,206,192,89,251,35,61,121,61,32,124,100,87,95,12,17,235,78,241,129,64,32,223,241,187,194,255,101,194,27,92,200,225,9,245,128,240,147,93,221,125,73,20,44,137,226,201,239,1,225,242,32,184,83,0,30,64,24,172,124,192,21,64,213,83,152,248,175,226,201,177,171,251,142,255,222,236,126,146,219,156,64,242,10,248,239,146,93,61,231,110,35,186,59,185,231,135,125,4,37,216,15,244,11,240,3,171,55,64,0,220,65,153,2,246,190,0,9,56,9,80,1,4,216,135,0,152,239,104,201,93,160,128,3,226,42,1,240,66,216,217,0,120,0,9,224,192,89,13,176,247,7,44,128,32,192,6,180,17,8,88,3,24,192,18,196,170,130,146,33,128,5,71,84,128,240,147,119,208,239,222,17,72,30,88,170,39,231,174,12,243,243,13,160,96,61,92,9,36,10,198,42,162,47,229,135,54,192,207,147,64,162,126,215,66,138,151,69,136,255,222,160,248,15,87,204,129,0,10,194,194,215,131,128,35,96,61,60,161,24,208,184,51,6,178,103,249,41,6,34,112,208,229,39,20,231,207,27,176,192,82,0,79,78,47,136,244,79,99,132,225,143,168,24,145,220,65,123,36,188,23,132,11,31,66,240,35,18,65,36,12,42,27,199,7,249,94,160,157,206,62,190,82,53,34,59,79,92,75,36,34,126,139,210,143,145,110,56,91,185,152,22,39,32,238,83,20,93,39,221,243,89,170,134,57,68,244,105,83,110,243,79,114,117,53,43,169,207,190,165,162,24,106,199,231,38,149,68,223,60,177,158,154,18,205,236,152,253,82,183,215,62,165,115,147,238,112,121,170,236,126,226,254,21,171,251,248,121,29,11,252,148,114,182,235,135,237,212,26,171,50,231,40,20,44,101,5,165,123,160,115,50,123,241,178,171,244,183,133,121,189,27,212,245,162,204,174,180,193,103,127,110,173,178,211,27,183,89,209,184,215,135,239,163,26,182,87,113,90,233,230,99,13,161,195,91,95,22,207,222,217,31,215,198,156,107,177,58,93,235,80,80,248,77,35,117,100,108,44,225,189,73,243,140,247,72,87,209,0,239,101,88,235,70,74,114,100,11,33,217,152,45,160,250,121,1,83,235,93,59,131,37,145,154,74,53,187,203,232,124,121,25,105,95,74,174,105,93,98,221,226,44,199,228,75,215,203,24,213,36,14,46,51,191,185,240,104,164,176,217,241,105,54,225,140,81,51,119,164,198,11,143,154,150,153,103,203,49,167,34,180,155,128,242,26,146,190,24,123,184,123,202,62,184,210,177,209,26,243,110,100,196,188,175,237,118,192,19,67,200,203,242,142,180,152,203,193,191,99,152,29,68,31,250,146,1,1,96,188,120,47,217,55,24,77,166,245,140,93,44,52,121,214,34,169,249,180,239,99,30,29,214,57,251,49,176,102,82,15,85,147,186,5,80,180,210,101,45,7,50,51,101,44,117,101,203,107,214,25,87,124,39,194,200,117,90,7,182,201,1,143,37,63,150,229,169,214,75,127,35,159,179,96,117,134,238,215,222,255,45,163,187,59,0,19,246,120,103,123,110,74,15,153,31,188,54,220,121,109,45,164,170,169,229,61,192,145,105,233,169,251,164,115,104,114,29,35,191,163,74,44,156,106,183,69,103,196,239,211,140,250,86,24,91,21,239,219,132,253,234,65,247,62,89,248,123,243,137,34,21,251,34,217,205,169,117,217,166,230,107,186,151,191,246,127,157,93,131,14,154,105,55,26,45,158,171,137,58,154,149,60,235,112,165,214,253,240,47,9,18,186,89,244,9,181,200,178,151,2,86,207,117,92,59,124,11,83,158,157,76,155,92,105,239,35,119,250,171,81,29,204,87,74,227,221,18,245,14,90,234,126,155,50,10,88,39,6,135,92,11,123,217,25,19,228,27,107,93,48,240,185,65,170,111,110,242,88,161,111,165,179,145,0,175,105,200,230,85,58,151,32,39,122,195,45,98,230,234,250,54,109,255,139,17,140,80,251,221,179,237,19,143,243,62,205,45,83,201,188,213,53,19,77,201,9,202,94,78,3,136,92,158,170,233,136,149,147,130,68,158,18,207,146,...>>},...]}}
2013-07-04 15:12:25.401 [error]
<0.11061.1>@riak_core_handoff_receiver:handle_info:80 Handoff receiver for
partition 17126972312471518572699431633393941636592959488 exited abnormally
after processing 0 objects:
{timeout,{gen_fsm,sync_send_all_state_event,[<0.1291.0>,{handoff_data,<<140,121,101,80,92,193,182,46,238,54,4,119,39,184,187,59,4,135,4,8,4,183,1,130,187,59,12,238,238,18,156,16,36,184,5,6,11,14,193,221,9,238,131,13,243,114,78,221,122,117,235,253,122,189,170,186,215,254,150,118,247,222,213,189,106,99,32,57,249,89,184,227,51,112,243,88,115,91,11,139,88,112,88,11,216,136,112,240,240,88,113,115,240,89,90,154,115,240,218,112,91,91,89,10,90,242,8,240,243,146,65,95,80,195,129,168,86,112,104,110,166,206,22,246,214,150,30,142,112,112,112,255,117,240,31,230,255,203,199,215,127,138,240,64,68,43,56,116,55,83,75,103,39,15,107,39,15,32,186,21,28,146,149,157,165,135,57,138,57,222,63,66,51,215,50,231,6,226,217,255,63,13,8,255,31,236,191,246,255,237,254,19,18,89,205,206,201,193,253,255,170,252,7,71,248,79,135,255,31,33,214,255,248,231,240,240,117,177,54,55,55,119,249,71,95,205,237,204,45,255,241,30,255,70,103,115,39,115,46,115,31,115,142,127,163,159,185,197,127,173,137,255,99,136,105,192,161,99,103,238,192,161,247,209,220,214,156,207,220,198,92,214,92,221,220,243,159,14,175,185,251,63,210,54,23,249,71,62,230,130,230,159,254,201,85,254,201,100,205,121,204,5,204,109,255,87,118,136,255,205,206,206,201,202,218,7,136,240,31,30,195,197,217,221,195,218,202,212,206,201,195,66,187,143,142,245,127,80,119,23,160,157,147,207,127,81,184,246,57,46,251,255,119,130,4,255,147,138,154,185,187,7,135,186,179,149,157,141,157,181,21,16,209,226,159,243,47,22,112,56,159,144,45,224,176,222,185,253,47,43,132,255,61,1,117,107,15,115,32,130,3,28,197,255,122,230,144,251,159,85,81,112,178,252,231,207,201,214,1,14,201,214,207,206,197,222,222,17,14,254,37,143,58,14,13,238,191,13,177,5,186,52,119,170,156,89,31,179,9,209,5,199,196,181,94,196,101,160,115,104,232,43,82,227,211,240,163,97,28,139,189,71,96,160,69,87,17,15,28,132,233,92,85,25,91,159,231,48,183,173,112,26,229,144,142,35,210,4,116,106,82,75,246,138,192,14,96,38,53,55,109,44,47,119,211,21,111,107,151,143,166,63,182,130,199,79,202,85,138,60,223,218,182,40,141,158,135,160,111,70,82,242,158,100,25,234,5,208,115,72,231,19,20,246,182,23,252,146,5,75,2,162,123,185,136,64,245,26,174,157,137,143,128,85,3,206,125,176,231,83,211,211,175,38,73,211,207,13,193,132,142,215,109,238,62,175,216,48,204,183,165,190,194,169,62,216,131,63,15,214,57,12,210,16,92,112,244,196,181,93,20,91,116,111,154,90,129,0,219,54,109,147,233,135,193,126,66,7,131,85,162,239,188,175,184,30,239,199,57,11,161,206,73,182,57,80,65,37,168,179,175,158,109,17,112,116,110,107,247,242,97,107,241,239,3,113,107,104,148,105,95,107,48,153,248,208,3,157,39,172,13,246,215,84,206,140,4,253,77,162,15,150,233,132,232,232,172,218,36,60,100,142,133,179,121,221,225,4,195,14,117,167,208,152,17,111,133,10,109,111,117,39,40,205,174,193,178,15,62,233,225,174,5,151,218,161,192,160,109,174,1,21,193,47,148,131,124,236,123,143,143,250,138,82,143,166,194,133,22,125,163,12,49,65,0,92,118,213,119,76,191,236,98,151,223,156,140,196,79,173,95,178,0,194,26,90,12,111,47,212,48,29,43,237,231,145,190,143,43,120,80,193,128,181,115,211,83,216,234,38,179,81,16,172,204,233,237,160,97,187,15,182,103,250,246,220,0,155,153,62,133,113,21,122,197,155,126,230,125,61,135,140,31,137,56,194,78,157,33,95,159,206,11,9,212,117,223,14,95,222,156,75,217,54,11,207,157,49,127,105,245,33,29,84,83,79,77,111,5,65,31,130,205,30,77,89,158,97,175,231,193,151,195,54,143,125,111,231,13,111,214,47,181,193,79,75,125,207,89,255,248,3,83,216,137,233,67,240,61,215,125,217,63,168,25,251,70,98,97,172,239,173,8,134,8,99,123,11,126,123,51,90,187,147,123,120,90,187,92,125,216,40,32,122,93,238,131,94,30,92,5,67,79,131,95,97,208,181,96,35,65,149,143,111,151,79,255,128,23,174,132,52,225,225,179,224,21,216,235,231,177,179,20,124,192,181,195,75,223,43,151,100,2,85,159,11,187,255,235,46,75,212,174,176,164,10,75,234,74,16,248,117,62,222,17,54,2,123,88,115,134,61,143,72,149,186,61,142,63,252,44,155,230,59,133,165,6,230,238,228,200,...>>},...]}}


this errors from "node to node" always the same.

Cluster rolling doesn't help.


How to fix it !?


Thanks, best regards
Sergey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130704/eef7dde8/attachment.html>


More information about the riak-users mailing list