adding nodes to cluster

Russell Brown russell.brown at me.com
Tue Jan 20 09:07:53 EST 2015


FYI https://github.com/basho/riak/issues/667

I’ll get to work on it at once, we might be able to hit the 2.0.5 build date at the end of the week.

On 20 Jan 2015, at 14:04, Russell Brown <russell.brown at me.com> wrote:

> I’ll open a ticket for it. We changed the internal structure of both Maps and Sets, and tested that they were backwards compatible, but clearly missed something.
> 
> On 20 Jan 2015, at 13:54, Alexander Popov <mogadanez at gmail.com> wrote:
> 
>> Upgraded riak 2.0.2-> 2.0.4
>> trying to add nodes to cluster( was single node ) 
>> all new nodes located on another host, but multi-nodes installation from source
>> each new node have config settings tuned like 
>> 
>> nodename = riak1 at 10.0.0.28
>> platform_data_dir = /var/lib/riak1
>> listener.http.internal = 0.0.0.0:18098
>> listener.protobuf.internal = 0.0.0.0:18087
>> handoff.port = 18099
>> search.solr.port = 18093
>> search.solr.jmx_port = 18985
>> search=on
>> 
>> after adding this nodes to cluster for long time was  nothing was transferred for new nodes.
>> I  was  down this nodes and force-remove it from cluster.
>> 
>> After this strange things happens.
>> On new nodes:
>> Even after restart cleaning entire data dir and start again, in folder yz indexes somehow appears again. 
>> 
>> logs contains only 
>> 2015-01-20 13:15:04.815 [info] <0.352.0>@riak_core:wait_for_service:483 Wait complete for service riak_kv (10 seconds)
>> 2015-01-20 13:15:06.660 [info] <0.547.0>@yz_index:local_create:189 Created index contacts_index with schema contacts
>> 2015-01-20 13:15:08.683 [info] <0.547.0>@yz_index:local_create:189 Created index users_index with schema users
>> 
>> 
>> Main Node now have bunch of errors  on start and  periodically after:
>> 
>> 015-01-20 13:29:00.382 [error] <0.871.0>@riak_core_vnode:vnode_command:348 riak_kv_vnode command failed {{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_map,'-filter_unique/4-fun-1-',4,[{file,"src/ria
>> k_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{l
>> ine,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"orddict.erl"},{line,170}]}]}
>> 2015-01-20 13:29:00.382 [error] <0.650.0>@riak_core_vnode:vnode_command:348 riak_kv_vnode command failed {{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_map,'-filter_unique/4-fun-1-',4,[{file,"src/ria
>> k_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{l
>> ine,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"orddict.erl"},{line,170}]}]}
>> 2015-01-20 13:29:00.383 [error] <0.4520.0> gen_fsm <0.4520.0> in state waiting_local_vnode terminated with reason: no case clause matching {vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_
>> map,'-filter_unique/4-fun-1-',4,[{file,"src/riak_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_
>> dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{line,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"or..."},...]}]}} in riak_kv_put_fsm:waiting_local_vnode/2 line 5
>> 67
>> 2015-01-20 13:29:00.384 [error] <0.4520.0> CRASH REPORT Process <0.4520.0> with 0 neighbours exited with reason: no case clause matching {vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_ma
>> p,'-filter_unique/4-fun-1-',4,[{file,"src/riak_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt
>> _map,merge,2,[{file,"src/riak_dt_map.erl"},{line,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"or..."},...]}]}} in riak_kv_put_fsm:waiting_local_vnode/2 line 567
>> in gen_fsm:terminate/7 line 622
>> 2015-01-20 13:29:00.384 [error] <0.4521.0> gen_fsm <0.4521.0> in state waiting_local_vnode terminated with reason: no case clause matching {vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_
>> map,'-filter_unique/4-fun-1-',4,[{file,"src/riak_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_
>> dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{line,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"or..."},...]}]}} in riak_kv_put_fsm:waiting_local_vnode/2 line 5
>> 67
>> 2015-01-20 13:29:00.384 [error] <0.4521.0> CRASH REPORT Process <0.4521.0> with 0 neighbours exited with reason: no case clause matching {vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_ma
>> p,'-filter_unique/4-fun-1-',4,[{file,"src/riak_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt
>> _map,merge,2,[{file,"src/riak_dt_map.erl"},{line,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"or..."},...]}]}} in riak_kv_put_fsm:waiting_local_vnode/2 line 567
>> in gen_fsm:terminate/7 line 622
>> 2015-01-20 13:29:00.385 [error] <0.871.0> gen_fsm <0.871.0> in state active terminated with reason: bad record dict in dict:filter_dict/2 line 464
>> 2015-01-20 13:29:00.385 [error] <0.871.0> CRASH REPORT Process <0.871.0> with 1 neighbours exited with reason: bad record dict in dict:filter_dict/2 line 464 in gen_fsm:terminate/7 line 622
>> 2015-01-20 13:29:00.385 [error] <0.167.0> Supervisor riak_core_vnode_sup had child undefined started with {riak_core_vnode,start_link,undefined} at <0.871.0> exit with reason bad record dict in dict:filter_dict/2 line 464 in context
>> child_terminated
>> 2015-01-20 13:29:00.385 [error] <0.988.0> gen_fsm <0.988.0> in state ready terminated with reason: bad record dict in dict:filter_dict/2 line 464
>> 2015-01-20 13:29:00.385 [error] <0.988.0> CRASH REPORT Process <0.988.0> with 10 neighbours exited with reason: bad record dict in dict:filter_dict/2 line 464 in gen_fsm:terminate/7 line 622
>> 2015-01-20 13:29:00.385 [error] <0.650.0> gen_fsm <0.650.0> in state active terminated with reason: bad record dict in dict:filter_dict/2 line 464
>> 2015-01-20 13:29:00.386 [error] <0.650.0> CRASH REPORT Process <0.650.0> with 1 neighbours exited with reason: bad record dict in dict:filter_dict/2 line 464 in gen_fsm:terminate/7 line 622
>> 2015-01-20 13:29:00.386 [error] <0.989.0> Supervisor {<0.989.0>,poolboy_sup} had child riak_core_vnode_worker started with riak_core_vnode_worker:start_link([{worker_module,riak_core_vnode_worker},{worker_args,[593735040165679310520
>> 246963290989976735222595584,...]},...]) at undefined exit with reason bad record dict in dict:filter_dict/2 line 464 in context shutdown_error
>> 2015-01-20 13:29:00.386 [error] <0.989.0> gen_server <0.989.0> terminated with reason: bad record dict in dict:filter_dict/2 line 464
>> 2015-01-20 13:29:00.386 [error] <0.989.0> CRASH REPORT Process <0.989.0> with 0 neighbours exited with reason: bad record dict in dict:filter_dict/2 line 464 in gen_server:terminate/6 line 744
>> 2015-01-20 13:29:00.386 [error] <0.167.0> Supervisor riak_core_vnode_sup had child undefined started with {riak_core_vnode,start_link,undefined} at <0.650.0> exit with reason bad record dict in dict:filter_dict/2 line 464 in context
>> child_terminated
>> 2015-01-20 13:29:00.386 [error] <0.696.0> gen_fsm <0.696.0> in state ready terminated with reason: bad record dict in dict:filter_dict/2 line 464
>> 2015-01-20 13:29:00.386 [error] <0.696.0> CRASH REPORT Process <0.696.0> with 10 neighbours exited with reason: bad record dict in dict:filter_dict/2 line 464 in gen_fsm:terminate/7 line 622
>> 2015-01-20 13:29:00.386 [error] <0.702.0> Supervisor {<0.702.0>,poolboy_sup} had child riak_core_vnode_worker started with riak_core_vnode_worker:start_link([{worker_module,riak_core_vnode_worker},{worker_args,[228359630832953580969
>> 325755111919221821239459840,...]},...]) at undefined exit with reason bad record dict in dict:filter_dict/2 line 464 in context shutdown_error
>> 2015-01-20 13:29:00.386 [error] <0.702.0> gen_server <0.702.0> terminated with reason: bad record dict in dict:filter_dict/2 line 464
>> 2015-01-20 13:29:00.386 [error] <0.702.0> CRASH REPORT Process <0.702.0> with 0 neighbours exited with reason: bad record dict in dict:filter_dict/2 line 464 in gen_server:terminate/6 line 744
>> 
>> 
>> 
>> any advice?
>> 
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> 
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





More information about the riak-users mailing list