adding nodes to cluster

Alexander Popov mogadanez at gmail.com
Wed Jan 21 14:28:05 EST 2015


Actually, problematic node is production. but when shit happens i was
migrate all data ( export/import) to clean setup.
so i can experiment  with old node.
It have not source installation, from package.
I will try compile it tomorrow.

On Wed, Jan 21, 2015 at 9:01 PM, Russell Brown <russell.brown at me.com> wrote:

> There’s a branch of riak_dt here https://github.com/basho/riak_dt/pull/111
>
> Since you’re running against a source install (I assume in development, or
> test?) would you be so kind as to try this branch and see if it solves your
> issue?
>
> Many thanks
>
> Russell
>
> On 20 Jan 2015, at 15:12, Alexander Popov <mogadanez at gmail.com> wrote:
>
> > Thanks.
> >
> > On Tue, Jan 20, 2015 at 5:07 PM, Russell Brown <russell.brown at me.com>
> wrote:
> > FYI https://github.com/basho/riak/issues/667
> >
> > I’ll get to work on it at once, we might be able to hit the 2.0.5 build
> date at the end of the week.
> >
> > On 20 Jan 2015, at 14:04, Russell Brown <russell.brown at me.com> wrote:
> >
> > > I’ll open a ticket for it. We changed the internal structure of both
> Maps and Sets, and tested that they were backwards compatible, but clearly
> missed something.
> > >
> > > On 20 Jan 2015, at 13:54, Alexander Popov <mogadanez at gmail.com> wrote:
> > >
> > >> Upgraded riak 2.0.2-> 2.0.4
> > >> trying to add nodes to cluster( was single node )
> > >> all new nodes located on another host, but multi-nodes installation
> from source
> > >> each new node have config settings tuned like
> > >>
> > >> nodename = riak1 at 10.0.0.28
> > >> platform_data_dir = /var/lib/riak1
> > >> listener.http.internal = 0.0.0.0:18098
> > >> listener.protobuf.internal = 0.0.0.0:18087
> > >> handoff.port = 18099
> > >> search.solr.port = 18093
> > >> search.solr.jmx_port = 18985
> > >> search=on
> > >>
> > >> after adding this nodes to cluster for long time was  nothing was
> transferred for new nodes.
> > >> I  was  down this nodes and force-remove it from cluster.
> > >>
> > >> After this strange things happens.
> > >> On new nodes:
> > >> Even after restart cleaning entire data dir and start again, in
> folder yz indexes somehow appears again.
> > >>
> > >> logs contains only
> > >> 2015-01-20 13:15:04.815 [info]
> <0.352.0>@riak_core:wait_for_service:483 Wait complete for service riak_kv
> (10 seconds)
> > >> 2015-01-20 13:15:06.660 [info] <0.547.0>@yz_index:local_create:189
> Created index contacts_index with schema contacts
> > >> 2015-01-20 13:15:08.683 [info] <0.547.0>@yz_index:local_create:189
> Created index users_index with schema users
> > >>
> > >>
> > >> Main Node now have bunch of errors  on start and  periodically after:
> > >>
> > >> 015-01-20 13:29:00.382 [error]
> <0.871.0>@riak_core_vnode:vnode_command:348 riak_kv_vnode command failed
> {{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_map,'-filter_unique/4-fun-1-',4,[{file,"src/ria
> > >>
> k_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{l
> > >>
> ine,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"orddict.erl"},{line,170}]}]}
> > >> 2015-01-20 13:29:00.382 [error]
> <0.650.0>@riak_core_vnode:vnode_command:348 riak_kv_vnode command failed
> {{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_map,'-filter_unique/4-fun-1-',4,[{file,"src/ria
> > >>
> k_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{l
> > >>
> ine,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"orddict.erl"},{line,170}]}]}
> > >> 2015-01-20 13:29:00.383 [error] <0.4520.0> gen_fsm <0.4520.0> in
> state waiting_local_vnode terminated with reason: no case clause matching
> {vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_
> > >>
> map,'-filter_unique/4-fun-1-',4,[{file,"src/riak_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_
> > >>
> dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{line,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"or..."},...]}]}}
> in riak_kv_put_fsm:waiting_local_vnode/2 line 5
> > >> 67
> > >> 2015-01-20 13:29:00.384 [error] <0.4520.0> CRASH REPORT Process
> <0.4520.0> with 0 neighbours exited with reason: no case clause matching
> {vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_ma
> > >>
> p,'-filter_unique/4-fun-1-',4,[{file,"src/riak_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt
> > >>
> _map,merge,2,[{file,"src/riak_dt_map.erl"},{line,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"or..."},...]}]}}
> in riak_kv_put_fsm:waiting_local_vnode/2 line 567
> > >> in gen_fsm:terminate/7 line 622
> > >> 2015-01-20 13:29:00.384 [error] <0.4521.0> gen_fsm <0.4521.0> in
> state waiting_local_vnode terminated with reason: no case clause matching
> {vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_
> > >>
> map,'-filter_unique/4-fun-1-',4,[{file,"src/riak_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_
> > >>
> dt_map,merge,2,[{file,"src/riak_dt_map.erl"},{line,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"or..."},...]}]}}
> in riak_kv_put_fsm:waiting_local_vnode/2 line 5
> > >> 67
> > >> 2015-01-20 13:29:00.384 [error] <0.4521.0> CRASH REPORT Process
> <0.4521.0> with 0 neighbours exited with reason: no case clause matching
> {vnode_error,{{badrecord,dict},[{dict,filter_dict,2,[{file,"dict.erl"},{line,464}]},{riak_dt_ma
> > >>
> p,'-filter_unique/4-fun-1-',4,[{file,"src/riak_dt_map.erl"},{line,466}]},{sets,fold_bucket,3,[{file,"sets.erl"},{line,313}]},{sets,fold_seg,4,[{file,"sets.erl"},{line,309}]},{sets,fold_segs,4,[{file,"sets.erl"},{line,305}]},{riak_dt
> > >>
> _map,merge,2,[{file,"src/riak_dt_map.erl"},{line,454}]},{riak_kv_crdt,'-merge_value/2-fun-0-',5,[{file,"src/riak_kv_crdt.erl"},{line,204}]},{orddict,update,4,[{file,"or..."},...]}]}}
> in riak_kv_put_fsm:waiting_local_vnode/2 line 567
> > >> in gen_fsm:terminate/7 line 622
> > >> 2015-01-20 13:29:00.385 [error] <0.871.0> gen_fsm <0.871.0> in state
> active terminated with reason: bad record dict in dict:filter_dict/2 line
> 464
> > >> 2015-01-20 13:29:00.385 [error] <0.871.0> CRASH REPORT Process
> <0.871.0> with 1 neighbours exited with reason: bad record dict in
> dict:filter_dict/2 line 464 in gen_fsm:terminate/7 line 622
> > >> 2015-01-20 13:29:00.385 [error] <0.167.0> Supervisor
> riak_core_vnode_sup had child undefined started with
> {riak_core_vnode,start_link,undefined} at <0.871.0> exit with reason bad
> record dict in dict:filter_dict/2 line 464 in context
> > >> child_terminated
> > >> 2015-01-20 13:29:00.385 [error] <0.988.0> gen_fsm <0.988.0> in state
> ready terminated with reason: bad record dict in dict:filter_dict/2 line 464
> > >> 2015-01-20 13:29:00.385 [error] <0.988.0> CRASH REPORT Process
> <0.988.0> with 10 neighbours exited with reason: bad record dict in
> dict:filter_dict/2 line 464 in gen_fsm:terminate/7 line 622
> > >> 2015-01-20 13:29:00.385 [error] <0.650.0> gen_fsm <0.650.0> in state
> active terminated with reason: bad record dict in dict:filter_dict/2 line
> 464
> > >> 2015-01-20 13:29:00.386 [error] <0.650.0> CRASH REPORT Process
> <0.650.0> with 1 neighbours exited with reason: bad record dict in
> dict:filter_dict/2 line 464 in gen_fsm:terminate/7 line 622
> > >> 2015-01-20 13:29:00.386 [error] <0.989.0> Supervisor
> {<0.989.0>,poolboy_sup} had child riak_core_vnode_worker started with
> riak_core_vnode_worker:start_link([{worker_module,riak_core_vnode_worker},{worker_args,[593735040165679310520
> > >> 246963290989976735222595584,...]},...]) at undefined exit with reason
> bad record dict in dict:filter_dict/2 line 464 in context shutdown_error
> > >> 2015-01-20 13:29:00.386 [error] <0.989.0> gen_server <0.989.0>
> terminated with reason: bad record dict in dict:filter_dict/2 line 464
> > >> 2015-01-20 13:29:00.386 [error] <0.989.0> CRASH REPORT Process
> <0.989.0> with 0 neighbours exited with reason: bad record dict in
> dict:filter_dict/2 line 464 in gen_server:terminate/6 line 744
> > >> 2015-01-20 13:29:00.386 [error] <0.167.0> Supervisor
> riak_core_vnode_sup had child undefined started with
> {riak_core_vnode,start_link,undefined} at <0.650.0> exit with reason bad
> record dict in dict:filter_dict/2 line 464 in context
> > >> child_terminated
> > >> 2015-01-20 13:29:00.386 [error] <0.696.0> gen_fsm <0.696.0> in state
> ready terminated with reason: bad record dict in dict:filter_dict/2 line 464
> > >> 2015-01-20 13:29:00.386 [error] <0.696.0> CRASH REPORT Process
> <0.696.0> with 10 neighbours exited with reason: bad record dict in
> dict:filter_dict/2 line 464 in gen_fsm:terminate/7 line 622
> > >> 2015-01-20 13:29:00.386 [error] <0.702.0> Supervisor
> {<0.702.0>,poolboy_sup} had child riak_core_vnode_worker started with
> riak_core_vnode_worker:start_link([{worker_module,riak_core_vnode_worker},{worker_args,[228359630832953580969
> > >> 325755111919221821239459840,...]},...]) at undefined exit with reason
> bad record dict in dict:filter_dict/2 line 464 in context shutdown_error
> > >> 2015-01-20 13:29:00.386 [error] <0.702.0> gen_server <0.702.0>
> terminated with reason: bad record dict in dict:filter_dict/2 line 464
> > >> 2015-01-20 13:29:00.386 [error] <0.702.0> CRASH REPORT Process
> <0.702.0> with 0 neighbours exited with reason: bad record dict in
> dict:filter_dict/2 line 464 in gen_server:terminate/6 line 744
> > >>
> > >>
> > >>
> > >> any advice?
> > >>
> > >> _______________________________________________
> > >> riak-users mailing list
> > >> riak-users at lists.basho.com
> > >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> > >
> > >
> > > _______________________________________________
> > > riak-users mailing list
> > > riak-users at lists.basho.com
> > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
> >
> > _______________________________________________
> > riak-users mailing list
> > riak-users at lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20150121/da5cf14e/attachment-0002.html>


More information about the riak-users mailing list