repair-2i stops with "bad argument in call to eleveldb:async_write"

Russell Brown russell.brown at me.com
Wed Jul 30 04:50:22 EDT 2014


Hi Simon, 
So the earlier “this is on wheezy, rest are on squeeze” thing is no longer a factor?

Any and all 2i repair you do ends with the same error?

Cheers

Russell

On 30 Jul 2014, at 07:29, Effenberg, Simon <seffenberg at team.mobile.de> wrote:

> I tried it now with one partition on 6 different machines and everywhere the same result: index_scan_timeout and the info: bad argument in call to eleveldb:async_get (2x) or async_write (4x).
> 
> 
> Von Samsung Mobile gesendet
> 
> 
> -------- Ursprüngliche Nachricht --------
> Von: "Effenberg, Simon"
> Datum:30.07.2014 07:49 (GMT+01:00)
> An: bryan hunt
> Cc: riak-users at lists.basho.com
> Betreff: AW: repair-2i stops with "bad argument in call to eleveldb:async_write"
> 
> Hi,
> 
>  I tried it on two different nodes with one partition each. Both multiple times before the upgrade and after the upgrade.
> 
> I will try it on other machines in a minute but because I tried it already on two different nodes and one of them is 2 weeks old and stored on a HP 3par I bet that this is not a disk corruption issue..
> 
> Simon
> 
> 
> Von Samsung Mobile gesendet
> 
> 
> -------- Ursprüngliche Nachricht --------
> Von: bryan hunt
> Datum:29.07.2014 18:21 (GMT+01:00)
> An: "Effenberg, Simon"
> Cc: riak-users at lists.basho.com
> Betreff: Re: repair-2i stops with "bad argument in call to eleveldb:async_write"
> 
> Hi Simon,
> 
> Does the problem persist if you run it again? 
> 
> Does it happen if you run it against any other partition?
> 
> Best Regards,
> 
> Bryan
> 
> 
> 
> Bryan Hunt - Client Services Engineer - Basho Technologies Limited - Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431
> 
> On 29 Jul 2014, at 09:35, Effenberg, Simon <seffenberg at team.mobile.de> wrote:
> 
> > Hi,
> > 
> > we have some issues with 2i queries like that:
> > 
> > seffenberg at kriak46-1:~$ while :; do curl -s localhost:8098/buckets/conversation/index/createdat_int/0/23182680 | ruby -rjson -e "o = JSON.parse(STDIN.read); puts o['keys'].size"; sleep 1; done
> > 
> > 13853
> > 13853
> > 0
> > 557
> > 557
> > 557
> > 13853
> > 0
> > 
> > 
> > ...
> > 
> > So I tried to start a repair-2i first on one vnode/partition on one node
> > (which is quiet new in the cluster.. 2 weeks or so).
> > 
> > The command is failing with the following log entries:
> > 
> > seffenberg at kriak46-7:~$ sudo riak-admin repair-2i 22835963083295358096932575511191922182123945984
> > Will repair 2i on these partitions:
> >        22835963083295358096932575511191922182123945984
> > Watch the logs for 2i repair progress reports
> > seffenberg at kriak46-7:~$ 2014-07-29 08:20:22.729 UTC [info] <0.5929.1061>@riak_kv_2i_aae:init:139 Starting 2i repair at speed 100 for partitions [22835963083295358096932575511191922182123945984]
> > 2014-07-29 08:20:22.729 UTC [info] <0.5930.1061>@riak_kv_2i_aae:repair_partition:257 Acquired lock on partition 22835963083295358096932575511191922182123945984
> > 2014-07-29 08:20:22.729 UTC [info] <0.5930.1061>@riak_kv_2i_aae:repair_partition:259 Repairing indexes in partition 22835963083295358096932575511191922182123945984
> > 2014-07-29 08:20:22.740 UTC [info] <0.5930.1061>@riak_kv_2i_aae:create_index_data_db:324 Creating temporary database of 2i data in /var/lib/riak/anti_entropy/2i/tmp_db
> > 2014-07-29 08:20:22.751 UTC [info] <0.5930.1061>@riak_kv_2i_aae:create_index_data_db:361 Grabbing all index data for partition 22835963083295358096932575511191922182123945984
> > 2014-07-29 08:25:22.752 UTC [info] <0.5929.1061>@riak_kv_2i_aae:next_partition:160 Finished 2i repair:
> >        Total partitions: 1
> >        Finished partitions: 1
> >        Speed: 100
> >        Total 2i items scanned: 0
> >        Total tree objects: 0
> >        Total objects fixed: 0
> > With errors:
> > Partition: 22835963083295358096932575511191922182123945984
> > Error: index_scan_timeout
> > 
> > 
> > 2014-07-29 08:25:22.752 UTC [error] <0.4711.1061> gen_server <0.4711.1061> terminated with reason: bad argument in call to eleveldb:async_write(#Ref<0.0.10120.211816>, <<>>, [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}], []) in eleveldb:write/3 line 155
> > 2014-07-29 08:25:22.753 UTC [error] <0.4711.1061> CRASH REPORT Process <0.4711.1061> with 0 neighbours exited with reason: bad argument in call to eleveldb:async_write(#Ref<0.0.10120.211816>, <<>>, [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}], []) in eleveldb:write/3 line 155 in gen_server:terminate/6 line 747
> > 2014-07-29 08:25:22.753 UTC [error] <0.1031.0> Supervisor {<0.1031.0>,poolboy_sup} had child riak_core_vnode_worker started with {riak_core_vnode_worker,start_link,undefined} at <0.4711.1061> exit with reason bad argument in call to eleveldb:async_write(#Ref<0.0.10120.211816>, <<>>, [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}], []) in eleveldb:write/3 line 155 in context child_terminated
> > 
> > 
> > Anything I can do about that? What's the issue here?
> > 
> > I'm using Riak 1.4.8 (.deb package).
> > 
> > Cheers
> > Simon
> > _______________________________________________
> > riak-users mailing list
> > riak-users at lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





More information about the riak-users mailing list