repair-2i stops with "bad argument in call to eleveldb:async_write"

bryan hunt bhunt at basho.com
Tue Jul 29 12:21:26 EDT 2014


Sounds like disk corruption to me.



Bryan Hunt - Client Services Engineer - Basho Technologies Limited - Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431

On 29 Jul 2014, at 13:06, Effenberg, Simon <seffenberg at team.mobile.de> wrote:

> Said to say but the issue stays the same.. even after the upgrade to
> 1.4.10.
> 
> Any ideas what is happening here?
> 
> Cheers
> Simon
> 
> On Tue, Jul 29, 2014 at 08:46:42AM +0000, Effenberg, Simon wrote:
>> Already started to prepare everything for it.. :)
>> 
>> On Tue, Jul 29, 2014 at 09:43:22AM +0100, Guido Medina wrote:
>>> Hi Simon,
>>> 
>>> There are some (maybe related) Level DB fixes in 1.4.9 and 1.4.10, I don't
>>> think there isn't any harm for you to do a rolling upgrade since nothing
>>> major changed, just bug fixes, here is the release notes' link for
>>> reference:
>>> 
>>> https://github.com/basho/riak/blob/1.4/RELEASE-NOTES.md
>>> 
>>> Best regards,
>>> 
>>> Guido.
>>> 
>>> On 29/07/14 09:35, Effenberg, Simon wrote:
>>>> Hi,
>>>> 
>>>> we have some issues with 2i queries like that:
>>>> 
>>>> seffenberg at kriak46-1:~$ while :; do curl -s localhost:8098/buckets/conversation/index/createdat_int/0/23182680 | ruby -rjson -e "o = JSON.parse(STDIN.read); puts o['keys'].size"; sleep 1; done
>>>> 
>>>> 13853
>>>> 13853
>>>> 0
>>>> 557
>>>> 557
>>>> 557
>>>> 13853
>>>> 0
>>>> 
>>>> 
>>>> ...
>>>> 
>>>> So I tried to start a repair-2i first on one vnode/partition on one node
>>>> (which is quiet new in the cluster.. 2 weeks or so).
>>>> 
>>>> The command is failing with the following log entries:
>>>> 
>>>> seffenberg at kriak46-7:~$ sudo riak-admin repair-2i 22835963083295358096932575511191922182123945984
>>>> Will repair 2i on these partitions:
>>>>        22835963083295358096932575511191922182123945984
>>>> Watch the logs for 2i repair progress reports
>>>> seffenberg at kriak46-7:~$ 2014-07-29 08:20:22.729 UTC [info] <0.5929.1061>@riak_kv_2i_aae:init:139 Starting 2i repair at speed 100 for partitions [22835963083295358096932575511191922182123945984]
>>>> 2014-07-29 08:20:22.729 UTC [info] <0.5930.1061>@riak_kv_2i_aae:repair_partition:257 Acquired lock on partition 22835963083295358096932575511191922182123945984
>>>> 2014-07-29 08:20:22.729 UTC [info] <0.5930.1061>@riak_kv_2i_aae:repair_partition:259 Repairing indexes in partition 22835963083295358096932575511191922182123945984
>>>> 2014-07-29 08:20:22.740 UTC [info] <0.5930.1061>@riak_kv_2i_aae:create_index_data_db:324 Creating temporary database of 2i data in /var/lib/riak/anti_entropy/2i/tmp_db
>>>> 2014-07-29 08:20:22.751 UTC [info] <0.5930.1061>@riak_kv_2i_aae:create_index_data_db:361 Grabbing all index data for partition 22835963083295358096932575511191922182123945984
>>>> 2014-07-29 08:25:22.752 UTC [info] <0.5929.1061>@riak_kv_2i_aae:next_partition:160 Finished 2i repair:
>>>>        Total partitions: 1
>>>>        Finished partitions: 1
>>>>        Speed: 100
>>>>        Total 2i items scanned: 0
>>>>        Total tree objects: 0
>>>>        Total objects fixed: 0
>>>> With errors:
>>>> Partition: 22835963083295358096932575511191922182123945984
>>>> Error: index_scan_timeout
>>>> 
>>>> 
>>>> 2014-07-29 08:25:22.752 UTC [error] <0.4711.1061> gen_server <0.4711.1061> terminated with reason: bad argument in call to eleveldb:async_write(#Ref<0.0.10120.211816>, <<>>, [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}], []) in eleveldb:write/3 line 155
>>>> 2014-07-29 08:25:22.753 UTC [error] <0.4711.1061> CRASH REPORT Process <0.4711.1061> with 0 neighbours exited with reason: bad argument in call to eleveldb:async_write(#Ref<0.0.10120.211816>, <<>>, [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}], []) in eleveldb:write/3 line 155 in gen_server:terminate/6 line 747
>>>> 2014-07-29 08:25:22.753 UTC [error] <0.1031.0> Supervisor {<0.1031.0>,poolboy_sup} had child riak_core_vnode_worker started with {riak_core_vnode_worker,start_link,undefined} at <0.4711.1061> exit with reason bad argument in call to eleveldb:async_write(#Ref<0.0.10120.211816>, <<>>, [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}], []) in eleveldb:write/3 line 155 in context child_terminated
>>>> 
>>>> 
>>>> Anything I can do about that? What's the issue here?
>>>> 
>>>> I'm using Riak 1.4.8 (.deb package).
>>>> 
>>>> Cheers
>>>> Simon
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> riak-users at lists.basho.com
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>> 
>>> 
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> 
>> -- 
>> Simon Effenberg | Site Op | mobile.international GmbH
>> 
>> Phone:    + 49. 30. 8109. 7173
>> M-Phone:  + 49. 151. 5266. 1558
>> Mail:     seffenberg at team.mobile.de
>> Web:      www.mobile.de
>> 
>> Marktplatz 1 | 14532 Europarc Dreilinden | Germany
>> 
>> ______________________________________________________
>> Geschäftsführer: Malte Krüger
>> HRB Nr.: 18517 P, Amtsgericht Potsdam
>> Sitz der Gesellschaft: Kleinmachnow
>> ______________________________________________________
>> 
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> -- 
> Simon Effenberg | Site Op | mobile.international GmbH
> 
> Phone:    + 49. 30. 8109. 7173
> M-Phone:  + 49. 151. 5266. 1558
> Mail:     seffenberg at team.mobile.de
> Web:      www.mobile.de
> 
> Marktplatz 1 | 14532 Europarc Dreilinden | Germany
> 
> ______________________________________________________
> Geschäftsführer: Malte Krüger
> HRB Nr.: 18517 P, Amtsgericht Potsdam
> Sitz der Gesellschaft: Kleinmachnow
> ______________________________________________________
> 
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





More information about the riak-users mailing list