repair-2i stops with "bad argument in call to eleveldb:async_write"

Effenberg, Simon seffenberg at team.mobile.de
Mon Aug 11 04:24:44 EDT 2014


Hi,

any updates on this issue? I'm still able to search a range of 2i and
I'm getting 3 results.. 0, 557 and 13853 :(..

I cannot rely on 2i right now nor can I repair it.

Cheers
Simon

On Fri, Aug 08, 2014 at 07:12:58AM +0000, Effenberg, Simon wrote:
> Hi Bryan,
> 
> thanks for this. I tried it but to be honest I cannot see any specific
> stuff in the logs (on the specific host).
> 
> I attached the logfile from the specific node. If you think it is
> also/more important to look into the logfiles on the other nodes I can
> send them as well.. but a quick look into all of them (searching for
> "2i" and "index") didn't show anything unusual.. the only stuff was
> 
> 2014-08-07 05:44:11.298 UTC [debug] <0.969.0>@riak_kv_index_hashtree:handle_call:240 Updating tree: (vnode)=633697975561446187189878970435575840553939501056 (preflist)={610862012478150829092946394924383918371815555072,12}
> 
> and searching for errors didn't show more than you see in the attached
> files:
> 
> $ for host in kriak46-{1..7} kriak47-{1..6}; do echo $host; ssh $host "grep '^2014-08-07 05' /var/log/riak/console.log | grep -i error" ; done
> kriak46-1
> 2014-08-07 05:38:28.596 UTC [error] <0.8949.566> ** Node 'c_24556_riak at 10.46.109.201' not responding **
> 2014-08-07 05:42:36.197 UTC [error] <0.24823.566> ** Node 'c_26945_riak at 10.46.109.201' not responding **
> 2014-08-07 05:43:16.213 UTC [error] <0.26434.566> ** Node 'c_27071_riak at 10.46.109.201' not responding **
> 2014-08-07 05:48:14.284 UTC [error] <0.1697.0> gen_server <0.1697.0> terminated with reason: bad argument in call to eleveldb:async_write(#Ref<0.0.567.170046>, <<>>, [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}], []) in eleveldb:write/3 line 155
> 2014-08-07 05:48:14.284 UTC [error] <0.1697.0> CRASH REPORT Process <0.1697.0> with 0 neighbours exited with reason: bad argument in call to eleveldb:async_write(#Ref<0.0.567.170046>, <<>>, [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}], []) in eleveldb:write/3 line 155 in gen_server:terminate/6 line 747
> 2014-08-07 05:48:14.284 UTC [error] <0.1692.0> Supervisor {<0.1692.0>,poolboy_sup} had child riak_core_vnode_worker started with {riak_core_vnode_worker,start_link,undefined} at <0.1697.0> exit with reason bad argument in call to eleveldb:async_write(#Ref<0.0.567.170046>, <<>>, [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}], []) in eleveldb:write/3 line 155 in context child_terminated
> 2014-08-i7 05:50:11.390 UTC [error] <0.20983.567> ** Node 'c_32188_riak at 10.46.109.201' not responding **
> kriak46-2
> kriak46-3
> kriak46-4
> kriak46-5
> kriak46-6
> kriak46-7
> kriak47-1
> kriak47-2
> kriak47-3
> kriak47-4
> kriak47-5
> kriak47-6
> 
> You mentioned the partition repair stuff.. do you think I need to try
> out the full repair? Is this maybe a way to fix it? Because it is quiet
> hard to do this on the cluster (~15 TB of data with AAE stuff and
> tombstones and maybe ~10 TB without tombstones and AAE stuff) and I
> don't want to start doing this if it won't help.
> 
> Cheers
> Simon
> 
> On Wed, Aug 06, 2014 at 01:08:36PM +0100, bryan hunt wrote:
> > Simon,
> > 
> > If you want to get more verbose logging information, you could perform the following to change the logging level, to debug, then run `repair-2i`, and finally switching back to the normal logging level.
> > 
> > - `riak attach`
> > - `(riak at nodename)1> SetDebug = fun() -> {node(), lager:set_loglevel(lager_file_backend, "/var/log/riak/console.log", debug)} end.`
> > - `(riak at nodename)2> rp(rpc:multicall(erlang, apply, [SetDebug,[]])).`
> > (don't forget the period at the end of these statements)
> > - Hit CTRL+C twice to quit from the node
> > 
> > You can then revert back to the normal `info` logging level by running the following command via `riak attach`:
> > 
> > - `riak attach`
> > - `(riak at nodename)1> SetInfo = fun() -> {node(), lager:set_loglevel(lager_file_backend, "/var/log/riak/console.log", info)} end.`
> > - `(riak at nodename)2> rp(rpc:multicall(erlang, apply, [SetInfo,[]])).`
> > (don't forget the period at the end of these statements)
> > - Hit CTRL+C twice to quit from a the node
> > 
> > Please also see the docs for info on `riak attach` monitoring of repairs.
> > 
> > http://docs.basho.com/riak/1.4.9/ops/running/recovery/repairing-partitions/#Monitoring-Repairs
> > 
> > Repairs can also be monitored using the `riak-admin transfers` command.
> > 
> > http://docs.basho.com/riak/1.4.9/ops/running/recovery/repairing-partitions/#Running-a-Repair
> > 
> > Best Regards,
> > 
> > Bryan Hunt 
> > 
> > Bryan Hunt - Client Services Engineer - Basho Technologies Limited - Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431
> > 

> 2014-08-07 05:43:04.469 UTC [notice] <0.57.0>@lager_file_backend:128 Changed loglevel of /var/log/riak/console.log to debug
> 2014-08-07 05:43:14.156 UTC [info] <0.27496.566>@riak_kv_2i_aae:init:139 Starting 2i repair at speed 100 for partitions [319703483166135013357056057156686910549735243776]
> 2014-08-07 05:43:14.156 UTC [info] <0.27497.566>@riak_kv_2i_aae:repair_partition:257 Acquired lock on partition 319703483166135013357056057156686910549735243776
> 2014-08-07 05:43:14.156 UTC [info] <0.27497.566>@riak_kv_2i_aae:repair_partition:259 Repairing indexes in partition 319703483166135013357056057156686910549735243776
> 2014-08-07 05:43:14.239 UTC [info] <0.27497.566>@riak_kv_2i_aae:create_index_data_db:324 Creating temporary database of 2i data in /var/lib/riak/anti_entropy/2i/tmp_db
> 2014-08-07 05:43:14.283 UTC [info] <0.27497.566>@riak_kv_2i_aae:create_index_data_db:361 Grabbing all index data for partition 319703483166135013357056057156686910549735243776
> 2014-08-07 05:43:14.289 UTC [debug] <0.588.0>@riak_kv_entropy_manager:requeue_exchange:717 Requeue: {822094670998632891489572718402909198556462055424,879184578706871286731904157180889004011771920384,{822094670998632891489572718402909198556462055424,12}}
> 2014-08-07 05:43:16.213 UTC [error] <0.26434.566> ** Node 'c_27071_riak at 10.46.109.201' not responding **
> ** Removing (timedout) connection **
> 
> 2014-08-07 05:43:29.290 UTC [debug] <0.28227.566>@riak_kv_exchange_fsm:init:76 Starting exchange: {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:43:29.290 UTC [debug] <0.28227.566>@riak_kv_exchange_fsm:update_trees:143 Sending to {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:43:29.290 UTC [debug] <0.28227.566>@riak_kv_exchange_fsm:update_trees:144 Sending to {896311551019342805304603588814282945648364879872,'riak at 10.46.109.202'}
> 2014-08-07 05:43:29.290 UTC [debug] <0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree: (vnode)=890602560248518965780370444936484965102833893376 (preflist)={833512652540280570538039006158505159647524028416,12}
> 2014-08-07 05:43:30.633 UTC [debug] <0.28227.566>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
> 2014-08-07 05:43:30.633 UTC [debug] <0.28227.566>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange between {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'} and {896311551019342805304603588814282945648364879872,'riak at 10.46.109.202'}
> 2014-08-07 05:43:30.633 UTC [debug] <0.28227.566>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for preflist {833512652540280570538039006158505159647524028416,12}
> 2014-08-07 05:43:33.673 UTC [debug] <0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange: 890602560248518965780370444936484965102833893376 :: normal
> 2014-08-07 05:43:44.291 UTC [debug] <0.29028.566>@riak_kv_exchange_fsm:init:76 Starting exchange: {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:43:44.291 UTC [debug] <0.29028.566>@riak_kv_exchange_fsm:update_trees:143 Sending to {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:43:44.291 UTC [debug] <0.29028.566>@riak_kv_exchange_fsm:update_trees:144 Sending to {896311551019342805304603588814282945648364879872,'riak at 10.46.109.202'}
> 2014-08-07 05:43:44.291 UTC [debug] <0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree: (vnode)=890602560248518965780370444936484965102833893376 (preflist)={839221643311104410062272150036303140193055014912,12}
> 2014-08-07 05:43:45.929 UTC [debug] <0.29028.566>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
> 2014-08-07 05:43:45.929 UTC [debug] <0.29028.566>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange between {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'} and {896311551019342805304603588814282945648364879872,'riak at 10.46.109.202'}
> 2014-08-07 05:43:45.929 UTC [debug] <0.29028.566>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for preflist {839221643311104410062272150036303140193055014912,12}
> 2014-08-07 05:43:49.312 UTC [debug] <0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange: 890602560248518965780370444936484965102833893376 :: normal
> 2014-08-07 05:43:59.292 UTC [debug] <0.29871.566>@riak_kv_exchange_fsm:init:76 Starting exchange: {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:43:59.292 UTC [debug] <0.29871.566>@riak_kv_exchange_fsm:update_trees:143 Sending to {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:43:59.292 UTC [debug] <0.29871.566>@riak_kv_exchange_fsm:update_trees:144 Sending to {896311551019342805304603588814282945648364879872,'riak at 10.46.109.202'}
> 2014-08-07 05:43:59.292 UTC [debug] <0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree: (vnode)=890602560248518965780370444936484965102833893376 (preflist)={844930634081928249586505293914101120738586001408,12}
> 2014-08-07 05:43:59.822 UTC [debug] <0.29871.566>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
> 2014-08-07 05:43:59.822 UTC [debug] <0.29871.566>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange between {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'} and {896311551019342805304603588814282945648364879872,'riak at 10.46.109.202'}
> 2014-08-07 05:43:59.822 UTC [debug] <0.29871.566>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for preflist {844930634081928249586505293914101120738586001408,12}
> 2014-08-07 05:44:02.535 UTC [debug] <0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange: 890602560248518965780370444936484965102833893376 :: normal
> 2014-08-07 05:44:14.293 UTC [debug] <0.30743.566>@riak_kv_exchange_fsm:init:76 Starting exchange: {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:44:14.293 UTC [debug] <0.30743.566>@riak_kv_exchange_fsm:update_trees:143 Sending to {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:44:14.293 UTC [debug] <0.30743.566>@riak_kv_exchange_fsm:update_trees:144 Sending to {896311551019342805304603588814282945648364879872,'riak at 10.46.109.202'}
> 2014-08-07 05:44:14.293 UTC [debug] <0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree: (vnode)=890602560248518965780370444936484965102833893376 (preflist)={850639624852752089110738437791899101284116987904,12}
> 2014-08-07 05:44:14.669 UTC [debug] <0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange: 822094670998632891489572718402909198556462055424 :: normal
> 2014-08-07 05:44:22.126 UTC [debug] <0.30743.566>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
> 2014-08-07 05:44:22.126 UTC [debug] <0.30743.566>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange between {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'} and {896311551019342805304603588814282945648364879872,'riak at 10.46.109.202'}
> 2014-08-07 05:44:22.126 UTC [debug] <0.30743.566>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for preflist {850639624852752089110738437791899101284116987904,12}
> 2014-08-07 05:44:25.055 UTC [debug] <0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange: 890602560248518965780370444936484965102833893376 :: normal
> 2014-08-07 05:44:29.294 UTC [debug] <0.31540.566>@riak_kv_exchange_fsm:init:76 Starting exchange: {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:44:29.294 UTC [debug] <0.31540.566>@riak_kv_exchange_fsm:update_trees:143 Sending to {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:44:29.294 UTC [debug] <0.31540.566>@riak_kv_exchange_fsm:update_trees:144 Sending to {896311551019342805304603588814282945648364879872,'riak at 10.46.109.202'}
> 2014-08-07 05:44:29.294 UTC [debug] <0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree: (vnode)=890602560248518965780370444936484965102833893376 (preflist)={856348615623575928634971581669697081829647974400,12}
> 2014-08-07 05:44:30.567 UTC [debug] <0.31540.566>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
> 2014-08-07 05:44:30.568 UTC [debug] <0.31540.566>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange between {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'} and {896311551019342805304603588814282945648364879872,'riak at 10.46.109.202'}
> 2014-08-07 05:44:30.568 UTC [debug] <0.31540.566>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for preflist {856348615623575928634971581669697081829647974400,12}
> 2014-08-07 05:44:33.509 UTC [debug] <0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange: 890602560248518965780370444936484965102833893376 :: normal
> 2014-08-07 05:44:44.295 UTC [debug] <0.32261.566>@riak_kv_exchange_fsm:init:76 Starting exchange: {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:44:44.295 UTC [debug] <0.32261.566>@riak_kv_exchange_fsm:update_trees:143 Sending to {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:44:44.295 UTC [debug] <0.32261.566>@riak_kv_exchange_fsm:update_trees:144 Sending to {896311551019342805304603588814282945648364879872,'riak at 10.46.109.202'}
> 2014-08-07 05:44:44.295 UTC [debug] <0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree: (vnode)=890602560248518965780370444936484965102833893376 (preflist)={862057606394399768159204725547495062375178960896,12}
> 2014-08-07 05:44:52.849 UTC [debug] <0.32261.566>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
> 2014-08-07 05:44:52.849 UTC [debug] <0.32261.566>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange between {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'} and {896311551019342805304603588814282945648364879872,'riak at 10.46.109.202'}
> 2014-08-07 05:44:52.849 UTC [debug] <0.32261.566>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for preflist {862057606394399768159204725547495062375178960896,12}
> 2014-08-07 05:44:55.580 UTC [debug] <0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange: 890602560248518965780370444936484965102833893376 :: normal
> 2014-08-07 05:44:59.296 UTC [debug] <0.281.567>@riak_kv_exchange_fsm:init:76 Starting exchange: {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:44:59.296 UTC [debug] <0.281.567>@riak_kv_exchange_fsm:update_trees:143 Sending to {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:44:59.296 UTC [debug] <0.281.567>@riak_kv_exchange_fsm:update_trees:144 Sending to {896311551019342805304603588814282945648364879872,'riak at 10.46.109.202'}
> 2014-08-07 05:44:59.296 UTC [debug] <0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree: (vnode)=890602560248518965780370444936484965102833893376 (preflist)={884893569477695126256137301058686984557302906880,3}
> 2014-08-07 05:45:09.895 UTC [debug] <0.281.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
> 2014-08-07 05:45:09.895 UTC [debug] <0.281.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange between {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'} and {896311551019342805304603588814282945648364879872,'riak at 10.46.109.202'}
> 2014-08-07 05:45:09.895 UTC [debug] <0.281.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for preflist {884893569477695126256137301058686984557302906880,3}
> 2014-08-07 05:45:11.747 UTC [info] <0.281.567>@riak_kv_exchange_fsm:key_exchange:206 Repaired 2 keys during active anti-entropy exchange of {884893569477695126256137301058686984557302906880,3} between {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'} and {896311551019342805304603588814282945648364879872,'riak at 10.46.109.202'}
> 2014-08-07 05:45:11.747 UTC [debug] <0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange: 890602560248518965780370444936484965102833893376 :: normal
> 2014-08-07 05:45:14.297 UTC [debug] <0.1111.567>@riak_kv_exchange_fsm:init:76 Starting exchange: {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:45:14.297 UTC [debug] <0.1111.567>@riak_kv_exchange_fsm:update_trees:143 Sending to {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:45:14.297 UTC [debug] <0.1111.567>@riak_kv_exchange_fsm:update_trees:144 Sending to {896311551019342805304603588814282945648364879872,'riak at 10.46.109.202'}
> 2014-08-07 05:45:14.297 UTC [debug] <0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree: (vnode)=890602560248518965780370444936484965102833893376 (preflist)={890602560248518965780370444936484965102833893376,3}
> 2014-08-07 05:45:24.617 UTC [debug] <0.1111.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
> 2014-08-07 05:45:24.617 UTC [debug] <0.1111.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange between {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'} and {896311551019342805304603588814282945648364879872,'riak at 10.46.109.202'}
> 2014-08-07 05:45:24.617 UTC [debug] <0.1111.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for preflist {890602560248518965780370444936484965102833893376,3}
> 2014-08-07 05:45:26.307 UTC [debug] <0.1404.0>@riak_kv_index_hashtree:handle_call:240 Updating tree: (vnode)=685078892498860742907977265335757665463718379520 (preflist)={673660910957213063859510977580161704372656406528,3}
> 2014-08-07 05:45:29.298 UTC [debug] <0.588.0>@riak_kv_entropy_manager:requeue_exchange:717 Requeue: {890602560248518965780370444936484965102833893376,913438523331814323877303020447676887284957839360,{884893569477695126256137301058686984557302906880,12}}
> 2014-08-07 05:45:29.964 UTC [info] <0.1111.567>@riak_kv_exchange_fsm:key_exchange:206 Repaired 1 keys during active anti-entropy exchange of {890602560248518965780370444936484965102833893376,3} between {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'} and {896311551019342805304603588814282945648364879872,'riak at 10.46.109.202'}
> 2014-08-07 05:45:29.964 UTC [debug] <0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange: 890602560248518965780370444936484965102833893376 :: normal
> 2014-08-07 05:45:44.299 UTC [debug] <0.2988.567>@riak_kv_exchange_fsm:init:76 Starting exchange: {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:45:44.299 UTC [debug] <0.2988.567>@riak_kv_exchange_fsm:update_trees:143 Sending to {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:45:44.299 UTC [debug] <0.2988.567>@riak_kv_exchange_fsm:update_trees:144 Sending to {913438523331814323877303020447676887284957839360,'riak at 10.46.109.205'}
> 2014-08-07 05:45:44.299 UTC [debug] <0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree: (vnode)=890602560248518965780370444936484965102833893376 (preflist)={890602560248518965780370444936484965102833893376,12}
> 2014-08-07 05:45:48.779 UTC [debug] <0.2988.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
> 2014-08-07 05:45:48.779 UTC [debug] <0.2988.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange between {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'} and {913438523331814323877303020447676887284957839360,'riak at 10.46.109.205'}
> 2014-08-07 05:45:48.779 UTC [debug] <0.2988.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for preflist {890602560248518965780370444936484965102833893376,12}
> 2014-08-07 05:45:56.046 UTC [debug] <0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange: 890602560248518965780370444936484965102833893376 :: normal
> 2014-08-07 05:45:59.300 UTC [debug] <0.4233.567>@riak_kv_exchange_fsm:init:76 Starting exchange: {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:45:59.300 UTC [debug] <0.4233.567>@riak_kv_exchange_fsm:update_trees:143 Sending to {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:45:59.300 UTC [debug] <0.4233.567>@riak_kv_exchange_fsm:update_trees:144 Sending to {919147514102638163401536164325474867830488825856,'riak at 10.46.109.206'}
> 2014-08-07 05:45:59.300 UTC [debug] <0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree: (vnode)=890602560248518965780370444936484965102833893376 (preflist)={856348615623575928634971581669697081829647974400,12}
> 2014-08-07 05:46:01.552 UTC [debug] <0.4233.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
> 2014-08-07 05:46:01.552 UTC [debug] <0.4233.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange between {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'} and {919147514102638163401536164325474867830488825856,'riak at 10.46.109.206'}
> 2014-08-07 05:46:01.552 UTC [debug] <0.4233.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for preflist {856348615623575928634971581669697081829647974400,12}
> 2014-08-07 05:46:06.327 UTC [debug] <0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange: 890602560248518965780370444936484965102833893376 :: normal
> 2014-08-07 05:46:14.301 UTC [debug] <0.5306.567>@riak_kv_exchange_fsm:init:76 Starting exchange: {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:46:14.301 UTC [debug] <0.5306.567>@riak_kv_exchange_fsm:update_trees:143 Sending to {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:46:14.301 UTC [debug] <0.5306.567>@riak_kv_exchange_fsm:update_trees:144 Sending to {919147514102638163401536164325474867830488825856,'riak at 10.46.109.206'}
> 2014-08-07 05:46:14.301 UTC [debug] <0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree: (vnode)=890602560248518965780370444936484965102833893376 (preflist)={862057606394399768159204725547495062375178960896,12}
> 2014-08-07 05:46:15.236 UTC [debug] <0.5306.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
> 2014-08-07 05:46:15.236 UTC [debug] <0.5306.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange between {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'} and {919147514102638163401536164325474867830488825856,'riak at 10.46.109.206'}
> 2014-08-07 05:46:15.236 UTC [debug] <0.5306.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for preflist {862057606394399768159204725547495062375178960896,12}
> 2014-08-07 05:46:20.366 UTC [debug] <0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange: 890602560248518965780370444936484965102833893376 :: normal
> 2014-08-07 05:46:29.302 UTC [debug] <0.6563.567>@riak_kv_exchange_fsm:init:76 Starting exchange: {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:46:29.302 UTC [debug] <0.6563.567>@riak_kv_exchange_fsm:update_trees:143 Sending to {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:46:29.302 UTC [debug] <0.6563.567>@riak_kv_exchange_fsm:update_trees:144 Sending to {919147514102638163401536164325474867830488825856,'riak at 10.46.109.206'}
> 2014-08-07 05:46:29.302 UTC [debug] <0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree: (vnode)=890602560248518965780370444936484965102833893376 (preflist)={873475587936047447207671013303091023466240933888,12}
> 2014-08-07 05:46:44.303 UTC [debug] <0.588.0>@riak_kv_entropy_manager:requeue_exchange:717 Requeue: {890602560248518965780370444936484965102833893376,919147514102638163401536164325474867830488825856,{879184578706871286731904157180889004011771920384,12}}
> 2014-08-07 05:46:52.665 UTC [debug] <0.6563.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
> 2014-08-07 05:46:52.665 UTC [debug] <0.6563.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange between {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'} and {919147514102638163401536164325474867830488825856,'riak at 10.46.109.206'}
> 2014-08-07 05:46:52.665 UTC [debug] <0.6563.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for preflist {873475587936047447207671013303091023466240933888,12}
> 2014-08-07 05:46:57.385 UTC [info] <0.6563.567>@riak_kv_exchange_fsm:key_exchange:206 Repaired 13 keys during active anti-entropy exchange of {873475587936047447207671013303091023466240933888,12} between {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'} and {919147514102638163401536164325474867830488825856,'riak at 10.46.109.206'}
> 2014-08-07 05:46:57.385 UTC [debug] <0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange: 890602560248518965780370444936484965102833893376 :: normal
> 2014-08-07 05:46:59.304 UTC [debug] <0.9160.567>@riak_kv_exchange_fsm:init:76 Starting exchange: {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:46:59.315 UTC [debug] <0.9160.567>@riak_kv_exchange_fsm:update_trees:143 Sending to {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:46:59.315 UTC [debug] <0.9160.567>@riak_kv_exchange_fsm:update_trees:144 Sending to {919147514102638163401536164325474867830488825856,'riak at 10.46.109.206'}
> 2014-08-07 05:46:59.315 UTC [debug] <0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree: (vnode)=890602560248518965780370444936484965102833893376 (preflist)={890602560248518965780370444936484965102833893376,12}
> 2014-08-07 05:46:59.368 UTC [info] <0.588.0>@riak_kv_entropy_manager:perhaps_log_throttle_change:826 Changing AAE throttle from 0 -> 10 msec/key, based on maximum vnode mailbox size 246 from 'riak at 10.47.109.202'
> 2014-08-07 05:47:00.465 UTC [debug] <0.9160.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
> 2014-08-07 05:47:00.465 UTC [debug] <0.9160.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange between {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'} and {919147514102638163401536164325474867830488825856,'riak at 10.46.109.206'}
> 2014-08-07 05:47:00.465 UTC [debug] <0.9160.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for preflist {890602560248518965780370444936484965102833893376,12}
> 2014-08-07 05:47:05.461 UTC [debug] <0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange: 890602560248518965780370444936484965102833893376 :: normal
> 2014-08-07 05:47:14.305 UTC [debug] <0.10096.567>@riak_kv_exchange_fsm:init:76 Starting exchange: {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:47:14.312 UTC [debug] <0.10096.567>@riak_kv_exchange_fsm:update_trees:143 Sending to {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:47:14.312 UTC [info] <0.588.0>@riak_kv_entropy_manager:perhaps_log_throttle_change:826 Changing AAE throttle from 10 -> 0 msec/key, based on maximum vnode mailbox size 1 from 'riak at 10.47.109.206'
> 2014-08-07 05:47:14.312 UTC [debug] <0.10096.567>@riak_kv_exchange_fsm:update_trees:144 Sending to {924856504873462002925769308203272848376019812352,'riak at 10.47.109.206'}
> 2014-08-07 05:47:14.312 UTC [debug] <0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree: (vnode)=890602560248518965780370444936484965102833893376 (preflist)={862057606394399768159204725547495062375178960896,12}
> 2014-08-07 05:47:15.540 UTC [debug] <0.10096.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
> 2014-08-07 05:47:15.540 UTC [debug] <0.10096.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange between {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'} and {924856504873462002925769308203272848376019812352,'riak at 10.47.109.206'}
> 2014-08-07 05:47:15.540 UTC [debug] <0.10096.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for preflist {862057606394399768159204725547495062375178960896,12}
> 2014-08-07 05:47:29.306 UTC [debug] <0.588.0>@riak_kv_entropy_manager:requeue_exchange:717 Requeue: {890602560248518965780370444936484965102833893376,924856504873462002925769308203272848376019812352,{873475587936047447207671013303091023466240933888,12}}
> 2014-08-07 05:47:44.307 UTC [debug] <0.588.0>@riak_kv_entropy_manager:requeue_exchange:717 Requeue: {890602560248518965780370444936484965102833893376,924856504873462002925769308203272848376019812352,{879184578706871286731904157180889004011771920384,12}}
> 2014-08-07 05:47:46.669 UTC [debug] <0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange: 890602560248518965780370444936484965102833893376 :: normal
> 2014-08-07 05:47:59.308 UTC [debug] <0.13725.567>@riak_kv_exchange_fsm:init:76 Starting exchange: {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:47:59.315 UTC [debug] <0.13725.567>@riak_kv_exchange_fsm:update_trees:143 Sending to {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'}
> 2014-08-07 05:47:59.315 UTC [debug] <0.13725.567>@riak_kv_exchange_fsm:update_trees:144 Sending to {924856504873462002925769308203272848376019812352,'riak at 10.47.109.206'}
> 2014-08-07 05:47:59.315 UTC [debug] <0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree: (vnode)=890602560248518965780370444936484965102833893376 (preflist)={884893569477695126256137301058686984557302906880,12}
> 2014-08-07 05:48:01.948 UTC [debug] <0.13725.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
> 2014-08-07 05:48:01.948 UTC [debug] <0.13725.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange between {890602560248518965780370444936484965102833893376,'riak at 10.46.109.201'} and {924856504873462002925769308203272848376019812352,'riak at 10.47.109.206'}
> 2014-08-07 05:48:01.948 UTC [debug] <0.13725.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for preflist {884893569477695126256137301058686984557302906880,12}
> 2014-08-07 05:48:14.284 UTC [error] <0.1697.0> gen_server <0.1697.0> terminated with reason: bad argument in call to eleveldb:async_write(#Ref<0.0.567.170046>, <<>>, [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}], []) in eleveldb:write/3 line 155
> 2014-08-07 05:48:14.284 UTC [info] <0.27496.566>@riak_kv_2i_aae:next_partition:160 Finished 2i repair:
> 	Total partitions: 1
> 	Finished partitions: 1
> 	Speed: 100
> 	Total 2i items scanned: 0
> 	Total tree objects: 0
> 	Total objects fixed: 0
> With errors:
> Partition: 319703483166135013357056057156686910549735243776
> Error: index_scan_timeout
> 
> 
> 2014-08-07 05:48:14.284 UTC [error] <0.1697.0> CRASH REPORT Process <0.1697.0> with 0 neighbours exited with reason: bad argument in call to eleveldb:async_write(#Ref<0.0.567.170046>, <<>>, [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}], []) in eleveldb:write/3 line 155 in gen_server:terminate/6 line 747
> 2014-08-07 05:48:14.284 UTC [error] <0.1692.0> Supervisor {<0.1692.0>,poolboy_sup} had child riak_core_vnode_worker started with {riak_core_vnode_worker,start_link,undefined} at <0.1697.0> exit with reason bad argument in call to eleveldb:async_write(#Ref<0.0.567.170046>, <<>>, [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}], []) in eleveldb:write/3 line 155 in context child_terminated

> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


-- 
Simon Effenberg | Site Op | mobile.international GmbH

Phone:    + 49. 30. 8109. 7173
M-Phone:  + 49. 151. 5266. 1558
Mail:     seffenberg at team.mobile.de
Web:      www.mobile.de

Marktplatz 1 | 14532 Europarc Dreilinden | Germany

______________________________________________________
Geschäftsführer: Malte Krüger
HRB Nr.: 18517 P, Amtsgericht Potsdam
Sitz der Gesellschaft: Kleinmachnow
______________________________________________________


More information about the riak-users mailing list