Riak 2.0.4 AAE spinning

Chris Read chris.read at gmail.com
Wed Feb 18 12:28:22 EST 2015


We recently upgraded to Riak 2.0.4 and started seeing the thousands of the
following error in one of our nodes:

2015-02-18 07:48:16.652 [error] <0.23966.264> CRASH REPORT Process
<0.23966.264> with 0 neighbours exited with reason: no match of right hand
value {error,{db_open,"IO error: lock
/srv/riak-0/data/anti_entropy/548063113999088594326381812268606132370974703616/LOCK:
already held by process"}} in hashtree:new_segment_store/2 line 555 in
gen_server:init_it/6 line 328
2015-02-18 07:48:16.652 [info]
<0.27823.1997>@riak_kv_vnode:maybe_create_hashtrees:178
riak_kv/548063113999088594326381812268606132370974703616: unable to start
index_hashtree: {error,{{badmatch,{error,{db_open,"IO error: lock
/srv/riak-0/data/anti_entropy/548063113999088594326381812268606132370974703616/LOCK:
already held by
process"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,555}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,224}]},{riak_kv_index_hashtree,do_new_tree,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,560}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{riak_kv_index_hashtree,init_trees,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,430}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,255}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}

This correlates with a massive spike in GET requests on the three nodes
that own the affected vnode.

Anyone have any ideas?

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20150218/688820cc/attachment-0002.html>


More information about the riak-users mailing list