Anti-Entropy Sudden Exchange

Justin Long justinlong at outlook.com
Thu Jan 2 19:39:02 EST 2014


Ok here’s an update on the issue, I’m really hoping someone can chime in. I disabled Anti-Entropy, restarted Riak, and then tried to run a repair on indexes. After running the command [riak_kv_vnode:repair(P) || P <- Partitions]. Riak then returned the error 

** exception exit: {timeout,{gen_server,call,
                                        [{riak_core_vnode_manager,'riak at 192.168.3.2'},
                                         {repair,riak_kv,
                                                 {riak_kv_vnode,22835963083295358096932575511191922182123945984},
                                                 {riak_kv_vnode,repair_filter}}]}}
     in function  gen_server:call/2 (gen_server.erl, line 180)

We then tried to use the Basho Riak Data Migrator tool to back up the most recent data on the node, but it takes 2 minutes to export a simple KV object (and these are less than 32kb in size).  Thanks for your help.

FYI Using Riak 1.4.6



On Jan 2, 2014, at 11:31 AM, Justin Long <justinlong at outlook.com> wrote:

> Hi everyone,
> 
> Experiencing some weird behaviour in a cluster of 3 nodes. As of this morning, we noticed a massive performance lag and when checking the logs Active Anti-Entropy Exchange suddenly began initiating a huge series of transfers. Here’s what we see in the logs:
> 
> 2014-01-02 19:26:34.396 [info] <0.1571.1800>@riak_kv_exchange_fsm:key_exchange:206 Repaired 882 keys during active anti-entropy exchange of {228359630832953580969325755111919221821239459840,3} between {228359630832953580969325755111919221821239459840,'riak at 192.168.3.2'} and {251195593916248939066258330623111144003363405824,'riak at 192.168.3.3’}
> 
> When we check the logs on all 3 nodes we only had a few process crashes 10 hours ago, nothing in between that time and now. It all came on suddenly. Any idea if this is normal and will fix itself or if we need to intervene?
> 
> Thanks!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140102/ebdae6e9/attachment.html>


More information about the riak-users mailing list