Many "Unrecognized message" errors in the logs while nodes are leaving

Chris Read chris.read at gmail.com
Mon Sep 29 11:15:28 EDT 2014


We were getting upto ~20 messages per second while nodes were leaving
the cluster. We went from 19 nodes down to 15. Ring size is 64, using
bitcask as the store.

Requests on this cluster were averaging around 10k/sec, spiking to
25k/sec occasionally. Request mix is 99% GET with a few PUT and DEL.
We don't use mr, search or list at all...

Chris



On Sat, Sep 27, 2014 at 8:39 AM, Sean Cribbs <sean at basho.com> wrote:
> Hi Chris,
>
> Those log messages are mostly harmless -- they don't indicate any
> specific thing going wrong. They only indicate that the process
> handling the client connection decided to stop waiting for additional
> internal protocol messages and replied to the client. Since the
> protocol state machine is now outside of the request handler, it
> doesn't recognize those internal responses any more.
>
> Those log messages were added so that we (Basho) could tell if we had
> egregiously missed some message in an internal protocol, but again,
> other than being a little spammy they are harmless.
>
> It would be nice to know what your request mix looks like, i.e. are
> you doing list-keys, mapreduce, search, etc? How much of each? How
> often do you get those log messages?
>
> On Fri, Sep 26, 2014 at 4:03 PM, Chris Read <chris.read at gmail.com> wrote:
>> Greetings...
>>
>> I started removing a few nodes from a cluster today, and started to
>> get a lot of the following errors in the logs:
>>
>> @riak_api_pb_server:handle_info:331 Unrecognized message
>>
>> Message is either '{<large int>,ok}' or '{<large int>,{error,timeout}}'
>>
>> We're running Riak 2.0 on Erlang R16B02-basho5, both built from
>> source. A quick Google hinted that it may be a regression of
>> https://github.com/basho/riak_kv/pull/366
>>
>> Chris
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> --
> Sean Cribbs <sean at basho.com>
> Software Engineer
> Basho Technologies, Inc.
> http://basho.com/




More information about the riak-users mailing list