Put failure: too many siblings

Vladyslav Zakhozhai v.zakhozhai at smartweb.com.ua
Fri Jun 3 05:55:45 EDT 2016


I have a trouble with PUT to Riak CS cluster. During this process I
periodically see the following message in Riak error.log:

2016-06-03 11:15:55.201 [error]
<0.15536.142>@riak_kv_vnode:encode_and_put:2253 Put failure: too many
siblings for object OBJECT_NAME (101)

and also

2016-06-03 12:41:50.678 [error]
<0.20448.515>@riak_api_pb_server:handle_info:331 Unrecognized message

Here OBJECT_NAME - is the name of object in Riak which has too many

I definitely sure that this objects are static. Nobody deletes is, nobody
rewrites it. I have no idea why more than 100 siblings of this object

The following effect of this issue occurs:

   1. Great amount of keys are loaded to RAM. I almost out of RAM (Do each
   sibling has it own key or key duplicate?).
   2. Nodes are slow - adding new nodes are too slow
   3. Presence of "too many siblings" affects ownership handoffs

So I have several questions:

   1. Do hinted or ownership handoffs can affect siblings count (I mean can
   siblings be created during ownership of hinted handoffs)
   2. Is there any workaround of this issue. Do I need remove siblings
   manually or it removes during merges, read repairs and so on

My configuration:

   1. riak from basho's packages - 2.1.3-1
   2. riak cs from basho's packages - 2.1.0-1
   3. 24 riak/riak-cs nodes
   4. 32 GB RAM per node
   5. AAE is disabled

I appreciate you help.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20160603/ba089c2d/attachment.html>

More information about the riak-users mailing list