Keys not listed during vnode transfers

Daniel Iwan iwan.daniel at
Tue Apr 2 19:10:26 EDT 2013

Hi Mark

Thanks for info. For the moment I thought we were the only one experiencing

At the moment the only workaround for us is to wait until vnode transfer is
I suspect attaching a node also will trigger that behaviour. Considering
that expanding cluster from 3 to 4 nodes can take quite some time
the fact that we are not able to receive all keys via 2i means system is
not available during attaching a node(s).

I'll keep an eye on the issue and I'll put as much information I can.


On 2 April 2013 22:46, Mark Phillips <mark at> wrote:

> Hi Daniel,
> Sorry for the delay here.
> It looks like it's going to take a more investigation to nail down the
> cause. Engel just opened an issue you can keep an eye on:
> Thanks for reporting. Feel free to add any relevant information to the
> issue.
> Mark
> On Thursday, March 14, 2013, Daniel Iwan wrote:
>> Maybe someone from Basho could shed some light on that issue?
>> Regards
>> Daniel
>> On 12 March 2013 11:55, Daniel Iwan <iwan.daniel at> wrote:
>>> Just to add to that.
>>> Further test shows that 2i searches aso suffer form the problem of not
>>> showing all results durring active vnode transfer.
>>> Is this a known issue with Riak 1.2.1? Has it been solved in 1.3?
>>> Anyone else experienced that? I guess attaching a node would trigger
>>> that as well, maybe in less severe way.
>>> Also I've read somewhere that you should attach one node at a time to
>>> Riak cluster, and wait until vnode transfer completes.
>>> I think it's no longer true in 1.2 since you have a plan that you
>>> commit, but attaching a node and shuffling vnodes will cause problem
>>> described
>>> What is the solution here? Waiting until vnode transfer finishes is not
>>> acceptable (availability) and recent findings show it may take a while on
>>> big clusters.
>>> Regards
>>> Daniel
>>> On 11 March 2013 23:06, Daniel Iwan <iwan.daniel at> wrote:
>>>> I'm aware that listing keys is not for production.
>>>> I'm using it mainly during testing, which started to be unreliable
>>>> after changes described above.
>>>> What I was not expecting at all was that some of the keys won't be
>>>> listed.
>>>> I'm not sure if that is stated in documentation to tell the truth.
>>>> To me it looks like it should be called 'listSomeKeys'
>>>> About alternatives.
>>>> Some time ago I did comparison of listing and 2i and MapReduce and
>>>> surprisingly listing was quickest.
>>>> I'm not sure why it was happening. I did similar tests today and what I
>>>> got is:
>>>> 1000 keys, grouped with 2i into 10 equal groups, each value < 1kB
>>>> Listing:
>>>> - via listkeys  276ms
>>>> - via keyindex $key: 255ms
>>>> - via 2i (10 calls), total 2480ms
>>>> So for that simple case 2i is 10 times slower.
>>>> Further test shows that 100k keys (100 groups, 1000 each) gives query
>>>> response between 250-5500ms.
>>>> Not good. It's almost silly NOT to use listing keys.
>>>> I may need to do that test on different hardware to compare. At the
>>>> moment I'm using just one 7200rpm HDD for Riak db.
>>>> Daniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>

More information about the riak-users mailing list