Keys not listed during vnode transfers

Daniel Iwan iwan.daniel at gmail.com
Tue Apr 2 19:10:26 EDT 2013


Hi Mark

Thanks for info. For the moment I thought we were the only one experiencing
this.

At the moment the only workaround for us is to wait until vnode transfer is
finished.
I suspect attaching a node also will trigger that behaviour. Considering
that expanding cluster from 3 to 4 nodes can take quite some time
the fact that we are not able to receive all keys via 2i means system is
not available during attaching a node(s).

I'll keep an eye on the issue and I'll put as much information I can.

Daniel.






On 2 April 2013 22:46, Mark Phillips <mark at basho.com> wrote:

> Hi Daniel,
>
> Sorry for the delay here.
>
> It looks like it's going to take a more investigation to nail down the
> cause. Engel just opened an issue you can keep an eye on:
>
> https://github.com/basho/riak/issues/305
>
> Thanks for reporting. Feel free to add any relevant information to the
> issue.
>
> Mark
>
> On Thursday, March 14, 2013, Daniel Iwan wrote:
>
>> Maybe someone from Basho could shed some light on that issue?
>>
>> Regards
>> Daniel
>>
>>
>> On 12 March 2013 11:55, Daniel Iwan <iwan.daniel at gmail.com> wrote:
>>
>>> Just to add to that.
>>> Further test shows that 2i searches aso suffer form the problem of not
>>> showing all results durring active vnode transfer.
>>> Is this a known issue with Riak 1.2.1? Has it been solved in 1.3?
>>>
>>> Anyone else experienced that? I guess attaching a node would trigger
>>> that as well, maybe in less severe way.
>>> Also I've read somewhere that you should attach one node at a time to
>>> Riak cluster, and wait until vnode transfer completes.
>>> I think it's no longer true in 1.2 since you have a plan that you
>>> commit, but attaching a node and shuffling vnodes will cause problem
>>> described
>>>
>>> What is the solution here? Waiting until vnode transfer finishes is not
>>> acceptable (availability) and recent findings show it may take a while on
>>> big clusters.
>>>
>>> Regards
>>> Daniel
>>>
>>>
>>>
>>> On 11 March 2013 23:06, Daniel Iwan <iwan.daniel at gmail.com> wrote:
>>>
>>>> I'm aware that listing keys is not for production.
>>>> I'm using it mainly during testing, which started to be unreliable
>>>> after changes described above.
>>>>
>>>> What I was not expecting at all was that some of the keys won't be
>>>> listed.
>>>> I'm not sure if that is stated in documentation to tell the truth.
>>>> To me it looks like it should be called 'listSomeKeys'
>>>>
>>>> About alternatives.
>>>> Some time ago I did comparison of listing and 2i and MapReduce and
>>>> surprisingly listing was quickest.
>>>> I'm not sure why it was happening. I did similar tests today and what I
>>>> got is:
>>>>
>>>> 1000 keys, grouped with 2i into 10 equal groups, each value < 1kB
>>>> Listing:
>>>> - via listkeys  276ms
>>>> - via keyindex $key: 255ms
>>>> - via 2i (10 calls), total 2480ms
>>>>
>>>> So for that simple case 2i is 10 times slower.
>>>> Further test shows that 100k keys (100 groups, 1000 each) gives query
>>>> response between 250-5500ms.
>>>> Not good. It's almost silly NOT to use listing keys.
>>>>
>>>> I may need to do that test on different hardware to compare. At the
>>>> moment I'm using just one 7200rpm HDD for Riak db.
>>>>
>>>> Daniel
>>>>
>>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130403/08345130/attachment.html>


More information about the riak-users mailing list