N = 3 and RW = 2 not finding some keys

Nicholas Adams nicholas.adams at tiot.jp
Fri May 18 10:42:33 EDT 2018


Sorry for being late to the party but have you tried repairing all partitions? https://docs.basho.com/riak/kv/2.1.4/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node

This forces every single piece of information to be read and thus causes read repair to kick in for any partitions that register less than 3 copies.

Best regards,

Nicholas

From: riak-users <riak-users-bounces at lists.basho.com> On Behalf Of Guido Medina
Sent: 18 May 2018 22:52
To: riak-users <riak-users at lists.basho.com>
Subject: Re: N = 3 and RW = 2 not finding some keys

Yes, but we haven't set back R = 2, we are currently with R = 3 until we resolve the issue.
I'm in the middle of writing something that will read every single key for the cluster using our relation DB IDs (each key has a counterpart on DB)

I'm guessing that 2i streaming won't help? we need to make sure we are fully consistent (for RW = 2) so I have to find a way to repair everything before turning R to 2.

After reading the object yes every request after was successful, but we even had a weird situation where after writing key with W = 2 we couldn't find with R = 2, not good for us.

Guido.
On 18/05/18 14:34, Bryan Hunt wrote:

Russell, Good question. I’m guessing they are iterating, and requesting a different object for each request?



Guido, given the behaviour you initially described, before applying the configuration I suggested - did you receive a successful response upon subsequent requests for the same object ?



On 18 May 2018, at 13:13, Russell Brown <russell.brown at icloud.com><mailto:russell.brown at icloud.com> wrote:



But why isn’t read repair “working”?



On 18 May 2018, at 11:07, Bryan Hunt <bryan.hunt at erlang-solutions.com><mailto:bryan.hunt at erlang-solutions.com> wrote:



Of course, AAE will eventually repair the missing object replicas but it seems like you need something more immediate.



On 18 May 2018, at 11:00, Bryan Hunt <bryan.hunt at erlang-solutions.com><mailto:bryan.hunt at erlang-solutions.com> wrote:



Hi Guido,



You should attempt to change the bucket property ‘notfound_ok’ from the default of ‘true' to ‘false'.



I.e



curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: application/json" -d '{"props":{"notfound_ok": false}}'



This makes GET operations for non-existent keys slower as it forces an internal GET for each of the three copies.



https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok



From what you describe, it sounds like only a single copy (out of the original three), somehow remain present in your cluster.



Best Regards,



Bryan Hunt



On 17 May 2018, at 15:42, Guido Medina <gmedina at temetra.com><mailto:gmedina at temetra.com> wrote:



Hi all,



After some big rebalance of our cluster some keys are not found anymore unless we set R = 3, we had N = 3 and R = W = 2



Is there any sort of repair that would correct such situation for Riak 2.2.3, this is really driving us nuts.



Any help will be truly appreciated.



Kind regards,

Guido.

_______________________________________________

riak-users mailing list

riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>

http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





_______________________________________________

riak-users mailing list

riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>

http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com







_______________________________________________

riak-users mailing list

riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>

http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20180518/8525987b/attachment.html>


More information about the riak-users mailing list