Understanding read_repairs

Belai Beshah belai.beshah at nwgeo.com
Thu Feb 21 18:48:24 EST 2013



Hi All,


We are evaluating Riak to see if it can be used to cache large blobs of data. Here is our test cluster setup:


 *   six Ubuntu LTS 12.04 dedicated nodes with 8 core 2.6 Ghz CPU, 32 GB RAM, 3.6T disk

 *

{pb_backlog, 64},

 *

{ring_creation_size, 256},

 *

{default_bucket_props, [{n_val, 2}, {allow_mult,false},{last_write_wins,true}]},

 *

using bitcask as the backend


Everything else default except the above. There is an HAProxy load balancer infront of the nodes that the clients talk too configured according to the basho wiki. Due to the nature of the application we are integrating we do about 1200/s writes of approximately 40-50KB each and read them back almost immediately. We noticed a lot of read repairs and since that was one of the things that could indicate performance problem we go worried. So we wrote a simple java client application that simulates our use case. The test program is dead simple:

 *   generate keys using random UUID and value using Apache commons RandomStringUtils

 *   create a thread pool of 5 and store key/value using “bucket.store()”

 *   read the values back using “bucket.fetch()” multiple times

I could provide the spike code if needed. What we noticed is that we get a lot of read repairs all over the place. We even made it use a single thread to read/write, played with the write/read quorum and even put a delay of 5 minutes between the writes before the reads start to give the cluster time to be eventually consistent. Nothing helps, we always see a lot of read repairs, sometime as many as the number of inserts. The good thing is that in all of these tests we have not seen any read failures. Performance is also not bad, a few maxs here and there we don't like but 90% looks good. Even when we killed a node, the reads are still successful.


We are wondering what the expected ratio of read repairs is and what is a reasonable time for the cluster not to restore to read_repair to fulfill a read request or is there something we are missing in our setup.


Thanks

Belai
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20130221/1f57b78a/attachment.html>


More information about the riak-users mailing list