Absolute consistency

Thomas Bakketun Thomas.Bakketun at Copyleft.no
Thu Jan 5 11:07:02 EST 2012


Riak is based on eventual consistency concept, but sometimes it's
desirable to have absolute consistency. I though it would be possible to
implement this on top of Riak with the help of pr and wr parameters of
fetch and store, but when testing this I sometimes get worrying results.

I have at test setup with Riak nodes in three virtual machines running
on my desktop machine. The test bucket has n_val=3 and allow_mult=true.

On my desktop machine I constantly read from every node like this:
watch curl -v "''"

When I pause two of the virtual machines, the remaining one starts to
fail with error message "PR-value unsatisfied: 1/2", just as expected.
But occasionally it will returns 200 OK for about 10 seconds, before it
starts returning "PR-value unsatisfied: 1/2". I can't predictably
reproduce this situation, but it's seems to happen when I resume a
machine that was paused when the other machines where running too.

Pausing virtual machines to simulate downtime might be a bit
unrealistic. Still it concerns me that Riak can return stale data. Here
is an example:

First I wrote a value when all nodes where up. Then I pause one of the
nodes and wrote a new value. Then I paused the two remaining nodes and
resume the first node. For a few seconds the first node then returned
200 OK with the old value when though I supplied pr=quorum.

I have also been able to do successful writes with w, dw and pw=quorum
with only one machine running. I thing the node also returned 200 OK
when the object was read with pr=quorum right after.

The tests where done with Riak 1.0.0-1 running on Ubuntu 10.10 64 bit.

Thomas Bakketun, Senior software developer
T. +47 21 53 69 40
Copyleft Solutions

More information about the riak-users mailing list