Inconsistent data when a node goes down

Luca Spiller luca at stackednotion.com
Mon Feb 28 13:29:41 EST 2011


Hi all,

I've come across some issues while testing what happens when failures happen
on our system, for example a machine failing. One of the (slightly scary)
issues I have come across is for a short while when a Riak node goes down,
data that is read from another node isn't always consistent. I have written
a small test script to demonstrate this issue:

https://gist.github.com/847749

Halfway through I switch off a node; here are the results:

Deleted 0
Wrote 100 454551
1298916758: 100 454551
1298916759: 100 454551
1298916760: 100 454551
1298916761: 100 454551
1298916762: 100 454551
1298916762: 100 454551
1298916763: 100 454551
1298916764: 100 454551  (Shutdown around here)
1298916765: 100 454551
1298916766: 99 460532
1298916767: 91 412241
1298916768: 100 454551
1298916769: 100 454551
1298916770: 100 454551
1298916771: 100 454551
1298916772: 100 454551
1298916773: 100 454551
1298916774: 100 454551
1298916775: 100 454551
1298916776: 100 454551
1298916777: 100 454551
^C1298916777: 100 454551
Deleted 100

Slightly more scary is that it appears to sometimes read old (deleted) data:

Deleted 0
Wrote 100 495792
1298916784: 100 495792
1298916785: 100 495792
1298916786: 100 495792
1298916786: 100 495792  (Shutdown around here)
1298916787: 100 495792
1298916788: 100 487322
1298916789: 100 495792
1298916790: 100 495792
1298916791: 100 495792
1298916792: 100 495792
1298916793: 100 495792
1298916794: 100 495792
1298916795: 100 495792
^C1298916796: 100 495792
1298916797: 100 495792
Deleted 100

This is using the Ripple library (0.8.3) talking directly to the local node,
however I believe the same problem is happening when using the Erlang PBC
library. This problem seems to be exacerbated when there are larger amounts
of data being stored in Riak, and the eventual consistency takes longer to
occur.

I am quite puzzled as to why this is happening, I could kind of understand
if data went missing, but the eventual consistency is what puzzles me, I
only have two nodes, so why does the data eventually sort itself out?
Secondly why does this still happen even with W and DW set to 3
(I originally had the script using the default values, but thought I would
try this)?

Both of the nodes are running Riak 0.14.0, here are the relavent configs:

https://gist.github.com/847756

Apologies if I am just doing something stupid, it has been a rather long day
:)

Regards,

Luca Spiller
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20110228/2d2f2216/attachment.html>


More information about the riak-users mailing list