Deleted keys come back

Kota Uenishi kota at basho.com
Wed Oct 7 20:11:10 EDT 2015


I'd also recommend to have "allow_mult" as true to prevent
resurrection caused by sloppy quorum, handoffs or any network
separation. Also, to make sure your data eventually consistent, R+W
should be > N. Your case is 1+1, not >2.

On Wed, Oct 7, 2015 at 10:19 PM, Dmitri Zagidulin <dzagidulin at basho.com> wrote:
> Hello,
>
> There are two things going on here: the W quorum value of the write and
> delete operations, and possibly the delete_mode setting.
>
> Let's walk through the scenario.
> You're writing to a 2 node cluster, two copies of each object (n_val=2),
> with your write quorum of 1 (W=1).
>
> So that's possibility #1 -- there's no guarantee that your writes succeed
> with both replicas. (It could've just written one, and the other one is
> missing).
>
> Then you're doing a List Keys (to delete the objects), which runs with an
> implicit quorum of R=1. (Meaning, it only contacts half of the replicas, and
> lists them.) So (if possibility #1 happened, above) the list keys could have
> not returned some keys, the first time around. (Because it may have
> contacted the partitions that had missing replicas).  Then you deleted, ran
> another List Keys, and that one could have returned the keys that it missed
> the first time.
>
> Possibility #2 -- your deletes are using W=1, meaning, they're only waiting
> for the delete operation from 1 replica to respond, before returning
> success. So, it's possible that a delete operation removed just one replica,
> but the second one still exists. And the second List Keys can now pick up
> the not-deleted replica.
>
> Possibility #3 -- by default, the delete_mode is set to keep deleted objects
> for 3 seconds.  So, if you ran your deletes, and then re-ran a List Keys
> before the 3 seconds expired, you could pick up some keys.
>
> The upshot of all this is:
> - Use W=2 when writing and deleting. (That is, your W value should be the
> same as your N value).
>
> - If that doesn't work, set delete_mode to 'immediate' in the config.
> Specifically, delete_mode is set in the advanced.config file
> (http://docs.basho.com/riak/latest/ops/advanced/configs/configuration-files/#Advanced-Configuration
> ). So, my advanced.config file looks like this:
>
> [
>   {riak_kv,
>     [
>       {delete_mode, immediate}
>     ]}
>   }
> ].
>
> Also, if you're deleting things for unit tests, there's an easier way.
> Instead of deleting the bucket object-by-object, you can just stop the node,
> and clear the bitcask (or leveldb) data directory. (That's going to get rid
> of all the data in the cluster, which is what you want to do for unit tests
> anyways.)
>
> You can learn more about these topics on the following pages:
> * http://docs.basho.com/riak/latest/ops/advanced/deletion/
> *
> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-October/006048.html
> (mailing list post introducing delete_mode)
> * http://basho.com/posts/technical/riaks-config-behaviors-part-3/  (the
> Tombstones section)
>
> Dmitri
>
>
>
>
> On Wed, Oct 7, 2015 at 1:35 PM, mtakahashi-ivi <mtakahashi at yourinventit.com>
> wrote:
>>
>> Hello,
>>
>> I'm using riak KV in 2 nodes cluster.
>> I inserted hundreds of key/value pair and deleted all keys in a bucket.
>> After above process, I can get some keys if I get list of keys in the
>> bucket.
>> Why those keys remain? How do I delete keys reliably?
>> If I increase number of nodes to 5 , I can delete all keys in the bucket
>> as
>> same way as I did.
>> My bucket property is the following.
>>
>> ----------------------
>> {
>>   "props": {
>>     "name": "BUCKET_A",
>>     "active": true,
>>     "allow_mult": false,
>>     "basic_quorum": false,
>>     "big_vclock": 50,
>>     "chash_keyfun": {
>>       "mod": "riak_core_util",
>>       "fun": "chash_std_keyfun"
>>     },
>>     "claimant": "riak at 172.17.0.171",
>>     "dvv_enabled": true,
>>     "dw": "quorum",
>>     "last_write_wins": false,
>>     "linkfun": {
>>       "mod": "riak_kv_wm_link_walker",
>>       "fun": "mapreduce_linkfun"
>>     },
>>     "n_val": 2,
>>     "notfound_ok": false,
>>     "old_vclock": 86400,
>>     "postcommit": [],
>>     "pr": 0,
>>     "precommit": [],
>>     "pw": 0,
>>     "r": 1,
>>     "rw": "quorum",
>>     "search_index": "BUCKET_A_INDEX",
>>     "small_vclock": 50,
>>     "w": 1,
>>     "young_vclock": 20
>>   }
>> }
>>
>>
>> Masanori Takahashi
>>
>>
>>
>> --
>> View this message in context:
>> http://riak-users.197444.n3.nabble.com/Deleted-keys-come-back-tp4033536.html
>> Sent from the Riak Users mailing list archive at Nabble.com.
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>



-- 
Kota UENISHI / @kuenishi
Basho Japan KK




More information about the riak-users mailing list