[Basho Riak] Fail To Update Document Repeatly With Cluster of 5 Nodes

my hue tranmyhue.grackle at gmail.com
Mon Feb 6 13:02:50 EST 2017


Dear Riak Team,

I and my team used riak as database for my production with an cluster
including 5 nodes.
While production run, we meet an critical bug that is sometimes fail to
update document.
I and my colleagues performed debug and detected an issue with the scenario
as follow:

+  fetch document
+  change value of document
+  update document

Repeat about 10 times and will meet fail. With the document is updated
continually,
sometimes will face update fail.

The first time,  5 nodes of cluster we used riak version 2.1.1.
After meet above bug, we upgraded to use riak version 2.2.0 and this issue
still occurs.

Via many time test,  debug using  Tcpdump at riak node :

*tcpdump -A -ttt  -i {interface} src host {host} and dst port {port} *

And together with the command:

*riak-admin status | grep "node_puts_map\| node_puts_map_total\|
node_puts_total\| vnode_map_update_total\| vnode_puts_total\"*

we  got that the riak server already get the update request.
However, do not know why riak backend fail to update document.
At the fail time,  from riak server log everything is ok.

Then we removed cluster and use a single riak server,  and see that above
bug never happen.

For that reason, think that is only happen with cluster work. We took
research on basho riak document and our riak configure
seems that like suggestion from document.  We totally blocked on this issue
and hope that can get support from you
so that can obtain a stable work from riak database for our production.
Thank you so much.  Hope that can get your reply soon.


* The following is our riak node information :

Riak version:  2.2.0
OS :  CentOS Linux release 7.2.1511
Cpu :  4 core
Memory : 4G
Riak configure : the attached file "riak.conf"

*Note :*

- We mostly using default configure of riak configure except that  we used
storage backend is multi

storage_backend = multi
multi_backend.bitcask_mult.storage_backend = bitcask
multi_backend.bitcask_mult.bitcask.data_root = /var/lib/riak/bitcask_mult
multi_backend.default = bitcask_mult

------------------------------------------------------------
-----------------------------------------------------------------

- Bucket type created with the following command:

riak-admin bucket-type create dev_restor '{"props":{"backend":"bitcask_
mult","datatype":"map"}}'
riak-admin bucket-type activate dev_restor

------------------------------------------------------------
-----------------------------------------------------------------

- Bucket Type Status :

>> riak-admin bucket-type status dev_restor

dev_restor is active
young_vclock: 20
w: quorum
small_vclock: 50
rw: quorum
r: quorum
pw: 0
precommit: []
pr: 0
postcommit: []
old_vclock: 86400
notfound_ok: true
n_val: 3
linkfun: {modfun,riak_kv_wm_link_walker,mapreduce_linkfun}
last_write_wins: false
dw: quorum
dvv_enabled: true
chash_keyfun: {riak_core_util,chash_std_keyfun}
big_vclock: 50
basic_quorum: false
backend: <<"bitcask_mult">>
allow_mult: true
datatype: map
active: true
claimant: 'riak-node1 at 64.137.190.244'

------------------------------------------------------------
-----------------------------------------------------------------

- Bucket Property :

{"props":{"name":"menu","active":true,"allow_mult":true,"
backend":"bitcask_mult","basic_quorum":false,"big_vcloc
k":50,"chash_keyfun":{"mod":"riak_core_util","fun":"chash_
std_keyfun"},"claimant":"riak-node1 at 64.137.190.244","datatyp
e":"map","dvv_enabled":true,"dw":"quorum","last_write_wins"
:false,"linkfun":{"mod":"riak_kv_wm_link_walker","fun":"
mapreduce_linkfun"},"n_val":3,"name":"menu","notfound_ok":
true,"old_vclock":86400,"postcommit":[],"pr":0,"
precommit":[],"pw":0,"r":"quorum","rw":"quorum","search_inde
x":"menu_idx","small_vclock":50,"w":"quorum","young_vclock":20}}


------------------------------------------------------------
-----------------------------------------------------------------

- Member status :

>> riak-admin member-status

================================= Membership ==============================
====
Status     Ring    Pending    Node
------------------------------------------------------------
-------------------
valid      18.8%      --      'riak-node1 at 64.137.190.244'
valid      18.8%      --      'riak-node2 at 64.137.247.82'
valid      18.8%      --      'riak-node3 at 64.137.162.64'
valid      25.0%      --      'riak-node4 at 64.137.161.229'
valid      18.8%      --      'riak-node5 at 64.137.217.73'
------------------------------------------------------------
-------------------
Valid:5 / Leaving:0 / Exiting:0 / Joining:0 / Down:0


------------------------------------------------------------
-----------------------------------------------------------------

- Ring

>> riak-admin status | grep ring

ring_creation_size : 64
ring_members : ['riak-node1 at 64.137.190.244','riak-node2 at 64.137.247.82', '
riak-node3 at 64.137.162.64','riak-node4 at 64.137.161.229', '
riak-node5 at 64.137.217.73']
ring_num_partitions : 64
ring_ownership : <<"[{'riak-node2 at 64.137.247.82',12},\n {'
riak-node5 at 64.137.217.73',12},\n {'riak-node1 at 64.137.190.244',12},\n {'
riak-node3 at 64.137.162.64',12},\n {'riak-node4 at 64.137.161.229',16}]">>
rings_reconciled : 0
rings_reconciled_total : 31

------------------------------------------------------------
-----------------------------------------------------------------

* The riak client :

+ riak-erlang-client:  https://github.com/basho/riak-erlang-client
+ release :   2.4.2

------------------------------------------------------------
-----------------------------------------------------------------

* Riak client API used:

+ Insert/Update:

riakc_pb_socket:update_type(Pid, {Bucket-Type, Bucket}, Key,
riakc_map:to_op(Map), []).

+ Fetch :

riakc_pb_socket:fetch_type(Pid, {BucketType, Bucket}, Key, []).

------------------------------------------------------------
-----------------------------------------------------------------

* Step to perform an  update :

- Fetch document
- Update document

------------------------------------------------------------
-----------------------------------------------------------------

*  Data got from fetch_type:

{map,  [{{<<"account_id">>,register},
<<"accounta25a424b8484181e8ba1bec25bf7c491">>},
{{<<"created_by_id">>,register},
<<"accounta25a424b8484181e8ba1bec25bf7c491">>},
{{<<"created_time_dt">>,register},<<"2017-01-27T03:34:04Z">>},
{{<<"currency">>,register},<<"cad">>},
{{<<"end_time">>,register},<<"dont_use">>},
{{<<"id">>,register},<<"menufe89488afa948875cab6b0b18d579f21">>},
{{<<"maintain_mode_b">>,register},<<"false">>},
{{<<"menu_category_revision_id">>,register},
<<"0-634736bc14e0bd3ed7e3fe0f1ee64443">>},
{{<<"name">>,register},<<"fullmenu">>}, {{<<"order_i">>,register},<<"0">>},
{{<<"rest_location_p">>,register},
<<"10.844117421366443,106.63982392275398">>},
{{<<"restaurant_id">>,register},
<<"rest848e042b3a0488640981c8a6dc4a8281">>},
{{<<"restaurant_status_id">>,register},<<"inactive">>},
{{<<"start_time">>,register},<<"dont_use">>},
{{<<"status_id">>,register},<<"hide">>}, {{<<"updated_by_id">>,register},
<<"accounta25a424b8484181e8ba1bec25bf7c491">>},
{{<<"updated_time_dt">>,register},<<"2017-02-06T17:22:39Z">>}],
 [],
 [],
<<131,108,0,0,0,3,104,2,109,0,0,0,12,39,21,84,209,219,42,57,233,0,0,156,252,97,34,104,2,109,0,0,0,12,132,107,248,226,103,5,182,208,0,0,118,2,97,40,104,2,109,0,0,0,12,137,252,139,186,176,202,25,96,0,0,195,164,97,54,106>>}


*  Update with update_type

Below is Map data before using riakc_map:to_op(Map) :

{map, [] ,
 [{{<<"account_id">>,register},{register,<<>>,<<"accounta25a424b8484181e8ba1bec25bf7c491">>}},{{<<"created_by_id">>,register},{register,<<>>,<<"accounta25a424b8484181e8ba1bec25bf7c491">>}},{{<<"created_time_dt">>,register},{register,<<>>,<<"2017-01-27T03:34:04Z">>}},{{<<"currency">>,register},{register,<<>>,<<"cad">>}},{{<<"end_time">>,register},{register,<<>>,<<"dont_use">>}},{{<<"id">>,register},{register,<<>>,<<"menufe89488afa948875cab6b0b18d579f21">>}},{{<<"maintain_mode_b">>,register},{register,<<>>,<<"false">>}},{{<<"menu_category_revision_id">>,register},{register,<<>>,<<"0-634736bc14e0bd3ed7e3fe0f1ee64443">>}},{{<<"name">>,register},{register,<<>>,<<"fullmenu">>}},{{<<"order_i">>,register},{register,<<>>,<<"0">>}},{{<<"rest_location_p">>,register},{register,<<>>,<<"10.844117421366443,106.63982392275398">>}},{{<<"restaurant_id">>,register},{register,<<>>,<<"rest848e042b3a0488640981c8a6dc4a8281">>}},{{<<"restaurant_status_id">>,register},{register,<<>>,<<"inactive">>}},{{<<"start_time">>,register},{register,<<>>,<<"dont_use">>}},{{<<"status_id">>,register},{register,<<>>,<<"show">>}},{{<<"updated_by_id">>,register},{register,<<>>,<<"accounta25a424b8484181e8ba1bec25bf7c491">>}},{{<<"updated_time_dt">>,register},{register,<<>>,<<"2017-02-06T17:22:39Z">>}}],
 [] ,
 <<131,108,0,0,0,3,104,2,109,0,0,0,12,39,21,84,209,219,42,57,233,0,0,156,252,97,34,104,2,109,0,0,0,12,132,107,248,226,103,5,182,208,0,0,118,2,97,39,104,2,109,0,0,0,12,137,252,139,186,176,202,25,96,0,0,195,164,97,53,106>>
}




-

Best regards,
Hue Tran
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20170207/a4c7a658/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riak.conf
Type: application/octet-stream
Size: 18972 bytes
Desc: not available
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20170207/a4c7a658/attachment.conf>


More information about the riak-users mailing list