Merge error in bitcask data store

Engel Sanchez engel at basho.com
Wed Sep 3 13:00:27 EDT 2014


Hi Lucas.

Unfortunately, you have hit a bug in the Bitcask backend that we
overlooked.  The attached beam file is a patched version of the backend
code that should resolve the issue for you. To use it, stop each node, then
place riak_kv_bitcask_backend.beam in your basho-patches directory, then
restart it. The merges should complete without crashing. I believe in your
system the actual location is /usr/lib/riak/lib/basho-patches.

We will add this problem and the patch to the known issues information soon.

Please let us know if you come across any other issues with the merges.



On Wed, Sep 3, 2014 at 11:25 AM, Lucas Grijander <
lucasgrinjander69 at gmail.com> wrote:

> Hi.
>
> I am new in the list, so i don't know if this is  the right place to open
> this thread.
>
> Our riak cluster is composed by 4 nodes. The O.S is Ubuntu 14.04 and the
> version of riak is 2.0.0
>
> We are getting a lot of errors in the logs of ours riak nodes. The errors
> are like this:
>
> 2014-09-03 13:05:14.212 [error] <0.19152.3672> Failed to merge
> {["/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/14.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/13.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/12.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/11.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/10.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/9.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/8.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/7.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/6.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/5.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/4.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/3.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/2.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/1.bitcask.data"],["/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/14.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/13.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/12.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/11.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/10.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/9.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553...",...]}:
> {generic_failure,error,function_clause,[{riak_kv_bitcask_backend,key_transform_to_1,[{tombstone,<<2,0,4,109,116,116,108,50,48,49,52,48,57,48,50,50,50,53,48,50,53,48,56,102,56,102,102,101,54,49,49,97,101,51,99,52,52,55,101,55,55,100,99,50,100,49,52,56,51,57,55,48,50>>}],[{file,"src/riak_kv_bitcask_backend.erl"},{line,99}]},{bitcask,'-expiry_merge/4-fun-0-',7,[{file,"src/bitcask.erl"},{line,1912}]},{bitcask_fileops,fold_hintfile_loop,5,[{file,"src/bitcask_fileops.erl"},{line,660}]},{bitcask_fileops,fold_file_loop,8,[{file,"src/bitcask_fileops.erl"},{line,720}]},{bitcask_fileops,fold_hintfile,3,[{file,"src/bitcask_fileops.erl"},{line,624}]},{bitcask,expiry_merge,4,[{file,"src/bitcask.erl"},{line,1915}]},{bitcask,merge1,4,[{file,"src/bitcask.erl"},{line,686}]},{bitcask,merge,3,[{file,"src/bitcask.erl"},{line,566}]}]}
>
> It seems riak can not merge data in bitcask data store.
>
> This is the configuration of bitcask in ten minutes TTL:
>
> {<<"ten_minutes_ttl">>,riak_kv_bitcask_backend,
>             [{io_mode,erlang},
>              {expiry_grace_time,0},
>              {small_file_threshold,5242880},
>              {dead_bytes_threshold,4194304},
>              {frag_threshold,15},
>              {dead_bytes_merge_trigger,4194304},
>              {frag_merge_trigger,10},
>              {max_file_size,10485760},
>              {open_timeout,4},
>              {data_root,"/var/lib/riak/data/ten_minutes"},
>              {sync_strategy,none},
>              {merge_window,always},
>              {max_fold_age,-1},
>              {max_fold_puts,0},
>              {expiry_secs,660},
>              {require_hint_crc,true}]}
>
>
> As a result, the amount of used memory(RAM) keeps growing until the server
> run out of free memory.
>
> Could you give me some clue that it can point to the cause of the problem?
>
> Thanks in advance.
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140903/c209b7b7/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riak_kv_bitcask_backend.beam
Type: application/octet-stream
Size: 39812 bytes
Desc: not available
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140903/c209b7b7/attachment.beam>


More information about the riak-users mailing list