Merge error in bitcask data store

Engel Sanchez engel at basho.com
Wed Sep 3 16:06:56 EDT 2014


Bringing this back to the mailing list:  A new patch was required to fix
the Bitcask merge problems seen by Lucas. Here it is.

Again, it can be loaded by placing it in the basho-patches directory and
stopping then starting the node again. It could also be loaded without a
restart by issuing the load command in the Riak console after placing it in
the basho-patches directory. You would need to run bin/riak attach, the
issue the command:

> l(riak_kv_bitcask_backend).

That is a lowercase L up there, btw.  This should resolve the Bitcask merge
issues.


On Wed, Sep 3, 2014 at 1:00 PM, Engel Sanchez <engel at basho.com> wrote:

> Hi Lucas.
>
> Unfortunately, you have hit a bug in the Bitcask backend that we
> overlooked.  The attached beam file is a patched version of the backend
> code that should resolve the issue for you. To use it, stop each node, then
> place riak_kv_bitcask_backend.beam in your basho-patches directory, then
> restart it. The merges should complete without crashing. I believe in your
> system the actual location is /usr/lib/riak/lib/basho-patches.
>
> We will add this problem and the patch to the known issues information
> soon.
>
> Please let us know if you come across any other issues with the merges.
>
>
>
> On Wed, Sep 3, 2014 at 11:25 AM, Lucas Grijander <
> lucasgrinjander69 at gmail.com> wrote:
>
>> Hi.
>>
>> I am new in the list, so i don't know if this is  the right place to open
>> this thread.
>>
>> Our riak cluster is composed by 4 nodes. The O.S is Ubuntu 14.04 and the
>> version of riak is 2.0.0
>>
>> We are getting a lot of errors in the logs of ours riak nodes. The errors
>> are like this:
>>
>> 2014-09-03 13:05:14.212 [error] <0.19152.3672> Failed to merge
>> {["/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/14.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/13.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/12.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/11.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/10.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/9.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/8.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/7.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/6.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/5.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/4.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/3.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/2.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/1.bitcask.data"],["/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/14.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/13.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/12.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/11.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/10.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/9.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553...",...]}:
>> {generic_failure,error,function_clause,[{riak_kv_bitcask_backend,key_transform_to_1,[{tombstone,<<2,0,4,109,116,116,108,50,48,49,52,48,57,48,50,50,50,53,48,50,53,48,56,102,56,102,102,101,54,49,49,97,101,51,99,52,52,55,101,55,55,100,99,50,100,49,52,56,51,57,55,48,50>>}],[{file,"src/riak_kv_bitcask_backend.erl"},{line,99}]},{bitcask,'-expiry_merge/4-fun-0-',7,[{file,"src/bitcask.erl"},{line,1912}]},{bitcask_fileops,fold_hintfile_loop,5,[{file,"src/bitcask_fileops.erl"},{line,660}]},{bitcask_fileops,fold_file_loop,8,[{file,"src/bitcask_fileops.erl"},{line,720}]},{bitcask_fileops,fold_hintfile,3,[{file,"src/bitcask_fileops.erl"},{line,624}]},{bitcask,expiry_merge,4,[{file,"src/bitcask.erl"},{line,1915}]},{bitcask,merge1,4,[{file,"src/bitcask.erl"},{line,686}]},{bitcask,merge,3,[{file,"src/bitcask.erl"},{line,566}]}]}
>>
>> It seems riak can not merge data in bitcask data store.
>>
>> This is the configuration of bitcask in ten minutes TTL:
>>
>> {<<"ten_minutes_ttl">>,riak_kv_bitcask_backend,
>>             [{io_mode,erlang},
>>              {expiry_grace_time,0},
>>              {small_file_threshold,5242880},
>>              {dead_bytes_threshold,4194304},
>>              {frag_threshold,15},
>>              {dead_bytes_merge_trigger,4194304},
>>              {frag_merge_trigger,10},
>>              {max_file_size,10485760},
>>              {open_timeout,4},
>>              {data_root,"/var/lib/riak/data/ten_minutes"},
>>              {sync_strategy,none},
>>              {merge_window,always},
>>              {max_fold_age,-1},
>>              {max_fold_puts,0},
>>              {expiry_secs,660},
>>              {require_hint_crc,true}]}
>>
>>
>> As a result, the amount of used memory(RAM) keeps growing until the
>> server run out of free memory.
>>
>> Could you give me some clue that it can point to the cause of the problem?
>>
>> Thanks in advance.
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140903/6f640efc/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riak_kv_bitcask_backend.beam
Type: application/octet-stream
Size: 39756 bytes
Desc: not available
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140903/6f640efc/attachment.beam>


More information about the riak-users mailing list