Merge error in bitcask data store

Tom Santero tsantero at gmail.com
Wed Sep 3 17:46:31 EDT 2014


Hi Engel,

Thanks for the speedy patch and bringing this back to the list. I see you
already created an issue on GitHub [0] and tagged it with the 2.0.1
milestone.

Can we expect a 2.0.1 release forthcoming? I was planning on upgrading Riak
today to 2.0, but since I use Bticask I'm affected by this. I'd prefer not
to upgrade to 2.0 in production until this is patched and packaged in a
proper release.

Thanks,
Tom

[0] https://github.com/basho/bitcask/issues/185


On Wed, Sep 3, 2014 at 4:06 PM, Engel Sanchez <engel at basho.com> wrote:

> Bringing this back to the mailing list:  A new patch was required to fix
> the Bitcask merge problems seen by Lucas. Here it is.
>
> Again, it can be loaded by placing it in the basho-patches directory and
> stopping then starting the node again. It could also be loaded without a
> restart by issuing the load command in the Riak console after placing it in
> the basho-patches directory. You would need to run bin/riak attach, the
> issue the command:
>
> > l(riak_kv_bitcask_backend).
>
> That is a lowercase L up there, btw.  This should resolve the Bitcask
> merge issues.
>
>
> On Wed, Sep 3, 2014 at 1:00 PM, Engel Sanchez <engel at basho.com> wrote:
>
>> Hi Lucas.
>>
>> Unfortunately, you have hit a bug in the Bitcask backend that we
>> overlooked.  The attached beam file is a patched version of the backend
>> code that should resolve the issue for you. To use it, stop each node, then
>> place riak_kv_bitcask_backend.beam in your basho-patches directory, then
>> restart it. The merges should complete without crashing. I believe in your
>> system the actual location is /usr/lib/riak/lib/basho-patches.
>>
>> We will add this problem and the patch to the known issues information
>> soon.
>>
>> Please let us know if you come across any other issues with the merges.
>>
>>
>>
>> On Wed, Sep 3, 2014 at 11:25 AM, Lucas Grijander <
>> lucasgrinjander69 at gmail.com> wrote:
>>
>>> Hi.
>>>
>>> I am new in the list, so i don't know if this is  the right place to
>>> open this thread.
>>>
>>> Our riak cluster is composed by 4 nodes. The O.S is Ubuntu 14.04 and the
>>> version of riak is 2.0.0
>>>
>>> We are getting a lot of errors in the logs of ours riak nodes. The
>>> errors are like this:
>>>
>>> 2014-09-03 13:05:14.212 [error] <0.19152.3672> Failed to merge
>>> {["/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/14.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/13.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/12.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/11.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/10.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/9.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/8.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/7.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/6.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/5.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/4.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/3.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/2.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/1.bitcask.data"],["/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/14.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/13.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/12.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/11.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/10.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/9.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553...",...]}:
>>> {generic_failure,error,function_clause,[{riak_kv_bitcask_backend,key_transform_to_1,[{tombstone,<<2,0,4,109,116,116,108,50,48,49,52,48,57,48,50,50,50,53,48,50,53,48,56,102,56,102,102,101,54,49,49,97,101,51,99,52,52,55,101,55,55,100,99,50,100,49,52,56,51,57,55,48,50>>}],[{file,"src/riak_kv_bitcask_backend.erl"},{line,99}]},{bitcask,'-expiry_merge/4-fun-0-',7,[{file,"src/bitcask.erl"},{line,1912}]},{bitcask_fileops,fold_hintfile_loop,5,[{file,"src/bitcask_fileops.erl"},{line,660}]},{bitcask_fileops,fold_file_loop,8,[{file,"src/bitcask_fileops.erl"},{line,720}]},{bitcask_fileops,fold_hintfile,3,[{file,"src/bitcask_fileops.erl"},{line,624}]},{bitcask,expiry_merge,4,[{file,"src/bitcask.erl"},{line,1915}]},{bitcask,merge1,4,[{file,"src/bitcask.erl"},{line,686}]},{bitcask,merge,3,[{file,"src/bitcask.erl"},{line,566}]}]}
>>>
>>> It seems riak can not merge data in bitcask data store.
>>>
>>> This is the configuration of bitcask in ten minutes TTL:
>>>
>>> {<<"ten_minutes_ttl">>,riak_kv_bitcask_backend,
>>>             [{io_mode,erlang},
>>>              {expiry_grace_time,0},
>>>              {small_file_threshold,5242880},
>>>              {dead_bytes_threshold,4194304},
>>>              {frag_threshold,15},
>>>              {dead_bytes_merge_trigger,4194304},
>>>              {frag_merge_trigger,10},
>>>              {max_file_size,10485760},
>>>              {open_timeout,4},
>>>              {data_root,"/var/lib/riak/data/ten_minutes"},
>>>              {sync_strategy,none},
>>>              {merge_window,always},
>>>              {max_fold_age,-1},
>>>              {max_fold_puts,0},
>>>              {expiry_secs,660},
>>>              {require_hint_crc,true}]}
>>>
>>>
>>> As a result, the amount of used memory(RAM) keeps growing until the
>>> server run out of free memory.
>>>
>>> Could you give me some clue that it can point to the cause of the
>>> problem?
>>>
>>> Thanks in advance.
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users at lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140903/554dab42/attachment.html>


More information about the riak-users mailing list