Merge error in bitcask data store

Lucas Grijander lucasgrinjander69 at gmail.com
Wed Sep 3 11:25:05 EDT 2014


Hi.

I am new in the list, so i don't know if this is  the right place to open
this thread.

Our riak cluster is composed by 4 nodes. The O.S is Ubuntu 14.04 and the
version of riak is 2.0.0

We are getting a lot of errors in the logs of ours riak nodes. The errors
are like this:

2014-09-03 13:05:14.212 [error] <0.19152.3672> Failed to merge
{["/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/14.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/13.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/12.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/11.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/10.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/9.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/8.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/7.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/6.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/5.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/4.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/3.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/2.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/1.bitcask.data"],["/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/14.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/13.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/12.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/11.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/10.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553098649600/9.bitcask.data","/var/lib/riak/data/ten_minutes/570899077082383952423314387779798054553...",...]}:
{generic_failure,error,function_clause,[{riak_kv_bitcask_backend,key_transform_to_1,[{tombstone,<<2,0,4,109,116,116,108,50,48,49,52,48,57,48,50,50,50,53,48,50,53,48,56,102,56,102,102,101,54,49,49,97,101,51,99,52,52,55,101,55,55,100,99,50,100,49,52,56,51,57,55,48,50>>}],[{file,"src/riak_kv_bitcask_backend.erl"},{line,99}]},{bitcask,'-expiry_merge/4-fun-0-',7,[{file,"src/bitcask.erl"},{line,1912}]},{bitcask_fileops,fold_hintfile_loop,5,[{file,"src/bitcask_fileops.erl"},{line,660}]},{bitcask_fileops,fold_file_loop,8,[{file,"src/bitcask_fileops.erl"},{line,720}]},{bitcask_fileops,fold_hintfile,3,[{file,"src/bitcask_fileops.erl"},{line,624}]},{bitcask,expiry_merge,4,[{file,"src/bitcask.erl"},{line,1915}]},{bitcask,merge1,4,[{file,"src/bitcask.erl"},{line,686}]},{bitcask,merge,3,[{file,"src/bitcask.erl"},{line,566}]}]}

It seems riak can not merge data in bitcask data store.

This is the configuration of bitcask in ten minutes TTL:

{<<"ten_minutes_ttl">>,riak_kv_bitcask_backend,
            [{io_mode,erlang},
             {expiry_grace_time,0},
             {small_file_threshold,5242880},
             {dead_bytes_threshold,4194304},
             {frag_threshold,15},
             {dead_bytes_merge_trigger,4194304},
             {frag_merge_trigger,10},
             {max_file_size,10485760},
             {open_timeout,4},
             {data_root,"/var/lib/riak/data/ten_minutes"},
             {sync_strategy,none},
             {merge_window,always},
             {max_fold_age,-1},
             {max_fold_puts,0},
             {expiry_secs,660},
             {require_hint_crc,true}]}


As a result, the amount of used memory(RAM) keeps growing until the server
run out of free memory.

Could you give me some clue that it can point to the cause of the problem?

Thanks in advance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20140903/4194d9ab/attachment.html>


More information about the riak-users mailing list