riak 1.2.0-1 / Failed to merge

Jon Meredith jmeredith at basho.com
Thu Oct 10 10:09:01 EDT 2013


Hi Sebastian,

Sorry the log file is a little opaque.  Bitcask is having problems running
the merge on /var/lib/riak/bitcask/276886052384956216925307478073
202056458252845056/4.bitcask.data and /var/lib/riak/bitcask/
276886052384956216925307478073202056458252845056/29.bitcask.data.  The zero
length files are being created by the failed merges.

Something is corrupting the internal merge state structure.  What version
of bitcask and riak are you running?

If you want some immediate reliefe, I would move the 4.bitcask.data,
4.bitcask.data,
29.bitcask.hint and 29.bitcask.hint files out of the bitcask directory to
somewhere safe and remove all the zero-length .data and .hint files and
restart the node.  Read repair will fix the objects over time, or you could
do more active repair by retrieving all the objects.  More recent versions
of Riak have a manual repair facility built in, but I'm guessing you're
running something before 1.2.

Jon





On Thu, Oct 10, 2013 at 7:05 AM, Sebastian Gerlach <
sebastian.gerlach at immonet.de> wrote:

> Hello,
>
> we have a 12 node riak cluster. Since a while we have a log of messages
> like this:
>
> "2013-10-10 00:22:04.725 [error] <0.5743.2160> Failed to merge
>
> ["/var/lib/riak/bitcask/276886052384956216925307478073202056458252845056",[{async_folds,true},{vnode_vclocks,true},{included_applications,[]},{allow_strfun,false},{reduce_js_vm_count,6},{storage_backend,riak_kv_bitcask_backend},{legacy_keylisting,false},{hook_js_vm_count,2},{listkeys_backpressure,true},{mapred_name,"mapred"},{stats_urlpath,"stats"},{legacy_stats,true},{js_thread_stack,16},{add_paths,[]},{http_url_encoding,on},{map_js_vm_count,8},{mapred_2i_pipe,true},{mapred_system,pipe},{js_max_vm_mem,8},{read_write,true}],["/var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/29.bitcask.data","/var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/4.bitcask.data"]]:
>
> {{badrecord,mstate},[{bitcask,merge_files,1,[{file,"src/bitcask.erl"},{line,835}]},{bitcask,merge1,3,[{file,"src/bitcask.erl"},{line,509}]},{bitcask_merge_worker,do_merge,1,[{file,"src/bitcask_merge_worker.erl"},{line,130}]}]}
> "
>
> The provides directory have a lot of files with a size of zero. And the
> number of files is much higher (2821 files) than in other directorys (30
> files).
>
> Does anyone has a solution or a hint.
>
> Thanks a lot an best regards
> Sebastian
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>



-- 
Jon Meredith
VP, Engineering
Basho Technologies, Inc.
jmeredith at basho.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20131010/7b28fc23/attachment.html>


More information about the riak-users mailing list