riak 1.2.0-1 / Failed to merge

Jon Meredith jmeredith at basho.com
Thu Oct 10 10:54:48 EDT 2013


Thanks for the version information.  Do you have any log entries like

 "bitcask_fileops:fold: <filename>: expected 14 bytes but got only <some
number> bytes, skipping\n",

That will tell you more accurately which file is causing the problem.

It'll take me a little while to check if I have anything to dump the
bucket/keys.



On Thu, Oct 10, 2013 at 8:14 AM, Sebastian Gerlach <
sebastian.gerlach at immonet.de> wrote:

> Hi Jon,
>
> thanks for the quick replay and the offerred solution. we are running
> the debian package version 1.2.0-1.
>
> Is there a way to get a list of keys witch are in this files. To trigger
> a read. We have over 25.000.000 Million records in riak an it's would be
> grat to receive the richt ones. Not all.
>
> Thanks in advanced.
>
> Best regards
> Sebastian
>
>
> On 10/10/2013 04:09 PM, Jon Meredith wrote:
> > Hi Sebastian,
> >
> > Sorry the log file is a little opaque.  Bitcask is having problems
> > running the merge
> > on
> /var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/4.bitcask.data
> > and
> /var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/29.bitcask.data.
> >  The zero length files are being created by the failed merges.
> >
> > Something is corrupting the internal merge state structure.  What
> > version of bitcask and riak are you running?
> >
> > If you want some immediate reliefe, I would move the
> > 4.bitcask.data, 4.bitcask.data, 29.bitcask.hint and 29.bitcask.hint
> > files out of the bitcask directory to somewhere safe and remove all the
> > zero-length .data and .hint files and restart the node.  Read repair
> > will fix the objects over time, or you could do more active repair by
> > retrieving all the objects.  More recent versions of Riak have a manual
> > repair facility built in, but I'm guessing you're running something
> > before 1.2.
> >
> > Jon
> >
> >
> >
> >
> >
> > On Thu, Oct 10, 2013 at 7:05 AM, Sebastian Gerlach
> > <sebastian.gerlach at immonet.de <mailto:sebastian.gerlach at immonet.de>>
> wrote:
> >
> >     Hello,
> >
> >     we have a 12 node riak cluster. Since a while we have a log of
> messages
> >     like this:
> >
> >     "2013-10-10 00:22:04.725 [error] <0.5743.2160> Failed to merge
> >
> ["/var/lib/riak/bitcask/276886052384956216925307478073202056458252845056",[{async_folds,true},{vnode_vclocks,true},{included_applications,[]},{allow_strfun,false},{reduce_js_vm_count,6},{storage_backend,riak_kv_bitcask_backend},{legacy_keylisting,false},{hook_js_vm_count,2},{listkeys_backpressure,true},{mapred_name,"mapred"},{stats_urlpath,"stats"},{legacy_stats,true},{js_thread_stack,16},{add_paths,[]},{http_url_encoding,on},{map_js_vm_count,8},{mapred_2i_pipe,true},{mapred_system,pipe},{js_max_vm_mem,8},{read_write,true}],["/var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/29.bitcask.data","/var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/4.bitcask.data"]]:
> >
> {{badrecord,mstate},[{bitcask,merge_files,1,[{file,"src/bitcask.erl"},{line,835}]},{bitcask,merge1,3,[{file,"src/bitcask.erl"},{line,509}]},{bitcask_merge_worker,do_merge,1,[{file,"src/bitcask_merge_worker.erl"},{line,130}]}]}
> >     "
> >
> >     The provides directory have a lot of files with a size of zero. And
> the
> >     number of files is much higher (2821 files) than in other directorys
> (30
> >     files).
> >
> >     Does anyone has a solution or a hint.
> >
> >     Thanks a lot an best regards
> >     Sebastian
> >
> >     _______________________________________________
> >     riak-users mailing list
> >     riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
> >     http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
> >
> >
> >
> > --
> > Jon Meredith
> > VP, Engineering
> > Basho Technologies, Inc.
> > jmeredith at basho.com <mailto:jmeredith at basho.com>
>
>


-- 
Jon Meredith
VP, Engineering
Basho Technologies, Inc.
jmeredith at basho.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20131010/9fbbe394/attachment.html>


More information about the riak-users mailing list