riak 1.2.0-1 / Failed to merge

Sebastian Gerlach sebastian.gerlach at immonet.de
Thu Oct 10 11:37:37 EDT 2013


He Jon,

Thanks a lot for the fast support. I couldn't find any logline like the
given example. "bitcask_fileop" didn's show up in any of my riak logs.

Best regards
Sebastian


On 10/10/2013 04:54 PM, Jon Meredith wrote:
> Thanks for the version information.  Do you have any log entries like
> 
>  "bitcask_fileops:fold: <filename>: expected 14 bytes but got only <some
> number> bytes, skipping\n",
> 
> That will tell you more accurately which file is causing the problem.
> 
> It'll take me a little while to check if I have anything to dump the
> bucket/keys.
> 
> 
> 
> On Thu, Oct 10, 2013 at 8:14 AM, Sebastian Gerlach
> <sebastian.gerlach at immonet.de <mailto:sebastian.gerlach at immonet.de>> wrote:
> 
>     Hi Jon,
> 
>     thanks for the quick replay and the offerred solution. we are running
>     the debian package version 1.2.0-1.
> 
>     Is there a way to get a list of keys witch are in this files. To trigger
>     a read. We have over 25.000.000 Million records in riak an it's would be
>     grat to receive the richt ones. Not all.
> 
>     Thanks in advanced.
> 
>     Best regards
>     Sebastian
> 
> 
>     On 10/10/2013 04:09 PM, Jon Meredith wrote:
>     > Hi Sebastian,
>     >
>     > Sorry the log file is a little opaque.  Bitcask is having problems
>     > running the merge
>     > on
>     /var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/4.bitcask.data
>     > and
>     /var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/29.bitcask.data.
>     >  The zero length files are being created by the failed merges.
>     >
>     > Something is corrupting the internal merge state structure.  What
>     > version of bitcask and riak are you running?
>     >
>     > If you want some immediate reliefe, I would move the
>     > 4.bitcask.data, 4.bitcask.data, 29.bitcask.hint and 29.bitcask.hint
>     > files out of the bitcask directory to somewhere safe and remove
>     all the
>     > zero-length .data and .hint files and restart the node.  Read repair
>     > will fix the objects over time, or you could do more active repair by
>     > retrieving all the objects.  More recent versions of Riak have a
>     manual
>     > repair facility built in, but I'm guessing you're running something
>     > before 1.2.
>     >
>     > Jon
>     >
>     >
>     >
>     >
>     >
>     > On Thu, Oct 10, 2013 at 7:05 AM, Sebastian Gerlach
>     > <sebastian.gerlach at immonet.de
>     <mailto:sebastian.gerlach at immonet.de>
>     <mailto:sebastian.gerlach at immonet.de
>     <mailto:sebastian.gerlach at immonet.de>>> wrote:
>     >
>     >     Hello,
>     >
>     >     we have a 12 node riak cluster. Since a while we have a log of
>     messages
>     >     like this:
>     >
>     >     "2013-10-10 00:22:04.725 [error] <0.5743.2160> Failed to merge
>     >    
>     ["/var/lib/riak/bitcask/276886052384956216925307478073202056458252845056",[{async_folds,true},{vnode_vclocks,true},{included_applications,[]},{allow_strfun,false},{reduce_js_vm_count,6},{storage_backend,riak_kv_bitcask_backend},{legacy_keylisting,false},{hook_js_vm_count,2},{listkeys_backpressure,true},{mapred_name,"mapred"},{stats_urlpath,"stats"},{legacy_stats,true},{js_thread_stack,16},{add_paths,[]},{http_url_encoding,on},{map_js_vm_count,8},{mapred_2i_pipe,true},{mapred_system,pipe},{js_max_vm_mem,8},{read_write,true}],["/var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/29.bitcask.data","/var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/4.bitcask.data"]]:
>     >    
>     {{badrecord,mstate},[{bitcask,merge_files,1,[{file,"src/bitcask.erl"},{line,835}]},{bitcask,merge1,3,[{file,"src/bitcask.erl"},{line,509}]},{bitcask_merge_worker,do_merge,1,[{file,"src/bitcask_merge_worker.erl"},{line,130}]}]}
>     >     "
>     >
>     >     The provides directory have a lot of files with a size of
>     zero. And the
>     >     number of files is much higher (2821 files) than in other
>     directorys (30
>     >     files).
>     >
>     >     Does anyone has a solution or a hint.
>     >
>     >     Thanks a lot an best regards
>     >     Sebastian
>     >
>     >     _______________________________________________
>     >     riak-users mailing list
>     >     riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>     <mailto:riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>>
>     >     http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>     >
>     >
>     >
>     >
>     > --
>     > Jon Meredith
>     > VP, Engineering
>     > Basho Technologies, Inc.
>     > jmeredith at basho.com <mailto:jmeredith at basho.com>
>     <mailto:jmeredith at basho.com <mailto:jmeredith at basho.com>>
> 
> 
> 
> 
> -- 
> Jon Meredith
> VP, Engineering
> Basho Technologies, Inc.
> jmeredith at basho.com <mailto:jmeredith at basho.com>





More information about the riak-users mailing list