riak 1.2.0-1 / Failed to merge

Sebastian Gerlach sebastian.gerlach at immonet.de
Thu Oct 10 10:14:52 EDT 2013


Hi Jon,

thanks for the quick replay and the offerred solution. we are running
the debian package version 1.2.0-1.

Is there a way to get a list of keys witch are in this files. To trigger
a read. We have over 25.000.000 Million records in riak an it's would be
grat to receive the richt ones. Not all.

Thanks in advanced.

Best regards
Sebastian


On 10/10/2013 04:09 PM, Jon Meredith wrote:
> Hi Sebastian,
> 
> Sorry the log file is a little opaque.  Bitcask is having problems
> running the merge
> on /var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/4.bitcask.data
> and /var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/29.bitcask.data.
>  The zero length files are being created by the failed merges.
> 
> Something is corrupting the internal merge state structure.  What
> version of bitcask and riak are you running?
> 
> If you want some immediate reliefe, I would move the
> 4.bitcask.data, 4.bitcask.data, 29.bitcask.hint and 29.bitcask.hint
> files out of the bitcask directory to somewhere safe and remove all the
> zero-length .data and .hint files and restart the node.  Read repair
> will fix the objects over time, or you could do more active repair by
> retrieving all the objects.  More recent versions of Riak have a manual
> repair facility built in, but I'm guessing you're running something
> before 1.2.
> 
> Jon
> 
> 
> 
> 
> 
> On Thu, Oct 10, 2013 at 7:05 AM, Sebastian Gerlach
> <sebastian.gerlach at immonet.de <mailto:sebastian.gerlach at immonet.de>> wrote:
> 
>     Hello,
> 
>     we have a 12 node riak cluster. Since a while we have a log of messages
>     like this:
> 
>     "2013-10-10 00:22:04.725 [error] <0.5743.2160> Failed to merge
>     ["/var/lib/riak/bitcask/276886052384956216925307478073202056458252845056",[{async_folds,true},{vnode_vclocks,true},{included_applications,[]},{allow_strfun,false},{reduce_js_vm_count,6},{storage_backend,riak_kv_bitcask_backend},{legacy_keylisting,false},{hook_js_vm_count,2},{listkeys_backpressure,true},{mapred_name,"mapred"},{stats_urlpath,"stats"},{legacy_stats,true},{js_thread_stack,16},{add_paths,[]},{http_url_encoding,on},{map_js_vm_count,8},{mapred_2i_pipe,true},{mapred_system,pipe},{js_max_vm_mem,8},{read_write,true}],["/var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/29.bitcask.data","/var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/4.bitcask.data"]]:
>     {{badrecord,mstate},[{bitcask,merge_files,1,[{file,"src/bitcask.erl"},{line,835}]},{bitcask,merge1,3,[{file,"src/bitcask.erl"},{line,509}]},{bitcask_merge_worker,do_merge,1,[{file,"src/bitcask_merge_worker.erl"},{line,130}]}]}
>     "
> 
>     The provides directory have a lot of files with a size of zero. And the
>     number of files is much higher (2821 files) than in other directorys (30
>     files).
> 
>     Does anyone has a solution or a hint.
> 
>     Thanks a lot an best regards
>     Sebastian
> 
>     _______________________________________________
>     riak-users mailing list
>     riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>     http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> 
> 
> 
> 
> -- 
> Jon Meredith
> VP, Engineering
> Basho Technologies, Inc.
> jmeredith at basho.com <mailto:jmeredith at basho.com>





More information about the riak-users mailing list