Problems with bitcask, file merge errors, too many 0 byte files
jacob.chapel at gmail.com
Tue May 29 20:44:37 EDT 2012
Our Riak server which is running 1.0.2 at the moment using bitcask backend
and search is crashing often and when restarted will crash again
immediately due to system_limit error.
2012-05-29 19:28:54.808 [error] <0.1001.0>@riak_kv_vnode:init:245 Failed to
start riak_kv_bitcask_backend Reason:
Before, we were getting emfile errors, so we upped the ulimit for open
files which helped. Soon after (about a week) it crashed again but due to
the above error. After looking into it and asking on IRC, there wasn't much
information but looked to be due to a ton of 0 (zero) byte files in the
bitcask folder. In fact when counting, at first there were over 20k 0 byte
files. Someone who had similar issues was instructed to delete them, and
common sense says that 0 byte files don't hold any data. So I backed them
up (just in case) and removed them from the bitcask folder. That allowed
the server to startup and run again.
Fast forward a few days to now, it appears to have crashed due to the same
issue. Having over 20k new 0 byte files. I asked on IRC again but not much
could be helped since they didn't know. I backed up and removed the files
again and it runs. How can I prevent these files?
Also, as this sample log output shows:
We are getting a lot of file merge errors and child processes dying
randomly (not sure how to read the error).
I am not really sure where to go from here, we can't keep removing 0 byte
files to keep the server up, and I am sure there is some setting or
configuration problem that just isn't apparent. Help would be very much
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the riak-users