Problems with bitcask, file merge errors, too many 0 byte files

Jacob Chapel jacob.chapel at
Tue May 29 20:44:37 EDT 2012

Our Riak server which is running 1.0.2 at the moment using bitcask backend
and search is crashing often and when restarted will crash again
immediately due to system_limit error.

2012-05-29 19:28:54.808 [error] <0.1001.0>@riak_kv_vnode:init:245 Failed to
start riak_kv_bitcask_backend Reason:

Before, we were getting emfile errors, so we upped the ulimit for open
files which helped. Soon after (about a week) it crashed again but due to
the above error. After looking into it and asking on IRC, there wasn't much
information but looked to be due to a ton of 0 (zero) byte files in the
bitcask folder. In fact when counting, at first there were over 20k 0 byte
files. Someone who had similar issues was instructed to delete them, and
common sense says that 0 byte files don't hold any data. So I backed them
up (just in case) and removed them from the bitcask folder. That allowed
the server to startup and run again.

Fast forward a few days to now, it appears to have crashed due to the same
issue. Having over 20k new 0 byte files. I asked on IRC again but not much
could be helped since they didn't know. I backed up and removed the files
again and it runs. How can I prevent these files?

Also, as this sample log output shows:

We are getting a lot of file merge errors and child processes dying
randomly (not sure how to read the error).

I am not really sure where to go from here, we can't keep removing 0 byte
files to keep the server up, and I am sure there is some setting or
configuration problem that just isn't apparent. Help would be very much

Jacob Chapel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>

More information about the riak-users mailing list