Riak 0.14 nodes crashing under light load

Mark Phillips mark at basho.com
Tue Feb 1 19:22:34 EST 2011


Hey Anthony,

At first glance, it looks like you may have run into an open file
handle problem.  Try increasing the ulimit to at least 1024. There is
more info on the wiki here:

http://wiki.basho.com/Open-Files-Limit.html

Give that a shot and let us know if the problem persists.

Mark

On Tue, Feb 1, 2011 at 4:08 PM, Anthony Molinaro
<anthonym at alumni.caltech.edu> wrote:
> Hi,
>
>  I just set up a 4 node cluster mostly vanilla config with the exception
> that I specified 1024 partitions, and I'm using a multi-backend with one
> entry for bitcask and the default as bitcask (I plan to deploy a cache
> backend at some point).  I have one bucket which stores a pretty small
> payload (key is 36 bytes, value is 36 bytes).
>
> Things ran fine under light load (~400 get, ~30 puts according to riak-admin
> status, so I think that's per minute).  Suddenly several nodes (3/4) all
> shutdown within a few minutes of each other.  They all seem to have
> errors like this
>
> =ERROR REPORT==== 1-Feb-2011::23:50:35 ===^M
> Failed to open lock file /var/lib/riak/bitcask/1156070631091827503657211635254091060470024765440/bitcask.write.lock: emfile
>
> then a state machine termination stacktrace.
>
> I had run cluster_info for all of them a few minutes before and the only
> machine which didn't crash was the one I ran cluster_info on.
>
> Not sure if that was the cause or not.  Any ideas what could cause these
> errors?
>
> I can send more info if it would help, but wanted to get the conversation
> started before I head home.
>
> Thanks,
>
> -Anthony
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro                           <anthonym at alumni.caltech.edu>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>




More information about the riak-users mailing list