Riak 0.14 nodes crashing under light load

Anthony Molinaro anthonym at alumni.caltech.edu
Tue Feb 1 19:34:25 EST 2011


Hi Mark,

Yeah, you know I thought about this as soon as I got on my bicycle to ride
home.  And remembered that while I did bump the limit on my QA setup I did
not on the production setup.  Oops, I'm sure that's it.  I'll write back
if that doesn't solve it.

Thanks for the quick response.

-Anthony

On Tue, Feb 01, 2011 at 04:22:34PM -0800, Mark Phillips wrote:
> Hey Anthony,
> 
> At first glance, it looks like you may have run into an open file
> handle problem.  Try increasing the ulimit to at least 1024. There is
> more info on the wiki here:
> 
> http://wiki.basho.com/Open-Files-Limit.html
> 
> Give that a shot and let us know if the problem persists.
> 
> Mark
> 
> On Tue, Feb 1, 2011 at 4:08 PM, Anthony Molinaro
> <anthonym at alumni.caltech.edu> wrote:
> > Hi,
> >
> >  I just set up a 4 node cluster mostly vanilla config with the exception
> > that I specified 1024 partitions, and I'm using a multi-backend with one
> > entry for bitcask and the default as bitcask (I plan to deploy a cache
> > backend at some point).  I have one bucket which stores a pretty small
> > payload (key is 36 bytes, value is 36 bytes).
> >
> > Things ran fine under light load (~400 get, ~30 puts according to riak-admin
> > status, so I think that's per minute).  Suddenly several nodes (3/4) all
> > shutdown within a few minutes of each other.  They all seem to have
> > errors like this
> >
> > =ERROR REPORT==== 1-Feb-2011::23:50:35 ===^M
> > Failed to open lock file /var/lib/riak/bitcask/1156070631091827503657211635254091060470024765440/bitcask.write.lock: emfile
> >
> > then a state machine termination stacktrace.
> >
> > I had run cluster_info for all of them a few minutes before and the only
> > machine which didn't crash was the one I ran cluster_info on.
> >
> > Not sure if that was the cause or not.  Any ideas what could cause these
> > errors?
> >
> > I can send more info if it would help, but wanted to get the conversation
> > started before I head home.
> >
> > Thanks,
> >
> > -Anthony
> >
> > --
> > ------------------------------------------------------------------------
> > Anthony Molinaro                           <anthonym at alumni.caltech.edu>
> >
> > _______________________________________________
> > riak-users mailing list
> > riak-users at lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >

-- 
------------------------------------------------------------------------
Anthony Molinaro                           <anthonym at alumni.caltech.edu>




More information about the riak-users mailing list