Riak doesn't start anymore after server crash - no errors logged

Charlie Voiselle cvoiselle at basho.com
Fri Nov 20 12:24:16 EST 2015


Another thing that I have seen cause riak_kv to stall like this is LevelDB corruption preventing vnodes from starting properly.  I use this command to look for LOG files with the word error in them.  It is also worth checking the anti_entropy levelDB LOG files as well.

find . -name "LOG" -exec grep -l -i 'error' {} \;

-cv


> On Nov 11, 2015, at 7:05 PM, Steven Sylvester <ssylvester at basho.com> wrote:
> 
> Hi Markus,
> 
> One thing that stands out in what you pasted is the following line:
> 
> 2015-11-09 11:31:45.263 [info] <0.73.0> alarm_handler: {set,{{disk_almost_full,"/"},[]}}
> 
> Check the output of `df` and add nodes/disk as necessary to keep your disk usage at a manageable level. Our docs recommend keeping disk use below 60%:
> http://docs.basho.com/riak/latest/ops/building/planning/cluster/#Disk <http://docs.basho.com/riak/latest/ops/building/planning/cluster/#Disk>
> 
> Disk capacity issues can sneak up on you due to some backend operations that require a lot of space all at once, such as Bitcask merging or AAE. *Bad things* tend to happen to a Riak cluster with full disks, including data loss/corruption.
> 
> Other than that, are there any changes that have been made to the environment lately? Have you performed any upgrades recently?
> 
> Check the output of `ps aux` and kill any processes related to Riak, such as any instances of `beam.smp` before attempting to start the node again.
> 
> If the node won't start after these troubleshooting steps, run `riak console`  which will attempt to start the node and put you into an Erlang attach shell. Monitor the messages that are printed to stdout and if/when it seems to hang, paste the output of the whole thing here and we can take a look.
> 
> -- 
> Steven Sylvester, CISSP
> Client Services Engineer
> Basho Technologies
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20151120/a69be521/attachment-0002.html>


More information about the riak-users mailing list