Riak Nodes Crashing

Matthew Von-Maszewski matthewv at basho.com
Mon Dec 8 12:28:37 EST 2014


Satish,

This is to be expected.  You have a ring size of 64 and 5 nodes.  5 does not evenly divide into 64.  4 nodes contain 13 vnodes.  One node only contains 12 vnodes:

13 / 64 = 20.3125%
12 / 64 = 18.75 %

All is fine.

Matthew


On Dec 8, 2014, at 12:17 PM, ender <extropy at gmail.com> wrote:

> I intend to upgrade to 2.0 at some point but that will be a bigger task.  Another question:
> 
> [ec2-user at ip-10-197-93-214 ~]$ sudo riak-admin member_status
> ================================= Membership ==================================
> Status     Ring    Pending    Node
> -------------------------------------------------------------------------------
> valid      20.3%      --      'riak at 10.196.72.106'
> valid      20.3%      --      'riak at 10.196.72.124'
> valid      20.3%      --      'riak at 10.196.72.247'
> valid      20.3%      --      'riak at 10.197.93.214'
> valid      18.8%      --      'riak at 10.197.94.33'
> -------------------------------------------------------------------------------
> Valid:5 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
> 
> When I first created the cluster, all 5 nodes Ring matched to within 0.1% of each other.  Now node 5 has consistently been at a lower percentage than the other 4.  Is this normal?
> 
> 
> On Mon, Dec 8, 2014 at 9:06 AM, Matthew Von-Maszewski <matthewv at basho.com> wrote:
> Satish,
> 
> This additional information continues to support my suspicion that the memory management is not fully accounting for your number of open files.  A large query can cause many files that were previously unused to open.  An open table file in leveldb uses memory heavily (for the file's block index and bloom filter).  Also, leveldb will allow the memory limit to be knowingly exceeded in the case of queries that cover large segments of the key space.
> 
> There are fixes for both of those scenarios in Riak 2.0, but not in the 1.x series.
> 
> Matthew 
> 
> 
> On Dec 8, 2014, at 11:22 AM, ender <extropy at gmail.com> wrote:
> 
>> Hello Matthew,
>> 
>> I was going through my cluster setup again, checking up on stuff, when I noticed something.  So just for some background, when I originally started using Riak it was as a replacement for MongoDB.  To get things up and running quickly I "cheated" and just wrote some code that took a MongoDB query and recast it as a Riak search query. Then I enabled the search hook on all my buckets, and it just worked!  Of course, Riak search wasn't the fastest thing on 2 legs, so I started to redo my data model to make it more key-value store friendly. Where I had to, I used secondary indexes.  Once I'd converted the data model for a bucket I would remove the search hook on that bucket.  On Friday evening I discovered that on 2 of my buckets I'd forgotten to remove the search hook.  One of the buckets only has a couple of thousand records in it, so no big deal.  But the other one! - that bucket has the most reads, the most writes, and has over 200M records stored in it.  I removed the search hook on both of those buckets and the cluster has been stable over the weekend and is still up as of now. I did not disable active anti-entropy, since I did not want to change too many variables at the same time.  I will do that today.  Question is, was this a coincidence or do you think it's possible the search indexing was causing the OOM errors?
>> 
>> Satish
>> 
>> On Sat, Dec 6, 2014 at 6:59 AM, Matthew Von-Maszewski <matthewv at basho.com> wrote:
>> Satish,
>> 
>> I do NOT recommend adding a sixth node before the other five are stable again.  There was another customer that did that recently and things just got worse due to the vnode handoff actions to the sixth node.
>> 
>> I do recommend one or both of the following:
>> 
>> - disable active anti-entropy in app.config, {anti_entropy, {off, []}}.  Then restart all nodes.  We quickly replaced 1.4.7 due to an bug in the active anti-entropy.  I do not know the details of the bug.  But no one had seen a crash from it.  However, you may be seeing a long term problem due to that same bug.  The anti-entropy feature in 1.4.7 is not really protecting your data anyway.  It might as well be disabled until you are ready to upgrade.
>> 
>> - further reduce the max_open_files parameter simply to get memory stable:  use 75 instead of the recent 150.  You must restart all nodes after making the change in app.config.
>> 
>> 
>> I will need to solicit support from others at Basho if the two workarounds above do not stabilize the cluster.  
>> 
>> Matthew
>> 
>> On Dec 5, 2014, at 5:54 PM, ender <extropy at gmail.com> wrote:
>> 
>>> Would adding a 6th node mean each node would use less memory as  a stopgap measure?
>>> 
>>> On Fri, Dec 5, 2014 at 2:20 PM, ender <extropy at gmail.com> wrote:
>>> Hey Matthew it just crashed again.  This time I got the syslog and leveldb logs right away.
>>> 
>>> 
>>> 
>>> On Fri, Dec 5, 2014 at 11:43 AM, Matthew Von-Maszewski <matthewv at basho.com> wrote:
>>> Satish,
>>> 
>>> Here is a key line from /var/log/messages:
>>> 
>>> Dec  5 06:52:43 ip-10-196-72-106 kernel: [26881589.804401] beam.smp invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
>>> 
>>> The log entry does NOT match the timestamps of the crash.log and error.log below.  But that is ok.  The operating system killed off Riak.  There would have be no notification in the Riak log's of the operating system's actions.
>>> 
>>> The fact that the out of memory monitor, oom-killer, killed Riak further supports the change to max_open_files.  I recommend we now wait to see if the problem occurs again.
>>> 
>>> 
>>> Matthew
>>> 
>>> 
>>> On Dec 5, 2014, at 2:35 PM, ender <extropy at gmail.com> wrote:
>>> 
>>>> Hey Matthew,
>>>> 
>>>> The crash occurred around 3:00am:
>>>> 
>>>> -rw-rw-r-- 1 riak riak    920 Dec  5 03:01 crash.log
>>>> -rw-rw-r-- 1 riak riak    617 Dec  5 03:01 error.log
>>>> 
>>>> I have attached the syslog that covers that time.  I also went ahead and changed max_open_files in app.config to to 150 from 315.
>>>> 
>>>> Satish
>>>> 
>>>> 
>>>> On Fri, Dec 5, 2014 at 11:29 AM, Matthew Von-Maszewski <matthewv at basho.com> wrote:
>>>> Satish,
>>>> 
>>>> The "key" system log varies by Linux platform.  Yes, /var/log/messages may hold some key clues.  Again, be sure the file covers the time of a crash.
>>>> 
>>>> Matthew
>>>> 
>>>> 
>>>> On Dec 5, 2014, at 1:29 PM, ender <extropy at gmail.com> wrote:
>>>> 
>>>>> Hey Matthew,
>>>>> 
>>>>> I see a /var/log/messages file, but no syslog or system.log etc.  Is it the messages file you want?
>>>>> 
>>>>> Satish
>>>>> 
>>>>> 
>>>>> On Fri, Dec 5, 2014 at 10:06 AM, Matthew Von-Maszewski <matthewv at basho.com> wrote:
>>>>> Satish,
>>>>> 
>>>>> I find nothing compelling in the log or the app.config.  Therefore I have two additional suggestions/requests:
>>>>> 
>>>>> - lower max_open_files in app.config to to 150 from 315.  There was one other customer report regarding the limit not properly stopping out of memory (OOM) conditions.
>>>>> 
>>>>> - try to locate a /var/log/syslog* file from a node that contains the time of the crash.  There may be helpful information there.  Please send that along.
>>>>> 
>>>>> 
>>>>> Unrelated to this crash … 1.4.7 has a known bug in its active anti-entropy (AAE) logic.  This bug is NOT known to cause a crash.  The bug does cause AAE to be unreliable for data restoration.  The proper steps for upgrading to the current release (1.4.12) are:
>>>>> 
>>>>> -- across the entire cluster
>>>>> - disable anti_entropy in app.config on all nodes: {anti_entropy, {off, []}}
>>>>> - perform a rolling restart of all nodes … AAE is now disabled in the cluster 
>>>>> 
>>>>> -- on each node
>>>>> - stop the node
>>>>> - remove (erase all files and directories) /vol/lib/riak/anti_entropy
>>>>> - update Riak to the new software revision
>>>>> - start the node again
>>>>> 
>>>>> -- across the entire cluster
>>>>> - enable anti_entropy in app.config on all nodes: {anti_entropy, {on, []}}
>>>>> - perform a rolling restart of all nodes … AAE is now enabled in the cluster 
>>>>> 
>>>>> The nodes will start rebuilding the AAE hash data.  Suggest you perform the last rolling restart during a low utilization time of your cluster.
>>>>> 
>>>>> 
>>>>> Matthew
>>>>> 
>>>>> 
>>>>> On Dec 5, 2014, at 11:02 AM, ender <extropy at gmail.com> wrote:
>>>>> 
>>>>>> Hi Matthew,
>>>>>> 
>>>>>> Riak version: 1.4.7
>>>>>> 5 Nodes in cluster
>>>>>> RAM: 30GB
>>>>>> 
>>>>>> The leveldb logs are attached.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Thu, Dec 4, 2014 at 1:34 PM, Matthew Von-Maszewski <matthewv at basho.com> wrote:
>>>>>> Satish,
>>>>>> 
>>>>>> Some questions:
>>>>>> 
>>>>>> - what version of Riak are you running?  logs suggest 1.4.7
>>>>>> - how many nodes in your cluster?
>>>>>> - what is the physical memory (RAM size) of each node?
>>>>>> - would you send the leveldb LOG  files from one of the crashed servers:
>>>>>>     tar -czf satish_LOG.tgz /vol/lib/riak/leveldb/*/LOG*
>>>>>> 
>>>>>> 
>>>>>> Matthew
>>>>>> 
>>>>>> On Dec 4, 2014, at 4:02 PM, ender <extropy at gmail.com> wrote:
>>>>>> 
>>>>>> > My RIak installation has been running successfully for about a year.  This week nodes suddenly started randomly crashing.  The machines have plenty of memory and free disk space, and looking in the ring directory nothing appears to amiss:
>>>>>> >
>>>>>> > [ec2-user at ip-10-196-72-247 ~]$ ls -l /vol/lib/riak/ring
>>>>>> > total 80
>>>>>> > -rw-rw-r-- 1 riak riak 17829 Nov 29 19:42 riak_core_ring.default.20141129194225
>>>>>> > -rw-rw-r-- 1 riak riak 17829 Dec  3 19:07 riak_core_ring.default.20141203190748
>>>>>> > -rw-rw-r-- 1 riak riak 17829 Dec  4 16:29 riak_core_ring.default.20141204162956
>>>>>> > -rw-rw-r-- 1 riak riak 17847 Dec  4 20:45 riak_core_ring.default.20141204204548
>>>>>> >
>>>>>> > [ec2-user at ip-10-196-72-247 ~]$ du -h /vol/lib/riak/ring
>>>>>> > 84K   /vol/lib/riak/ring
>>>>>> >
>>>>>> > I have attached a tarball with the app.config file plus all the logs from the node at the time of the crash.  Any help much appreciated!
>>>>>> >
>>>>>> > Satish
>>>>>> >
>>>>>> > <riak-crash-data.tar.gz>_______________________________________________
>>>>>> > riak-users mailing list
>>>>>> > riak-users at lists.basho.com
>>>>>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>> 
>>>>>> 
>>>>>> <satish_LOG.tgz>
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> <messages>
>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20141208/643db6a2/attachment.html>


More information about the riak-users mailing list