riak 2.1.1 : Erlang crash dump

Girish Shankarraman gshankarraman at vmware.com
Sat Oct 3 20:34:11 EDT 2015

Thank you for the response, Jon.

So I changed it to 50% and it crashed again.
I have a 5 nodes cluster with 60GB RAM on each node. Ring size is set to 64. (Attached riak conf if any one has some ideas).

I still see the erlang process consuming the entire capacity of the system (52 GB).

24256 riak      20   0 67.134g 0.052t  18740 D   0.0 90.0   2772:44 beam.smp

---- Cluster Status ----
Ring ready: true

|        node        |status| avail |ring |pending|
| (C) riak at<mailto:riak at> |valid |  up   | 20.3|  --   |
|     riak at<mailto:riak at> |valid |  up   | 20.3|  --   |
|     riak at<mailto:riak at> |valid |  up   | 20.3|  --   |
|     riak at<mailto:riak at> |valid |  up   | 20.3|  --   |
|     riak at<mailto:riak at> |valid |  up   | 18.8|  --   |


- Girish Shankarraman

From: Jon Meredith <jmeredith at basho.com<mailto:jmeredith at basho.com>>
Date: Thursday, October 1, 2015 at 2:06 PM
To: girish shankarraman <gshankarraman at vmware.com<mailto:gshankarraman at vmware.com>>, "riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>" <riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>>
Subject: Re: riak 2.1.1 : Erlang crash dump

It looks like Riak was unable to allocate 4Gb of memory.  You may have to reduce the amount of memory allocated for leveldb from the default 70%, try setting this in your /etc/riak/riak.conf file

leveldb.maximum_memory.percent = 50

The memory footprint for Riak should stabilize after a few hours and on servers with smaller amounts of memory, the 30% left over may not be enough.

Please let us know how you get on.

On Wed, Sep 30, 2015 at 5:31 PM Girish Shankarraman <gshankarraman at vmware.com<mailto:gshankarraman at vmware.com>> wrote:
I have 7 node cluster for riak with a ring_size of 128.

System Details:
Each node is a VM with 16GB of memory.
The backend is using leveldb.
sys_system_architecture : <<"x86_64-unknown-linux-gnu">>
sys_system_version : <<"Erlang R16B02_basho8 (erts-5.10.3) [source] [64-bit] [smp:4:4] [async-threads:64] [kernel-poll:true] [frame-pointer]">>
riak_control_version : <<"2.1.1-0-g5898c40">>
cluster_info_version : <<"2.0.2-0-ge231144">>
yokozuna_version : <<"2.1.0-0-gcb41c27">>

We have up to 400-1000 json records being written/sec. Each record might be a few 100 bytes.
I see the following crash message in the erlang logs after a few hours of processing. Any suggestions on what could be going on here ?

===== Tue Sep 29 20:20:56 UTC 2015
[os_mon] memory supervisor port (memsup): Erlang has closed^M
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed^M
Crash dump was written to: /var/log/riak/erl_crash.dump^M
eheap_alloc: Cannot allocate 3936326656 bytes of memory (of type "heap").^M

Also tested running this at 50GB per Riak Node(VM) and things work but memory keeps growing, so throwing hardware at it doesn't seem very scalable.


- Girish Shankarraman

riak-users mailing list
riak-users at lists.basho.com<mailto:riak-users at lists.basho.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20151004/228e56a9/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: riak.conf
Type: application/octet-stream
Size: 13421 bytes
Desc: riak.conf
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20151004/228e56a9/attachment.conf>

More information about the riak-users mailing list