Simple 3 node test cluster eating all my memory

Juan Luis Francés jl.frances.j at conekia.es
Thu Feb 12 05:07:40 EST 2015


Hi Simon.

Maybe sadly you have been hit by this bug: https://github.com/basho/riak_kv/issues/1064

Sadly, because I think that there aren’t a soon resolution about this issue.

Juan Luis Francés

> El 12/2/2015, a las 10:51, Simon Hartley <Simon.Hartley at williamhill.com> escribió:
> 
> Hi,
>  
> We have a simple 3 node test cluster, and we are seeing this cluster fall-over under quite modest loads, 2-3 times a day with the node reporting out of memory problems.
>  
> Our basic details are:
>  
> ·         Riak 1.4.9
> ·         3 nodes, each being:
> o   A virtual RedHat EL
> o   2 x 2.2GHz CPU
> o   4GB RAM
> ·         In-memory back end
> ·         128 segment ring
>  
> We have tried to limit the in-memory max memory by the following setting in app.config:
>  
> %% Memory Config
> {memory_backend, [
>              {max_memory, 32}, %% 32 megabytes
>              {ttl, 86400}  %% 1 Day in seconds
>            ]},
>  
> The behaviour we are seeing is a sudden and rapid increase in memory allocated to riak, up to 100% available RAM, and then a crash.
>  
> This is happening on all 3 nodes (at different times).
>  
> When looking through the console.log just prior to the out of memory / crash we see log entries similar to the following appearing:
>  
> 2015-02-11 16:50:27.356 [info] <0.98.0>@riak_core_sysmon_handler:handle_event:92 monitor large_heap <0.238.0> [{name,riak_core_gossip},{initial_call,{riak_core_gossip,init,1}},{almost_current_function,{riak_core_gossip,update_gossip_vers
> ion,1}},{message_queue_len,78}] [{old_heap_block_size,0},{heap_block_size,47828850},{mbuf_size,0},{stack_size,15},{old_heap_size,0},{heap_size,13944475}]
>  
> We can’t see any errors at the appropriate times in the error.log
>  
> We see the following in erlang.log.1:
>  
> ===== ALIVE Wed Feb 11 17:00:24 GMT 2015
> /usr/lib64/riak/lib/os_mon-2.2.9/priv/bin/memsup: Erlang has closed.
> Erlang has closed
>  
> Crash dump was written to: /var/log/riak/erl_crash.dump
> eheap_alloc: Cannot allocate 4454408120 bytes of memory
>  
> =====
>  
> Anyone any ideas?
>  
> Simon Hartley
> Solutions Architect
>  
> Email: Simon.Hartley at williamhill.com
> Skype: +44 (0)113 397 6747
> Skype: sijomons
>  
> William Hill Online, St. Johns, Merrion St. Leeds, LS2 8LQ
> <image001.png>
>  
> Confidentiality: The contents of this e-mail and any attachments transmitted with it are intended to be confidential to the intended recipient; and may be privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. This e-mail is sent by a William Hill PLC group company. The William Hill group companies include, among others, William Hill PLC (registered number 4212563), William Hill Organization Limited (registered number 278208), William Hill US HoldCo Inc, WHG (International) Limited (registered number 99191) and WHG Trading Limited (registered number 101439). Each of William Hill PLC, William Hill Organization Limited is registered in England and Wales and has its registered office at Greenside House, 50 Station Road, Wood Green, London N22 7TP. William Hill U.S. HoldCo, Inc. is 160 Greentree Drive, Suite 101, Dover 19904, Kent, Delaware, United States of America. Each of WHG (International) Limited and WHG Trading Limited is registered in Gibraltar and has its registered office at 6/1 Waterport Place, Gibraltar. Unless specifically indicated otherwise, the contents of this e-mail are subject to contract; and are not an official statement, and do not necessarily represent the views, of William Hill PLC, its subsidiaries or affiliated companies. Please note that neither William Hill PLC, nor its subsidiaries and affiliated companies can accept any responsibility for any viruses contained within this e-mail and it is your responsibility to scan any emails and their attachments. William Hill PLC, its subsidiaries and affiliated companies may monitor e-mail traffic data and also the content of e-mails for effective operation of the e-mail system, or for security, purposes.. _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





More information about the riak-users mailing list