Riak database fails after a short period

Mark Phillips mark at basho.com
Mon Jul 8 16:27:56 EDT 2013


Hi Kirill,

Based on a *very* quick read of this email, I think the issue here is
that your VM is too small. Riak needs a certain amount of memory to
run and 1GB of RAM probably isn't going to cut it. If it's possible,
grab three new VMs with at least 8GB of RAM and spread your cluster
over those machines.

Hope that helps.

Mark

On Thu, Jun 27, 2013 at 12:35 PM, Kirill K. <kirill.mailbx at gmail.com> wrote:
> Hi,
>
> I created a simple erlang application which periodically collects required
> data and puts it in a riak database.
>
> As I start my application it runs smoothly.. but after a period of time it
> stucks as PUT requests to riak database becomes too slow.. It is logs from
> my app:
>
> 2013-06-26 12:44:09.090 [info] <0.60.0> data processed in [16476 ms]
> 2013-06-26 12:45:51.472 [info] <0.60.0> data processed in [18793 ms]
> ...
> 2013-06-26 12:57:28.138 [info] <0.60.0> data processed in [15135 ms]
> 2013-06-26 13:07:01.484 [info] <0.60.0> data processed in [488420 ms]
> 2013-06-26 14:03:11.561 [info] <0.60.0> data processed in [3370075 ms]
>
> In riak crash logs I can see a lot of messages like
>
> 2013-06-26 17:06:20 =CRASH REPORT====
> crasher:
> initial call: riak_kv_index_hashtree:init/1
> pid: <0.13660.7>
> registered_name: []
> exception exit: {{{badmatch,{error,{db_open,"IO error: ./data/anti_entropy/
>     433883298582611803841718934712646521460354973696/MANIFEST-000004:
>     Cannot allocate memory"}}}, [{hashtree,new_segment_store,2,
>     [{file,"src/hashtree.erl"},{line,499}]},
>     {hashtree,new,2,[{file,"src/hashtree.erl"},{line,215}]},
>     {riak_kv_index_hashtree,do_new_tree,2,
>     [{file,"src/riak_kv_index_hashtree.erl"},
>     {line,426}]},{lists,foldl,3,[{file,"lists.erl"},
>     {line,1197}]},{riak_kv_index_hashtree,
>     init_trees,2,[{file,"src/riak_kv_index_hashtree.erl"},
>     {line,368}]},{riak_kv_index_hashtree,init,1,
>     [{file,"src/riak_kv_index_hashtree.erl"},
>
> {line,225}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},
>     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]},
>     [{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},
>     {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
> ancestors: [<0.955.0>,riak_core_vnode_sup,riak_core_sup,<0.129.0>]
> messages: []
> links: []
> dictionary: []
> trap_exit: false
> status: running
> heap_size: 1597
> stack_size: 24
> reductions: 593
> neighbours:
>
> I can see the same behavior on Amazon AWS and local virtual machine. my VM's
> are quite small 512-1024 mb.. AWS is Micro, so it has the same amount of
> memory.
>
> There are no cluster currently. Just single node with riak and my app
> running on it.
>
> I've checked riak documentation and basic things they recommend to do is to
> increase ulimit and to update sysctl. So, my server ulimit shows: ulimit -n
> 65536 AND sysctl updated as recommended.
>
> I've tried bitcask and eleveldb, but result is the same.
>
> Currently, I can't figure out what is broken and why riak Cannot allocate
> memory..
>
> Thanks.
>
>
> Kirill
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>




More information about the riak-users mailing list