Crash Log: yz_anti_entropy

Magnus Kessler mkessler at basho.com
Thu Jan 19 04:33:14 EST 2017


On 18 January 2017 at 20:36, Junk, Damion A <junkda at purdue.edu> wrote:

> Hi All -
>
> I'm seeing some strange behavior on a dev server. Right now the riak kv
> 2.2.0 server has 9GB free Ram, and no keys/values stored in Riak at all.
> (I've been trying to figure out what's going on with the server, so I
> completely wiped /var/lib/riak and re-installed from packagecould). Ulimit
> -n is set appropriately as well.
>
> If I make the following changes to /etc/riak/riak.conf I get crash error
> messages:
>
> storage_backend = leveldb
> search = on
>
> Turning search on with bitcask results in no such crash/error messages.
> When switching between bitcask/leveldb on this test instance, i'm wiping
> the /var/lib/riak directory each time.
>
> Not much is turning up in google, and there is definitely plenty of memory
> available for allocation.
>
>
> Just wondering if anyone else has seen this.
>
>
> Thanks!
>
>
> The error messages:
>
> (crash.log)
>
> 2017-01-18 15:27:23 =CRASH REPORT====
>   crasher:
>     initial call: yz_index_hashtree:init/1
>     pid: <0.2356.0>
>     registered_name: []
>     exception exit: {{{badmatch,{error,{db_open,"IO error: lock
> /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK:
> Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/
> hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/
> hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,
> 3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{
> lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_
> hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{
> line,340}]},{yz_index_hashtree,init,1,[{file,"src/
> yz_index_hashtree.erl"},{line,191}]},{gen_server,init_it,6,[
> {file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_
> apply,3,[{file,"proc_lib.erl"},{line,239}]}]},[{gen_server,
> init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,
> init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
>     ancestors: [yz_index_hashtree_sup,yz_general_sup,yz_sup,<0.711.0>]
>     messages: []
>     links: [<0.2206.0>]
>     dictionary: []
>     trap_exit: true
>     status: running
>     heap_size: 1598
>     stack_size: 27
>     reductions: 689
>   neighbours:
> 2017-01-18 15:27:24 =SUPERVISOR REPORT====
>      Supervisor: {local,yz_index_hashtree_sup}
>      Context:    child_terminated
>      Reason:     {{badmatch,{error,{db_open,"IO error: lock
> /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK:
> Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/
> hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/
> hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,
> 3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{
> lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_
> hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{
> line,340}]},{yz_index_hashtree,init,1,[{file,"src/
> yz_index_hashtree.erl"},{line,191}]},{gen_server,init_it,6,[
> {file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_
> apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
>      Offender:   [{pid,<0.2356.0>},{name,ignored},{mfargs,{yz_index_
> hashtree,start_link,undefined}},{restart_type,temporary},{
> shutdown,5000},{child_type,worker}]
>
>
>
> (console.log)
>
> 2017-01-18 15:35:39.100 [error] <0.3049.0> CRASH REPORT Process <0.3049.0>
> with 0 neighbours exited with reason: no match of right hand value
> {error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/
> 639406966332270026714112114313373821099470487552/LOCK: Cannot allocate
> memory"}} in hashtree:new_segment_store/2 line 725 in gen_server:init_it/6
> line 328
> 2017-01-18 15:35:39.101 [error] <0.2206.0> Supervisor
> yz_index_hashtree_sup had child ignored started with
> {yz_index_hashtree,start_link,undefined} at <0.3049.0> exit with reason
> no match of right hand value {error,{db_open,"IO error: lock
> /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK:
> Cannot allocate memory"}} in hashtree:new_segment_store/2 line 725 in
> context child_terminated
> 2017-01-18 15:35:39.101 [info] <0.3050.0>@riak_core_throttle:enable_throttle:119
> Enabling throttle for yokozuna/aae_throttle.
> 2017-01-18 15:35:39.101 [error] <0.3030.0> gen_server yz_entropy_mgr
> terminated with reason: no match of right hand value
> {error,{{badmatch,{error,{db_open,"IO error: lock
> /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK:
> Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/
> hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/
> hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,
> 3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{
> lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_
> hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},...]}}
> in yz_entropy_mgr:'-reload_hashtrees/3-fun-0-'/2 line 371
> 2017-01-18 15:35:39.101 [error] <0.3030.0> CRASH REPORT Process
> yz_entropy_mgr with 0 neighbours exited with reason: no match of right hand
> value {error,{{badmatch,{error,{db_open,"IO error: lock
> /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK:
> Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/
> hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/
> hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,
> 3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{
> lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_
> hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},...]}}
> in yz_entropy_mgr:'-reload_hashtrees/3-fun-0-'/2 line 371 in
> gen_server:terminate/6 line 744
> 2017-01-18 15:35:39.102 [error] <0.2010.0> Supervisor yz_general_sup had
> child yz_entropy_mgr started with yz_entropy_mgr:start_link() at <0.3030.0>
> exit with reason no match of right hand value {error,{{badmatch,{error,{db_open,"IO
> error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK:
> Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/
> hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/
> hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,
> 3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{
> lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_
> hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},...]}}
> in yz_entropy_mgr:'-reload_hashtrees/3-fun-0-'/2 line 371 in context
> child_terminated
>

Hi Damion,

Let me first state that AAE always uses leveldb, regardless of the storage
backend chosen for Riak KV data. Could you please state how much physical
memory your Riak nodes have, and what you have configured for
"leveldb.maximum_memory.percent" in "riak.conf"? Have you changed the
settings for "search.solr.jvm_options", in particular the memory allocated
to Solr?

As a general rule, leveldb should have at least 350MB of memory available
per partition, and performance has been shown to increase with up to 2GB
(2.5 GB when also using Search and AAE) per partition. Please check that
you have enough memory available in your system.

Kind Regards,

Magnus

-- 
Magnus Kessler
Client Services Engineer
Basho Technologies Limited

Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20170119/d6956bf2/attachment-0002.html>


More information about the riak-users mailing list