Crash Log: yz_anti_entropy

Junk, Damion A junkda at purdue.edu
Wed Jan 18 15:36:43 EST 2017


Hi All -

I'm seeing some strange behavior on a dev server. Right now the riak kv 2.2.0 server has 9GB free Ram, and no keys/values stored in Riak at all. (I've been trying to figure out what's going on with the server, so I completely wiped /var/lib/riak and re-installed from packagecould). Ulimit -n is set appropriately as well.

If I make the following changes to /etc/riak/riak.conf I get crash error messages:

storage_backend = leveldb
search = on

Turning search on with bitcask results in no such crash/error messages. When switching between bitcask/leveldb on this test instance, i'm wiping the /var/lib/riak directory each time. 

Not much is turning up in google, and there is definitely plenty of memory available for allocation. 


Just wondering if anyone else has seen this. 


Thanks!


The error messages:

(crash.log)

2017-01-18 15:27:23 =CRASH REPORT====
  crasher:
    initial call: yz_index_hashtree:init/1
    pid: <0.2356.0>
    registered_name: []
    exception exit: {{{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},{yz_index_hashtree,init,1,[{file,"src/yz_index_hashtree.erl"},{line,191}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
    ancestors: [yz_index_hashtree_sup,yz_general_sup,yz_sup,<0.711.0>]
    messages: []
    links: [<0.2206.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 1598
    stack_size: 27
    reductions: 689
  neighbours:
2017-01-18 15:27:24 =SUPERVISOR REPORT====
     Supervisor: {local,yz_index_hashtree_sup}
     Context:    child_terminated
     Reason:     {{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},{yz_index_hashtree,init,1,[{file,"src/yz_index_hashtree.erl"},{line,191}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
     Offender:   [{pid,<0.2356.0>},{name,ignored},{mfargs,{yz_index_hashtree,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]



(console.log)

2017-01-18 15:35:39.100 [error] <0.3049.0> CRASH REPORT Process <0.3049.0> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}} in hashtree:new_segment_store/2 line 725 in gen_server:init_it/6 line 328
2017-01-18 15:35:39.101 [error] <0.2206.0> Supervisor yz_index_hashtree_sup had child ignored started with {yz_index_hashtree,start_link,undefined} at <0.3049.0> exit with reason no match of right hand value {error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}} in hashtree:new_segment_store/2 line 725 in context child_terminated
2017-01-18 15:35:39.101 [info] <0.3050.0>@riak_core_throttle:enable_throttle:119 Enabling throttle for yokozuna/aae_throttle.
2017-01-18 15:35:39.101 [error] <0.3030.0> gen_server yz_entropy_mgr terminated with reason: no match of right hand value {error,{{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},...]}} in yz_entropy_mgr:'-reload_hashtrees/3-fun-0-'/2 line 371
2017-01-18 15:35:39.101 [error] <0.3030.0> CRASH REPORT Process yz_entropy_mgr with 0 neighbours exited with reason: no match of right hand value {error,{{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},...]}} in yz_entropy_mgr:'-reload_hashtrees/3-fun-0-'/2 line 371 in gen_server:terminate/6 line 744
2017-01-18 15:35:39.102 [error] <0.2010.0> Supervisor yz_general_sup had child yz_entropy_mgr started with yz_entropy_mgr:start_link() at <0.3030.0> exit with reason no match of right hand value {error,{{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},...]}} in yz_entropy_mgr:'-reload_hashtrees/3-fun-0-'/2 line 371 in context child_terminated



More information about the riak-users mailing list