Riak crashed and crashed again when recovering

Germain Maurice germain.maurice at linkfluence.net
Wed May 5 05:35:41 EDT 2010


Hi all,
I am testing Riak for my document base and i got a problem when i was 
migrating documents from my previous
system to Riak.
I have two nodes and one bucket for the beginning.
There are more than 480 000 documents in the bucket and the documents 
are html pages.

In the following you'll find all the files and informations after a node 
was restarted.
After a while, riak crashed again for the two nodes I restarted ... :(

$ ls -lh /reiser/riak/dets/
total 49G
[...]
-rw-r--r-- 1 riak riak 2,0G 2010-05-05 07:22 
639406966332270026714112114313373821099470487552
-rw-r--r-- 1 riak riak 260K 2010-05-05 11:06 
639406966332270026714112114313373821099470487552.TMP
-rw-r--r-- 1 riak riak 3,7M 2010-05-05 07:24 
662242929415565384811044689824565743281594433536
-rw-r--r-- 1 riak riak 3,3M 2010-05-05 07:24 
68507889249886074290797726533575766546371837952
-rw-r--r-- 1 riak riak 2,0G 2010-05-05 07:27 
685078892498860742907977265335757665463718379520
-rw-r--r-- 1 riak riak 2,8M 2010-05-05 07:24 
707914855582156101004909840846949587645842325504
-rw-r--r-- 1 riak riak 2,0G 2010-05-05 07:27 
730750818665451459101842416358141509827966271488
-rw-r--r-- 1 riak riak 2,0M 2010-05-05 07:24 
753586781748746817198774991869333432010090217472
-rw-r--r-- 1 riak riak 1,9G 2010-05-05 07:28 
776422744832042175295707567380525354192214163456
-rw-r--r-- 1 riak riak 2,9M 2010-05-05 07:24 
799258707915337533392640142891717276374338109440
-rw-r--r-- 1 riak riak 2,0G 2010-05-05 07:28 
822094670998632891489572718402909198556462055424
-rw-r--r-- 1 riak riak 3,4M 2010-05-05 07:24 
844930634081928249586505293914101120738586001408
-rw-r--r-- 1 riak riak 2,0G 2010-05-05 07:27 
867766597165223607683437869425293042920709947392
-rw-r--r-- 1 riak riak 4,7M 2010-05-05 07:24 
890602560248518965780370444936484965102833893376
-rw-r--r-- 1 riak riak 1,9G 2010-05-05 07:27 
91343852333181432387730302044767688728495783936
-rw-r--r-- 1 riak riak 2,0G 2010-05-05 07:27 
913438523331814323877303020447676887284957839360
-rw-r--r-- 1 riak riak 5,3M 2010-05-05 07:24 
936274486415109681974235595958868809467081785344
-rw-r--r-- 1 riak riak 2,0G 2010-05-05 07:27 
959110449498405040071168171470060731649205731328
-rw-r--r-- 1 riak riak 2,9M 2010-05-05 07:24 
981946412581700398168100746981252653831329677312
-rw-r--r-- 1 riak riak 140M 2010-05-05 11:08 
fs_riak at 10.0.0.40_5223_1273_50399_129540.273
-rw-r--r-- 1 riak riak 8,7M 2010-05-05 11:08 
fs_riak at 10.0.0.40_5223_1273_50399_129540.290
-rw-r--r-- 1 riak riak 9,1M 2010-05-05 11:08 
fs_riak at 10.0.0.40_5223_1273_50399_129540.307
-rw-r--r-- 1 riak riak 8,6M 2010-05-05 11:08 
fs_riak at 10.0.0.40_5223_1273_50399_129540.324
-rw-r--r-- 1 riak riak 8,9M 2010-05-05 11:08 
fs_riak at 10.0.0.40_5223_1273_50399_129540.341
-rw-r--r-- 1 riak riak 8,7M 2010-05-05 11:08 
fs_riak at 10.0.0.40_5223_1273_50399_129540.358
-rw-r--r-- 1 riak riak 9,0M 2010-05-05 11:08 
fs_riak at 10.0.0.40_5223_1273_50399_129540.375
-rw-r--r-- 1 riak riak 8,7M 2010-05-05 11:08 
fs_riak at 10.0.0.40_5223_1273_50399_129540.392
-rw-r--r-- 1 riak riak 8,7M 2010-05-05 11:08 
fs_riak at 10.0.0.40_5223_1273_50399_129540.409
-rw-r--r-- 1 riak riak 8,5M 2010-05-05 11:09 
fs_riak at 10.0.0.40_5223_1273_50399_129540.426
-rw-r--r-- 1 riak riak 9,0M 2010-05-05 11:09 
fs_riak at 10.0.0.40_5223_1273_50399_129540.443
-rw-r--r-- 1 riak riak 8,8M 2010-05-05 11:09 
fs_riak at 10.0.0.40_5223_1273_50399_129540.460
-rw-r--r-- 1 riak riak 8,7M 2010-05-05 11:09 
fs_riak at 10.0.0.40_5223_1273_50399_129540.477
-rw-r--r-- 1 riak riak 9,3M 2010-05-05 11:09 
fs_riak at 10.0.0.40_5223_1273_50399_129540.494
-rw-r--r-- 1 riak riak 581K 2010-05-05 11:09 
fs_riak at 10.0.0.40_5223_1273_50399_129540.495
-rw-r--r-- 1 riak riak 546K 2010-05-05 11:09 
fs_riak at 10.0.0.40_5223_1273_50399_129540.496
-rw-r--r-- 1 riak riak 571K 2010-05-05 11:09 
fs_riak at 10.0.0.40_5223_1273_50399_129540.497




$ curl 
http://10.0.0.40:8098/riak/mybucket/1714432724f7f975610be47146fec6c7e74bf4bbccdbeea5208ac6e3540e6f4b
<html><head><title>500 Internal Server 
Error</title></head><body><h1>Internal Server Error</h1>The server 
encountered an error while processing this 
request:<br><pre>[{webmachine_decision_core,'-decision/1-lc$^1/1-1-',
      [{error,
           {error,
               {case_clause,{error,timeout}},
               [{riak_kv_wm_raw,content_types_provided,2},
                {webmachine_resource,resource_call,3},
                {webmachine_resource,do,3},
                {webmachine_decision_core,resource_call,1},
                {webmachine_decision_core,decision,1},
                {webmachine_decision_core,handle_request,2},
                {webmachine_mochiweb,loop,1},
                {mochiweb_http,headers,5}]}}]},
  {webmachine_decision_core,decision,1},
  {webmachine_decision_core,handle_request,2},
  {webmachine_mochiweb,loop,1},
  {mochiweb_http,headers,5},
  {proc_lib,init_p_do_apply,3}]</pre><P><HR><ADDRESS>mochiweb+webmachine 
web server</ADDRESS></body></html>


$ riak-admin status
1-minute stats for 'riak at 10.0.0.40'
-------------------------------------------
vnode gets : 0
vnode_puts : 0
vnode_gets_total : 0
vnode_puts_total : 0
node_gets : 0
node_gets_total : 3
node_get_fsm_time_mean : undefined
node_get_fsm_time_median : undefined
node_get_fsm_time_95 : undefined
node_get_fsm_time_99 : undefined
node_get_fsm_time_100 : undefined
node_puts : 0
node_puts_total : 0
node_put_fsm_time_mean : undefined
node_put_fsm_time_median : undefined
node_put_fsm_time_95 : undefined
node_put_fsm_time_99 : undefined
node_put_fsm_time_100 : undefined
cpu_nprocs : 124
cpu_avg1 : 312
cpu_avg5 : 248
cpu_avg15 : 207
mem_total : 1950601216
mem_allocated : 1931788288
disk : [{"/",86796672,5},
         {"/dev",952440,1},
         {"/dev/shm",952440,0},
         {"/var/run",952440,1},
         {"/var/lock",952440,0},
         {"/lib/init/rw",952440,0},
         {"/reiser",1218709872,28}]
nodename : 'riak at 10.0.0.40'
connected_nodes : ['riak_maint_5420 at 10.0.0.40']
sys_driver_version : <<"1.5">>
sys_global_heaps_size : 0
sys_heap_type : private
sys_logical_processors : 2
sys_otp_release : <<"R13B04">>
sys_process_count : 140
sys_smp_support : true
sys_system_version : <<"Erlang R13B04 (erts-5.7.5) [source] [64-bit] 
[smp:2:2] [rq:2] [async-threads:5] [hipe] [kernel-poll:true]\n">>
sys_system_architecture : <<"x86_64-unknown-linux-gnu">>
sys_threads_enabled : true
sys_thread_pool_size : 5
sys_wordsize : 8
ring_members : ['riak at 10.0.0.40','riak at 10.0.0.41']
ring_num_partitions : 64
ring_ownership : <<"[{'riak at 10.0.0.40',32},{'riak at 10.0.0.41',32}]">>
ring_creation_size : 64
storage_backend : riak_kv_dets_backend
pbc_connects_total : 0
pbc_connects : 0
pbc_active : 0

Any idea about this behavior ?
Can you explain what does riak with the fs_riak at 10.0.0.40_* files ?


Thank you

-- 
Germain Maurice
Administrateur Système/Réseau
Tel : +33.(0)1.42.43.54.33

http://www.linkfluence.net





More information about the riak-users mailing list