Write_lock error has occurred after inserting 12M data

Ken Matsumoto ken at nri.com
Fri Jul 30 17:33:08 EDT 2010


Hi Grant, Alexander, David,

Thank you for your messages.
Here are the infos:
Version : 0.11.0-1344 debian package
Key length : 36B

I append the error part of the log file:

=ERROR REPORT==== 30-Jul-2010::04:00:54 ===
** State machine <0.28753.1381> terminating
** Last event in was {put,<0.28729.1381>,
                          {<<"lpb_probes">>,
                           <<"e2ba34b6-20c0-493a-b248-a031e1ab6299">>},
                          {r_object,<<"lpb_probes">>,
                              <<"e2ba34b6-20c0-493a-b248-a031e1ab6299">>,
                              [{r_content,
                                   {dict,3,16,16,8,80,48,

{[],[],[],[],[],[],[],[],[],[],[],[],[],
                                        [],[],[]},
                                       {{[],[],[],[],[],[],[],[],[],[],
 
[[<<"content-type">>,97,112,112,108,
                                           105,99,97,116,105,111,110,47,106,
                                           115,111,110],

[<<"X-Riak-VTag">>,51,111,113,68,70,

105,50,68,117,56,72,66,90,102,68,73,
                                           84,56,80,67,112,65]],
                                         [],[],
                                         [[<<"X-Riak-Last-Modified">>|
                                           {1280,487640,458574}]],
                                         [],[]}}},

<<"{\"gettime\":\"121850\",\"getdate\":\"20080802\",\"meshcode\":\"503005\",\"longitude\":\"8154510\",\"latitude\":\"1940739\"}">>}],
                              [{<<5,60,63,24>>,{1,63447706840}}],
                              {dict,1,16,16,8,80,48,
 
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                   [],[]},

{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                                    [[clean|true]],
                                    []}}},
                              undefined},
                          72719594,63447706840,
                          [{returnbody,false}]}
** When State == active
**      Data  == {state,456719261665907161938651510223838443642478919680,[],
                         riak_kv_bitcask_backend,
                         {#Ref<0.0.172.25968>,

"/var/lib/riak/bitcask/456719261665907161938651510223838443642478919680"},
                         not_in_handoff,undefined}
** Reason for termination =
** {bad_return_value,{error,{write_locked,locked}}}



Best,

Ken.

On 2010/07/30 14:13, Alex Wolfe wrote:
> Hey David.
>
> Does the below log output look like it could be caused by the issue you
> fixed?
>
> Alex
>
>
> ==== Fri Jul 30 14:22:34 CDT 2010
>
> =ERROR REPORT==== 30-Jul-2010::14:22:34 ===
> ** State machine <0.176.0> terminating
> ** Last event in was {riak_vnode_req_v1,
> 593735040165679310520246963290989976735222595584,
> {fsm,undefined,<0.12466.0>},
> {riak_kv_put_req_v1,
> {<<"test.groups">>,<<"EghzXywWrGGtp2fCcSLoatIdjML">>},
> {r_object,<<"test.groups">>,
> <<"EghzXywWrGGtp2fCcSLoatIdjML">>,
> [{r_content,
> {dict,5,16,16,8,80,48,
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
> {{[],[],
> [[<<"Links">>]],
> [],[],[],[],[],[],[],
> [[<<"content-type">>,97,112,112,108,105,99,97,
> 116,105,111,110,47,106,115,111,110],
> [<<"X-Riak-VTag">>,89,69,78,55,55,111,66,121,73,
> 69,78,53,122,101,85,105,117,68,89,80,52]],
> [],[],
> [[<<"X-Riak-Last-Modified">>|
> {1280,517754,951062}]],
> [],
> [[<<"X-Riak-Meta">>]]}}},
> <<"{\"name\":\"foo\",\"created_at\":\"2010-07-30T19:22:34.947Z\",\"type\":\"group\",\"version\":1}">>}],
> [{<<0,55,119,231>>,{1,63447736954}}],
> {dict,1,16,16,8,80,48,
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> [[clean|true]],
> []}}},
> undefined},
> 33218311,63447736954,
> [{returnbody,true}]}}
> ** When State == active
> ** Data == {state,593735040165679310520246963290989976735222595584,
> riak_kv_vnode,
> {state,593735040165679310520246963290989976735222595584,
> riak_kv_bitcask_backend,
> {#Ref<0.0.0.611>,
> "data/bitcask/593735040165679310520246963290989976735222595584"},
> [],false},
> undefined,none}
> ** Reason for termination =
> ** {{badmatch,{error,emfile}},
> [{bitcask_fileops,create_file_loop,3},
> {bitcask,put,3},
> {riak_kv_bitcask_backend,put,3},
> {riak_kv_vnode,perform_put,3},
> {riak_kv_vnode,do_put,7},
> {riak_kv_vnode,handle_command,3},
> {riak_core_vnode,vnode_command,3},
> {gen_fsm,handle_msg,7}]}
>
> =ERROR REPORT==== 30-Jul-2010::14:22:35 ===
> Failed to open lock file
> data/bitcask/593735040165679310520246963290989976735222595584/bitcask.write.lock:
> emfile
>
> =ERROR REPORT==== 30-Jul-2010::14:22:35 ===
> ** State machine <0.12471.0> terminating
> ** Last event in was {riak_vnode_req_v1,
> 593735040165679310520246963290989976735222595584,
> {fsm,undefined,<0.12470.0>},
> {riak_kv_put_req_v1,
> {<<"test.users">>,<<"ZrAxzFghd51VG902GuCcJ2gYOMJ">>},
> {r_object,<<"test.users">>,
> <<"ZrAxzFghd51VG902GuCcJ2gYOMJ">>,
> [{r_content,
> {dict,5,16,16,8,80,48,
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
> {{[],[],
> [[<<"Links">>,
> {{<<"test.groups">>,
> <<"EghzXywWrGGtp2fCcSLoatIdjML">>},
> <<"groups">>}]],
> [],[],[],[],[],[],[],
> [[<<"content-type">>,97,112,112,108,105,99,97,
> 116,105,111,110,47,106,115,111,110],
> [<<"X-Riak-VTag">>,50,88,97,105,85,49,70,49,55,
> 54,48,71,113,115,75,103,54,102,84,56,118,84]],
> [],[],
> [[<<"X-Riak-Last-Modified">>|{1280,517755,638}]],
> [],
> [[<<"X-Riak-Meta">>]]}}},
> <<"{\"name\":\"bar\",\"created_at\":\"2010-07-30T19:22:34.998Z\",\"type\":\"user\",\"version\":1}">>}],
> [{<<3,30,180,15>>,{1,63447736955}}],
> {dict,1,16,16,8,80,48,
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> [[clean|true]],
> []}}},
> undefined},
> 122945524,63447736955,
> [{returnbody,true}]}}
> ** When State == active
> ** Data == {state,593735040165679310520246963290989976735222595584,
> riak_kv_vnode,
> {state,593735040165679310520246963290989976735222595584,
> riak_kv_bitcask_backend,
> {#Ref<0.0.0.16357>,
> "data/bitcask/593735040165679310520246963290989976735222595584"},
> [],false},
> undefined,none}
> ** Reason for termination =
> ** {bad_return_value,{error,{write_locked,emfile}}}
>
> ==> sasl-error.log <==
>
> =CRASH REPORT==== 30-Jul-2010::14:22:35 ===
> crasher:
> initial call: riak_core_vnode:init/1
> pid: <0.176.0>
> registered_name: []
> exception exit: {{badmatch,{error,emfile}},
> [{bitcask_fileops,create_file_loop,3},
> {bitcask,put,3},
> {riak_kv_bitcask_backend,put,3},
> {riak_kv_vnode,perform_put,3},
> {riak_kv_vnode,do_put,7},
> {riak_kv_vnode,handle_command,3},
> {riak_core_vnode,vnode_command,3},
> {gen_fsm,handle_msg,7}]}
> in function gen_fsm:terminate/7
> ancestors: [riak_core_vnode_sup,riak_core_sup,<0.98.0>]
> messages: []
> links: [#Port<0.3507>,<0.100.0>]
> dictionary: []
> trap_exit: true
> status: running
> heap_size: 1597
> stack_size: 24
> reductions: 11185
> neighbours:
>
> =SUPERVISOR REPORT==== 30-Jul-2010::14:22:35 ===
> Supervisor: {local,riak_core_vnode_sup}
> Context: child_terminated
> Reason: {{badmatch,{error,emfile}},
> [{bitcask_fileops,create_file_loop,3},
> {bitcask,put,3},
> {riak_kv_bitcask_backend,put,3},
> {riak_kv_vnode,perform_put,3},
> {riak_kv_vnode,do_put,7},
> {riak_kv_vnode,handle_command,3},
> {riak_core_vnode,vnode_command,3},
> {gen_fsm,handle_msg,7}]}
> Offender: [{pid,<0.176.0>},
> {name,undefined},
> {mfa,
> {riak_core_vnode,start_link,
> [riak_kv_vnode,
> 593735040165679310520246963290989976735222595584]}},
> {restart_type,temporary},
> {shutdown,brutal_kill},
> {child_type,worker}]
>
>
> =CRASH REPORT==== 30-Jul-2010::14:22:35 ===
> crasher:
> initial call: riak_core_vnode:init/1
> pid: <0.12471.0>
> registered_name: []
> exception exit: {bad_return_value,{error,{write_locked,emfile}}}
> in function gen_fsm:terminate/7
> ancestors: [riak_core_vnode_sup,riak_core_sup,<0.98.0>]
> messages: []
> links: [<0.100.0>]
> dictionary: []
> trap_exit: true
> status: running
> heap_size: 2584
> stack_size: 24
> reductions: 1955
> neighbours:
>
> =SUPERVISOR REPORT==== 30-Jul-2010::14:22:35 ===
> Supervisor: {local,riak_core_vnode_sup}
> Context: child_terminated
> Reason: {bad_return_value,{error,{write_locked,emfile}}}
> Offender: [{pid,<0.12471.0>},
> {name,undefined},
> {mfa,
> {riak_core_vnode,start_link,
> [riak_kv_vnode,
> 593735040165679310520246963290989976735222595584]}},
> {restart_type,temporary},
> {shutdown,brutal_kill},
> {child_type,worker}]
>
>
> ==> erlang.log.4 <==
>
> =ERROR REPORT==== 30-Jul-2010::14:23:27 ===
> ** Generic server memsup terminating
> ** Last message in was {'EXIT',<0.21020.78>,
> {emfile,
> [{erlang,open_port,
> [{spawn,"/bin/sh -s unix:cmd 2>&1"},
> [stream]]},
> {os,start_port_srv_loop,2}]}}
> ** When Server state == {state,{unix,darwin},
> false,
> {1897500000,7776508000},
> {<0.81.0>,972008},
> false,60000,30000,0.8,0.05,<0.21020.78>,
> #Ref<0.0.2.58535>,undefined,
> [reg],
> []}
> ** Reason for termination ==
> ** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
> 2>&1"},[stream]]},
> {os,start_port_srv_loop,2}]}
>
> =ERROR REPORT==== 30-Jul-2010::14:23:27 ===
> ** Generic server memsup terminating
> ** Last message in was {'EXIT',<0.21186.78>,
> {emfile,
> [{erlang,open_port,
> [{spawn,"/bin/sh -s unix:cmd 2>&1"},
> [stream]]},
> {os,start_port_srv_loop,2}]}}
> ** When Server state == {state,{unix,darwin},
> false,undefined,undefined,false,60000,30000,
> 0.8,0.05,<0.21186.78>,#Ref<0.0.2.58551>,
> undefined,
> [reg],
> []}
> ** Reason for termination ==
> ** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
> 2>&1"},[stream]]},
> {os,start_port_srv_loop,2}]}
>
> =ERROR REPORT==== 30-Jul-2010::14:23:27 ===
> ** Generic server memsup terminating
> ** Last message in was {'EXIT',<0.21252.78>,
> {emfile,
> [{erlang,open_port,
> [{spawn,"/bin/sh -s unix:cmd 2>&1"},
> [stream]]},
> {os,start_port_srv_loop,2}]}}
> ** When Server state == {state,{unix,darwin},
> false,undefined,undefined,false,60000,30000,
> 0.8,0.05,<0.21252.78>,#Ref<0.0.2.58559>,
> undefined,
> [reg],
> []}
> ** Reason for termination ==
> ** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
> 2>&1"},[stream]]},
> {os,start_port_srv_loop,2}]}
>
> =ERROR REPORT==== 30-Jul-2010::14:23:27 ===
> ** Generic server memsup terminating
> ** Last message in was {'EXIT',<0.21374.78>,
> {emfile,
> [{erlang,open_port,
> [{spawn,"/bin/sh -s unix:cmd 2>&1"},
> [stream]]},
> {os,start_port_srv_loop,2}]}}
> ** When Server state == {state,{unix,darwin},
> false,undefined,undefined,false,60000,30000,
> 0.8,0.05,<0.21374.78>,#Ref<0.0.2.58569>,
> undefined,
> [reg],
> []}
> ** Reason for termination ==
> ** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
> 2>&1"},[stream]]},
> {os,start_port_srv_loop,2}]}
>
> =ERROR REPORT==== 30-Jul-2010::14:23:27 ===
> ** Generic server memsup terminating
> ** Last message in was {'EXIT',<0.21396.78>,
> {emfile,
> [{erlang,open_port,
> [{spawn,"/bin/sh -s unix:cmd 2>&1"},
> [stream]]},
> {os,start_port_srv_loop,2}]}}
> ** When Server state == {state,{unix,darwin},
> false,undefined,undefined,false,60000,30000,
> 0.8,0.05,<0.21396.78>,#Ref<0.0.2.58575>,
> undefined,
> [reg],
> []}
> ** Reason for termination ==
> ** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd
> 2>&1"},[stream]]},
> {os,start_port_srv_loop,2}]}
>
> =ERROR REPORT==== 30-Jul-2010::14:23:27 ===
> ** State machine <0.175.0> terminating
> ** Last event in was {riak_vnode_req_v1,
> 570899077082383952423314387779798054553098649600,
> {fsm,undefined,<0.12470.0>},
> {riak_kv_put_req_v1,
> {<<"test.users">>,<<"ZrAxzFghd51VG902GuCcJ2gYOMJ">>},
> {r_object,<<"test.users">>,
> <<"ZrAxzFghd51VG902GuCcJ2gYOMJ">>,
> [{r_content,
> {dict,5,16,16,8,80,48,
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
> {{[],[],
> [[<<"Links">>,
> {{<<"test.groups">>,
> <<"EghzXywWrGGtp2fCcSLoatIdjML">>},
> <<"groups">>}]],
> [],[],[],[],[],[],[],
> [[<<"content-type">>,97,112,112,108,105,99,97,
> 116,105,111,110,47,106,115,111,110],
> [<<"X-Riak-VTag">>,50,88,97,105,85,49,70,49,55,
> 54,48,71,113,115,75,103,54,102,84,56,118,84]],
> [],[],
> [[<<"X-Riak-Last-Modified">>|{1280,517755,638}]],
> [],
> [[<<"X-Riak-Meta">>]]}}},
> <<"{\"name\":\"bar\",\"created_at\":\"2010-07-30T19:22:34.998Z\",\"type\":\"user\",\"version\":1}">>}],
> [{<<3,30,180,15>>,{1,63447736955}}],
> {dict,1,16,16,8,80,48,
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> [[clean|true]],
> []}}},
> undefined},
> 122945524,63447736955,
> [{returnbody,true}]}}
> ** When State == active
> ** Data == {state,570899077082383952423314387779798054553098649600,
> riak_kv_vnode,
> {state,570899077082383952423314387779798054553098649600,
> riak_kv_bitcask_backend,
> {#Ref<0.0.0.598>,
> "data/bitcask/570899077082383952423314387779798054553098649600"},
> [],false},
> undefined,none}
> ** Reason for termination =
> ** {{badmatch,{error,emfile}},
> [{bitcask_fileops,create_file_loop,3},
> {bitcask,put,3},
> {riak_kv_bitcask_backend,put,3},
> {riak_kv_vnode,perform_put,3},
> {riak_kv_vnode,do_put,7},
> {riak_kv_vnode,handle_command,3},
> {riak_core_vnode,vnode_command,3},
> {gen_fsm,handle_msg,7}]}
>
> =INFO REPORT==== 30-Jul-2010::14:23:27 ===
> application: os_mon
> exited: shutdown
> type: permanent
>
> =INFO REPORT==== 30-Jul-2010::14:23:27 ===
> Spidermonkey VM host stopping (<0.139.0>)
>
> =INFO REPORT==== 30-Jul-2010::14:23:27 ===
> Spidermonkey VM host stopping (<0.137.0>)
>
> =INFO REPORT==== 30-Jul-2010::14:23:27 ===
> Spidermonkey VM host stopping (<0.144.0>)
>
> =INFO REPORT==== 30-Jul-2010::14:23:27 ===
> Spidermonkey VM host stopping (<0.140.0>)
>
> =INFO REPORT==== 30-Jul-2010::14:23:27 ===
> Spidermonkey VM host stopping (<0.138.0>)
>
> =INFO REPORT==== 30-Jul-2010::14:23:27 ===
> Spidermonkey VM host stopping (<0.142.0>)
>
> ==> sasl-error.log <==
>
> =CRASH REPORT==== 30-Jul-2010::14:23:27 ===
> crasher:
> initial call: memsup:init/1
> pid: <0.52.0>
> registered_name: memsup
> exception exit: {emfile,[{erlang,open_port,
> [{spawn,"/bin/sh -s unix:cmd 2>&1"},
> [stream]]},
> {os,start_port_srv_loop,2}]}
> in function gen_server:terminate/6
> ancestors: [os_mon_sup,<0.49.0>]
> messages: []
> links: [<0.50.0>]
> dictionary: []
> trap_exit: true
> status: running
> heap_size: 2584
> stack_size: 24
> reductions: 53031
> neighbours:
>
> =SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===
> Supervisor: {local,os_mon_sup}
> Context: child_terminated
> Reason: {emfile,[{erlang,open_port,
> [{spawn,"/bin/sh -s unix:cmd 2>&1"},
> [stream]]},
> {os,start_port_srv_loop,2}]}
> Offender: [{pid,<0.52.0>},
> {name,memsup},
> {mfa,{memsup,start_link,[]}},
> {restart_type,permanent},
> {shutdown,2000},
> {child_type,worker}]
>
>
> =CRASH REPORT==== 30-Jul-2010::14:23:27 ===
> crasher:
> initial call: memsup:init/1
> pid: <0.21163.78>
> registered_name: memsup
> exception exit: {emfile,[{erlang,open_port,
> [{spawn,"/bin/sh -s unix:cmd 2>&1"},
> [stream]]},
> {os,start_port_srv_loop,2}]}
> in function gen_server:terminate/6
> ancestors: [os_mon_sup,<0.49.0>]
> messages: []
> links: [<0.50.0>]
> dictionary: []
> trap_exit: true
> status: running
> heap_size: 377
> stack_size: 24
> reductions: 181
> neighbours:
>
> =SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===
> Supervisor: {local,os_mon_sup}
> Context: child_terminated
> Reason: {emfile,[{erlang,open_port,
> [{spawn,"/bin/sh -s unix:cmd 2>&1"},
> [stream]]},
> {os,start_port_srv_loop,2}]}
> Offender: [{pid,<0.21163.78>},
> {name,memsup},
> {mfa,{memsup,start_link,[]}},
> {restart_type,permanent},
> {shutdown,2000},
> {child_type,worker}]
>
>
> =CRASH REPORT==== 30-Jul-2010::14:23:27 ===
> crasher:
> initial call: memsup:init/1
> pid: <0.21226.78>
> registered_name: memsup
> exception exit: {emfile,[{erlang,open_port,
> [{spawn,"/bin/sh -s unix:cmd 2>&1"},
> [stream]]},
> {os,start_port_srv_loop,2}]}
> in function gen_server:terminate/6
> ancestors: [os_mon_sup,<0.49.0>]
> messages: []
> links: [<0.50.0>]
> dictionary: []
> trap_exit: true
> status: running
> heap_size: 377
> stack_size: 24
> reductions: 181
> neighbours:
>
> =SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===
> Supervisor: {local,os_mon_sup}
> Context: child_terminated
> Reason: {emfile,[{erlang,open_port,
> [{spawn,"/bin/sh -s unix:cmd 2>&1"},
> [stream]]},
> {os,start_port_srv_loop,2}]}
> Offender: [{pid,<0.21226.78>},
> {name,memsup},
> {mfa,{memsup,start_link,[]}},
> {restart_type,permanent},
> {shutdown,2000},
> {child_type,worker}]
>
>
> =CRASH REPORT==== 30-Jul-2010::14:23:27 ===
> crasher:
> initial call: memsup:init/1
> pid: <0.21343.78>
> registered_name: memsup
> exception exit: {emfile,[{erlang,open_port,
> [{spawn,"/bin/sh -s unix:cmd 2>&1"},
> [stream]]},
> {os,start_port_srv_loop,2}]}
> in function gen_server:terminate/6
> ancestors: [os_mon_sup,<0.49.0>]
> messages: []
> links: [<0.50.0>]
> dictionary: []
> trap_exit: true
> status: running
> heap_size: 377
> stack_size: 24
> reductions: 193
> neighbours:
>
> =SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===
> Supervisor: {local,os_mon_sup}
> Context: child_terminated
> Reason: {emfile,[{erlang,open_port,
> [{spawn,"/bin/sh -s unix:cmd 2>&1"},
> [stream]]},
> {os,start_port_srv_loop,2}]}
> Offender: [{pid,<0.21343.78>},
> {name,memsup},
> {mfa,{memsup,start_link,[]}},
> {restart_type,permanent},
> {shutdown,2000},
> {child_type,worker}]
>
>
> =CRASH REPORT==== 30-Jul-2010::14:23:27 ===
> crasher:
> initial call: memsup:init/1
> pid: <0.21393.78>
> registered_name: memsup
> exception exit: {emfile,[{erlang,open_port,
> [{spawn,"/bin/sh -s unix:cmd 2>&1"},
> [stream]]},
> {os,start_port_srv_loop,2}]}
> in function gen_server:terminate/6
> ancestors: [os_mon_sup,<0.49.0>]
> messages: []
> links: [<0.50.0>]
> dictionary: []
> trap_exit: true
> status: running
> heap_size: 377
> stack_size: 24
> reductions: 189
> neighbours:
>
> =SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===
> Supervisor: {local,os_mon_sup}
> Context: child_terminated
> Reason: {emfile,[{erlang,open_port,
> [{spawn,"/bin/sh -s unix:cmd 2>&1"},
> [stream]]},
> {os,start_port_srv_loop,2}]}
> Offender: [{pid,<0.21393.78>},
> {name,memsup},
> {mfa,{memsup,start_link,[]}},
> {restart_type,permanent},
> {shutdown,2000},
> {child_type,worker}]
>
>
> =SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===
> Supervisor: {local,os_mon_sup}
> Context: shutdown
> Reason: reached_max_restart_intensity
> Offender: [{pid,<0.21393.78>},
> {name,memsup},
> {mfa,{memsup,start_link,[]}},
> {restart_type,permanent},
> {shutdown,2000},
> {child_type,worker}]
>
>
> =CRASH REPORT==== 30-Jul-2010::14:23:27 ===
> crasher:
> initial call: riak_core_vnode:init/1
> pid: <0.175.0>
> registered_name: []
> exception exit: {{badmatch,{error,emfile}},
> [{bitcask_fileops,create_file_loop,3},
> {bitcask,put,3},
> {riak_kv_bitcask_backend,put,3},
> {riak_kv_vnode,perform_put,3},
> {riak_kv_vnode,do_put,7},
> {riak_kv_vnode,handle_command,3},
> {riak_core_vnode,vnode_command,3},
> {gen_fsm,handle_msg,7}]}
> in function gen_fsm:terminate/7
> ancestors: [riak_core_vnode_sup,riak_core_sup,<0.98.0>]
> messages: []
> links: [#Port<0.3521>,<0.100.0>]
> dictionary: []
> trap_exit: true
> status: running
> heap_size: 4181
> stack_size: 24
> reductions: 221476677
> neighbours:
>
> =SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===
> Supervisor: {local,riak_core_vnode_sup}
> Context: child_terminated
> Reason: {{badmatch,{error,emfile}},
> [{bitcask_fileops,create_file_loop,3},
> {bitcask,put,3},
> {riak_kv_bitcask_backend,put,3},
> {riak_kv_vnode,perform_put,3},
> {riak_kv_vnode,do_put,7},
> {riak_kv_vnode,handle_command,3},
> {riak_core_vnode,vnode_command,3},
> {gen_fsm,handle_msg,7}]}
> Offender: [{pid,<0.175.0>},
> {name,undefined},
> {mfa,
> {riak_core_vnode,start_link,
> [riak_kv_vnode,
> 570899077082383952423314387779798054553098649600]}},
> {restart_type,temporary},
> {shutdown,brutal_kill},
> {child_type,worker}]
>
>
> ==> erlang.log.4 <==
>
> =INFO REPORT==== 30-Jul-2010::14:23:27 ===
> Spidermonkey VM host stopping (<0.143.0>)
>
> =INFO REPORT==== 30-Jul-2010::14:23:27 ===
> Spidermonkey VM host stopping (<0.141.0>)
> {"Kernel pid
> terminated",application_controller,"{application_terminated,os_mon,shutdown}"}
>
> Crash dump was written to: erl_crash.dump
> Kernel pid terminated (application_controller)
> ({application_terminated,os_mon,shutdown})
>
>
>
>
>
> On Jul 30, 2010, at 4:10 PM, David Smith wrote:
>
>> Given the amount of data Ken has inserted, I'd say there's a 90%
>> chance this problem is related to a bug I fixed earlier this week:
>>
>> http://bitbucket.org/basho/bitcask/changeset/6a74d3aac4fb
>>
>> But without more information, it's hard to say. I presume, Ken, you
>> are also seeing a vnode crash error before this happens and I'm
>> betting it's emfile related.
>>
>> D.
>>
>> On Fri, Jul 30, 2010 at 3:06 PM, Alexander Sicular <siculars at gmail.com
>> <mailto:siculars at gmail.com>> wrote:
>>
>>     This may be another max file handler type of error. Or maybe even
>>     an oom thing if the key length is large.
>>
>>
>>     On Jul 30, 2010, at 4:59 PM, Grant Schofield wrote:
>>
>>     > I am not sure if you hit an already fixed bug in Bitcask or not.
>>     What version of Riak are you running on currently?
>>     >
>>     > Grant Schofield
>>     > Developer Advocate
>>     > Basho Technologies
>>     >
>>     > On Jul 30, 2010, at 1:28 PM, Ken Matsumoto wrote:
>>     >
>>     >> Hi all,
>>     >>
>>     >> I just tried to insert 1Billion data records.
>>     >> But I got the "write_lock" error after 12Million data.
>>     >> What is the reason and how should I avoid this?
>>     >> I use bitcask (default) backend and no parameters changed in
>>     config file.
>>     >> 1 record is just 70B text data.
>>     >>
>>     >> Regards,
>>     >>
>>     >> Ken.
>>     >>
>>     >> --
>>     >> Ken Matsumoto
>>     >> VP / Research & Development
>>     >> Nomura Research Institute America, Inc.
>>     >> NRI Pacific
>>     >> 1400 Fashion Island Blvd., Suite 1010
>>     >> San Mateo, CA 94404, U.S.A.
>>     >>
>>     >> PLEASE READ:This e-mail is confidential and intended for the
>>     named recipient only. If you are not an intended recipient, please
>>     notify the sender and delete this e-mail.
>>     >>
>>     >>
>>     >> _______________________________________________
>>     >> riak-users mailing list
>>     >> riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>>     >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>     >
>>     >
>>     > _______________________________________________
>>     > riak-users mailing list
>>     > riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>>     > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>>     _______________________________________________
>>     riak-users mailing list
>>     riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>>     http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users at lists.basho.com <mailto:riak-users at lists.basho.com>
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>


-- 
Ken Matsumoto
VP / Research & Development
Nomura Research Institute America, Inc.
NRI Pacific
1400 Fashion Island Blvd., Suite 1010
San Mateo, CA 94404, U.S.A.

PLEASE READ:This e-mail is confidential and intended for the named 
recipient only. If you are not an intended recipient, please notify the 
sender and delete this e-mail.





More information about the riak-users mailing list