<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hey David.<div><br></div><div>Does the below log output look like it could be caused by the issue you fixed?</div><div><br></div><div>Alex</div><div><br></div><div><br></div><div><div>==== Fri Jul 30 14:22:34 CDT 2010</div><div><br></div><div>=ERROR REPORT==== 30-Jul-2010::14:22:34 ===</div><div>** State machine <0.176.0> terminating </div><div>** Last event in was {riak_vnode_req_v1,</div><div>                     593735040165679310520246963290989976735222595584,</div><div>                     {fsm,undefined,<0.12466.0>},</div><div>                     {riak_kv_put_req_v1,</div><div>                      {<<"test.groups">>,<<"EghzXywWrGGtp2fCcSLoatIdjML">>},</div><div>                      {r_object,<<"test.groups">>,</div><div>                       <<"EghzXywWrGGtp2fCcSLoatIdjML">>,</div><div>                       [{r_content,</div><div>                         {dict,5,16,16,8,80,48,</div><div>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div>                          {{[],[],</div><div>                            [[<<"Links">>]],</div><div>                            [],[],[],[],[],[],[],</div><div>                            [[<<"content-type">>,97,112,112,108,105,99,97,</div><div>                              116,105,111,110,47,106,115,111,110],</div><div>                             [<<"X-Riak-VTag">>,89,69,78,55,55,111,66,121,73,</div><div>                              69,78,53,122,101,85,105,117,68,89,80,52]],</div><div>                            [],[],</div><div>                            [[<<"X-Riak-Last-Modified">>|</div><div>                              {1280,517754,951062}]],</div><div>                            [],</div><div>                            [[<<"X-Riak-Meta">>]]}}},</div><div>                         <<"{\"name\":\"foo\",\"created_at\":\"2010-07-30T19:22:34.947Z\",\"type\":\"group\",\"version\":1}">>}],</div><div>                       [{<<0,55,119,231>>,{1,63447736954}}],</div><div>                       {dict,1,16,16,8,80,48,</div><div>                        {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div>                        {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div>                          [[clean|true]],</div><div>                          []}}},</div><div>                       undefined},</div><div>                      33218311,63447736954,</div><div>                      [{returnbody,true}]}}</div><div>** When State == active</div><div>**      Data  == {state,593735040165679310520246963290989976735222595584,</div><div>                       riak_kv_vnode,</div><div>                       {state,593735040165679310520246963290989976735222595584, </div><div>                              riak_kv_bitcask_backend,</div><div>                              {#Ref<0.0.0.611>,</div><div>                               "data/bitcask/593735040165679310520246963290989976735222595584"},</div><div>                              [],false},</div><div>                       undefined,none}</div><div>** Reason for termination = </div><div>** {{badmatch,{error,emfile}},</div><div>   [{bitcask_fileops,create_file_loop,3},</div><div>    {bitcask,put,3},</div><div>    {riak_kv_bitcask_backend,put,3},</div><div>    {riak_kv_vnode,perform_put,3},</div><div>    {riak_kv_vnode,do_put,7},</div><div>    {riak_kv_vnode,handle_command,3},</div><div>    {riak_core_vnode,vnode_command,3},</div><div>    {gen_fsm,handle_msg,7}]}</div><div><br></div><div>=ERROR REPORT==== 30-Jul-2010::14:22:35 ===</div><div>Failed to open lock file data/bitcask/593735040165679310520246963290989976735222595584/bitcask.write.lock: emfile</div><div><br></div><div>=ERROR REPORT==== 30-Jul-2010::14:22:35 ===</div><div>** State machine <0.12471.0> terminating </div><div>** Last event in was {riak_vnode_req_v1,</div><div>                     593735040165679310520246963290989976735222595584,</div><div>                     {fsm,undefined,<0.12470.0>},</div><div>                     {riak_kv_put_req_v1,</div><div>                      {<<"test.users">>,<<"ZrAxzFghd51VG902GuCcJ2gYOMJ">>},</div><div>                      {r_object,<<"test.users">>,</div><div>                       <<"ZrAxzFghd51VG902GuCcJ2gYOMJ">>,</div><div>                       [{r_content,</div><div>                         {dict,5,16,16,8,80,48,</div><div>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div>                          {{[],[],</div><div>                            [[<<"Links">>,</div><div>                              {{<<"test.groups">>,</div><div>                                <<"EghzXywWrGGtp2fCcSLoatIdjML">>},</div><div>                               <<"groups">>}]],</div><div>                            [],[],[],[],[],[],[],</div><div>                            [[<<"content-type">>,97,112,112,108,105,99,97,</div><div>                              116,105,111,110,47,106,115,111,110],</div><div>                             [<<"X-Riak-VTag">>,50,88,97,105,85,49,70,49,55,</div><div>                              54,48,71,113,115,75,103,54,102,84,56,118,84]],</div><div>                            [],[],</div><div>                            [[<<"X-Riak-Last-Modified">>|{1280,517755,638}]],</div><div>                            [],</div><div>                            [[<<"X-Riak-Meta">>]]}}},</div><div>                         <<"{\"name\":\"bar\",\"created_at\":\"2010-07-30T19:22:34.998Z\",\"type\":\"user\",\"version\":1}">>}],</div><div>                       [{<<3,30,180,15>>,{1,63447736955}}],</div><div>                       {dict,1,16,16,8,80,48,</div><div>                        {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div>                        {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div>                          [[clean|true]],</div><div>                          []}}},</div><div>                       undefined},</div><div>                      122945524,63447736955,</div><div>                      [{returnbody,true}]}}</div><div>** When State == active</div><div>**      Data  == {state,593735040165679310520246963290989976735222595584,</div><div>                       riak_kv_vnode,</div><div>                       {state,593735040165679310520246963290989976735222595584, </div><div>                              riak_kv_bitcask_backend,</div><div>                              {#Ref<0.0.0.16357>,</div><div>                               "data/bitcask/593735040165679310520246963290989976735222595584"},</div><div>                              [],false},</div><div>                       undefined,none}</div><div>** Reason for termination = </div><div>** {bad_return_value,{error,{write_locked,emfile}}}</div><div><br></div><div>==> sasl-error.log <==</div><div><br></div><div>=CRASH REPORT==== 30-Jul-2010::14:22:35 ===</div><div> crasher:</div><div>   initial call: riak_core_vnode:init/1</div><div>   pid: <0.176.0></div><div>   registered_name: []</div><div>   exception exit: {{badmatch,{error,emfile}},</div><div>                    [{bitcask_fileops,create_file_loop,3},</div><div>                     {bitcask,put,3},</div><div>                     {riak_kv_bitcask_backend,put,3},</div><div>                     {riak_kv_vnode,perform_put,3},</div><div>                     {riak_kv_vnode,do_put,7},</div><div>                     {riak_kv_vnode,handle_command,3},</div><div>                     {riak_core_vnode,vnode_command,3},</div><div>                     {gen_fsm,handle_msg,7}]}</div><div>     in function  gen_fsm:terminate/7</div><div>   ancestors: [riak_core_vnode_sup,riak_core_sup,<0.98.0>]</div><div>   messages: []</div><div>   links: [#Port<0.3507>,<0.100.0>]</div><div>   dictionary: []</div><div>   trap_exit: true</div><div>   status: running</div><div>   heap_size: 1597</div><div>   stack_size: 24</div><div>   reductions: 11185</div><div> neighbours:</div><div><br></div><div>=SUPERVISOR REPORT==== 30-Jul-2010::14:22:35 ===</div><div>    Supervisor: {local,riak_core_vnode_sup}</div><div>    Context:    child_terminated</div><div>    Reason:     {{badmatch,{error,emfile}},</div><div>                 [{bitcask_fileops,create_file_loop,3},</div><div>                  {bitcask,put,3},</div><div>                  {riak_kv_bitcask_backend,put,3},</div><div>                  {riak_kv_vnode,perform_put,3},</div><div>                  {riak_kv_vnode,do_put,7},</div><div>                  {riak_kv_vnode,handle_command,3},</div><div>                  {riak_core_vnode,vnode_command,3},</div><div>                  {gen_fsm,handle_msg,7}]}</div><div>    Offender:   [{pid,<0.176.0>},</div><div>                 {name,undefined},</div><div>                 {mfa,</div><div>                     {riak_core_vnode,start_link,</div><div>                         [riak_kv_vnode,</div><div>                          593735040165679310520246963290989976735222595584]}},</div><div>                 {restart_type,temporary},</div><div>                 {shutdown,brutal_kill},</div><div>                 {child_type,worker}]</div><div><br></div><div><br></div><div>=CRASH REPORT==== 30-Jul-2010::14:22:35 ===</div><div> crasher:</div><div>   initial call: riak_core_vnode:init/1</div><div>   pid: <0.12471.0></div><div>   registered_name: []</div><div>   exception exit: {bad_return_value,{error,{write_locked,emfile}}}</div><div>     in function  gen_fsm:terminate/7</div><div>   ancestors: [riak_core_vnode_sup,riak_core_sup,<0.98.0>]</div><div>   messages: []</div><div>   links: [<0.100.0>]</div><div>   dictionary: []</div><div>   trap_exit: true</div><div>   status: running</div><div>   heap_size: 2584</div><div>   stack_size: 24</div><div>   reductions: 1955</div><div> neighbours:</div><div><br></div><div>=SUPERVISOR REPORT==== 30-Jul-2010::14:22:35 ===</div><div>    Supervisor: {local,riak_core_vnode_sup}</div><div>    Context:    child_terminated</div><div>    Reason:     {bad_return_value,{error,{write_locked,emfile}}}</div><div>    Offender:   [{pid,<0.12471.0>},</div><div>                 {name,undefined},</div><div>                 {mfa,</div><div>                     {riak_core_vnode,start_link,</div><div>                         [riak_kv_vnode,</div><div>                          593735040165679310520246963290989976735222595584]}},</div><div>                 {restart_type,temporary},</div><div>                 {shutdown,brutal_kill},</div><div>                 {child_type,worker}]</div><div><br></div><div><br></div><div>==> erlang.log.4 <==</div><div><br></div><div>=ERROR REPORT==== 30-Jul-2010::14:23:27 ===</div><div>** Generic server memsup terminating </div><div>** Last message in was {'EXIT',<0.21020.78>,</div><div>                          {emfile,</div><div>                              [{erlang,open_port,</div><div>                                   [{spawn,"/bin/sh -s unix:cmd 2>&1"},</div><div>                                    [stream]]},</div><div>                               {os,start_port_srv_loop,2}]}}</div><div>** When Server state == {state,{unix,darwin},</div><div>                              false,</div><div>                              {1897500000,7776508000},</div><div>                              {<0.81.0>,972008},</div><div>                              false,60000,30000,0.8,0.05,<0.21020.78>,</div><div>                              #Ref<0.0.2.58535>,undefined,</div><div>                              [reg],</div><div>                              []}</div><div>** Reason for termination == </div><div>** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},</div><div>           {os,start_port_srv_loop,2}]}</div><div><br></div><div>=ERROR REPORT==== 30-Jul-2010::14:23:27 ===</div><div>** Generic server memsup terminating </div><div>** Last message in was {'EXIT',<0.21186.78>,</div><div>                          {emfile,</div><div>                              [{erlang,open_port,</div><div>                                   [{spawn,"/bin/sh -s unix:cmd 2>&1"},</div><div>                                    [stream]]},</div><div>                               {os,start_port_srv_loop,2}]}}</div><div>** When Server state == {state,{unix,darwin},</div><div>                              false,undefined,undefined,false,60000,30000,</div><div>                              0.8,0.05,<0.21186.78>,#Ref<0.0.2.58551>,</div><div>                              undefined,</div><div>                              [reg],</div><div>                              []}</div><div>** Reason for termination == </div><div>** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},</div><div>           {os,start_port_srv_loop,2}]}</div><div><br></div><div>=ERROR REPORT==== 30-Jul-2010::14:23:27 ===</div><div>** Generic server memsup terminating </div><div>** Last message in was {'EXIT',<0.21252.78>,</div><div>                          {emfile,</div><div>                              [{erlang,open_port,</div><div>                                   [{spawn,"/bin/sh -s unix:cmd 2>&1"},</div><div>                                    [stream]]},</div><div>                               {os,start_port_srv_loop,2}]}}</div><div>** When Server state == {state,{unix,darwin},</div><div>                              false,undefined,undefined,false,60000,30000,</div><div>                              0.8,0.05,<0.21252.78>,#Ref<0.0.2.58559>,</div><div>                              undefined,</div><div>                              [reg],</div><div>                              []}</div><div>** Reason for termination == </div><div>** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},</div><div>           {os,start_port_srv_loop,2}]}</div><div><br></div><div>=ERROR REPORT==== 30-Jul-2010::14:23:27 ===</div><div>** Generic server memsup terminating </div><div>** Last message in was {'EXIT',<0.21374.78>,</div><div>                          {emfile,</div><div>                              [{erlang,open_port,</div><div>                                   [{spawn,"/bin/sh -s unix:cmd 2>&1"},</div><div>                                    [stream]]},</div><div>                               {os,start_port_srv_loop,2}]}}</div><div>** When Server state == {state,{unix,darwin},</div><div>                              false,undefined,undefined,false,60000,30000,</div><div>                              0.8,0.05,<0.21374.78>,#Ref<0.0.2.58569>,</div><div>                              undefined,</div><div>                              [reg],</div><div>                              []}</div><div>** Reason for termination == </div><div>** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},</div><div>           {os,start_port_srv_loop,2}]}</div><div><br></div><div>=ERROR REPORT==== 30-Jul-2010::14:23:27 ===</div><div>** Generic server memsup terminating </div><div>** Last message in was {'EXIT',<0.21396.78>,</div><div>                          {emfile,</div><div>                              [{erlang,open_port,</div><div>                                   [{spawn,"/bin/sh -s unix:cmd 2>&1"},</div><div>                                    [stream]]},</div><div>                               {os,start_port_srv_loop,2}]}}</div><div>** When Server state == {state,{unix,darwin},</div><div>                              false,undefined,undefined,false,60000,30000,</div><div>                              0.8,0.05,<0.21396.78>,#Ref<0.0.2.58575>,</div><div>                              undefined,</div><div>                              [reg],</div><div>                              []}</div><div>** Reason for termination == </div><div>** {emfile,[{erlang,open_port,[{spawn,"/bin/sh -s unix:cmd 2>&1"},[stream]]},</div><div>           {os,start_port_srv_loop,2}]}</div><div><br></div><div>=ERROR REPORT==== 30-Jul-2010::14:23:27 ===</div><div>** State machine <0.175.0> terminating </div><div>** Last event in was {riak_vnode_req_v1,</div><div>                     570899077082383952423314387779798054553098649600,</div><div>                     {fsm,undefined,<0.12470.0>},</div><div>                     {riak_kv_put_req_v1,</div><div>                      {<<"test.users">>,<<"ZrAxzFghd51VG902GuCcJ2gYOMJ">>},</div><div>                      {r_object,<<"test.users">>,</div><div>                       <<"ZrAxzFghd51VG902GuCcJ2gYOMJ">>,</div><div>                       [{r_content,</div><div>                         {dict,5,16,16,8,80,48,</div><div>                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div>                          {{[],[],</div><div>                            [[<<"Links">>,</div><div>                              {{<<"test.groups">>,</div><div>                                <<"EghzXywWrGGtp2fCcSLoatIdjML">>},</div><div>                               <<"groups">>}]],</div><div>                            [],[],[],[],[],[],[],</div><div>                            [[<<"content-type">>,97,112,112,108,105,99,97,</div><div>                              116,105,111,110,47,106,115,111,110],</div><div>                             [<<"X-Riak-VTag">>,50,88,97,105,85,49,70,49,55,</div><div>                              54,48,71,113,115,75,103,54,102,84,56,118,84]],</div><div>                            [],[],</div><div>                            [[<<"X-Riak-Last-Modified">>|{1280,517755,638}]],</div><div>                            [],</div><div>                            [[<<"X-Riak-Meta">>]]}}},</div><div>                         <<"{\"name\":\"bar\",\"created_at\":\"2010-07-30T19:22:34.998Z\",\"type\":\"user\",\"version\":1}">>}],</div><div>                       [{<<3,30,180,15>>,{1,63447736955}}],</div><div>                       {dict,1,16,16,8,80,48,</div><div>                        {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},</div><div>                        {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],</div><div>                          [[clean|true]],</div><div>                          []}}},</div><div>                       undefined},</div><div>                      122945524,63447736955,</div><div>                      [{returnbody,true}]}}</div><div>** When State == active</div><div>**      Data  == {state,570899077082383952423314387779798054553098649600,</div><div>                       riak_kv_vnode,</div><div>                       {state,570899077082383952423314387779798054553098649600, </div><div>                              riak_kv_bitcask_backend,</div><div>                              {#Ref<0.0.0.598>,</div><div>                               "data/bitcask/570899077082383952423314387779798054553098649600"},</div><div>                              [],false},</div><div>                       undefined,none}</div><div>** Reason for termination = </div><div>** {{badmatch,{error,emfile}},</div><div>   [{bitcask_fileops,create_file_loop,3},</div><div>    {bitcask,put,3},</div><div>    {riak_kv_bitcask_backend,put,3},</div><div>    {riak_kv_vnode,perform_put,3},</div><div>    {riak_kv_vnode,do_put,7},</div><div>    {riak_kv_vnode,handle_command,3},</div><div>    {riak_core_vnode,vnode_command,3},</div><div>    {gen_fsm,handle_msg,7}]}</div><div><br></div><div>=INFO REPORT==== 30-Jul-2010::14:23:27 ===</div><div>   application: os_mon</div><div>   exited: shutdown</div><div>   type: permanent</div><div><br></div><div>=INFO REPORT==== 30-Jul-2010::14:23:27 ===</div><div>Spidermonkey VM host stopping (<0.139.0>)</div><div><br></div><div>=INFO REPORT==== 30-Jul-2010::14:23:27 ===</div><div>Spidermonkey VM host stopping (<0.137.0>)</div><div><br></div><div>=INFO REPORT==== 30-Jul-2010::14:23:27 ===</div><div>Spidermonkey VM host stopping (<0.144.0>)</div><div><br></div><div>=INFO REPORT==== 30-Jul-2010::14:23:27 ===</div><div>Spidermonkey VM host stopping (<0.140.0>)</div><div><br></div><div>=INFO REPORT==== 30-Jul-2010::14:23:27 ===</div><div>Spidermonkey VM host stopping (<0.138.0>)</div><div><br></div><div>=INFO REPORT==== 30-Jul-2010::14:23:27 ===</div><div>Spidermonkey VM host stopping (<0.142.0>)</div><div><br></div><div>==> sasl-error.log <==</div><div><br></div><div>=CRASH REPORT==== 30-Jul-2010::14:23:27 ===</div><div> crasher:</div><div>   initial call: memsup:init/1</div><div>   pid: <0.52.0></div><div>   registered_name: memsup</div><div>   exception exit: {emfile,[{erlang,open_port,</div><div>                                    [{spawn,"/bin/sh -s unix:cmd 2>&1"},</div><div>                                     [stream]]},</div><div>                            {os,start_port_srv_loop,2}]}</div><div>     in function  gen_server:terminate/6</div><div>   ancestors: [os_mon_sup,<0.49.0>]</div><div>   messages: []</div><div>   links: [<0.50.0>]</div><div>   dictionary: []</div><div>   trap_exit: true</div><div>   status: running</div><div>   heap_size: 2584</div><div>   stack_size: 24</div><div>   reductions: 53031</div><div> neighbours:</div><div><br></div><div>=SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===</div><div>    Supervisor: {local,os_mon_sup}</div><div>    Context:    child_terminated</div><div>    Reason:     {emfile,[{erlang,open_port,</div><div>                                 [{spawn,"/bin/sh -s unix:cmd 2>&1"},</div><div>                                  [stream]]},</div><div>                         {os,start_port_srv_loop,2}]}</div><div>    Offender:   [{pid,<0.52.0>},</div><div>                 {name,memsup},</div><div>                 {mfa,{memsup,start_link,[]}},</div><div>                 {restart_type,permanent},</div><div>                 {shutdown,2000},</div><div>                 {child_type,worker}]</div><div><br></div><div><br></div><div>=CRASH REPORT==== 30-Jul-2010::14:23:27 ===</div><div> crasher:</div><div>   initial call: memsup:init/1</div><div>   pid: <0.21163.78></div><div>   registered_name: memsup</div><div>   exception exit: {emfile,[{erlang,open_port,</div><div>                                    [{spawn,"/bin/sh -s unix:cmd 2>&1"},</div><div>                                     [stream]]},</div><div>                            {os,start_port_srv_loop,2}]}</div><div>     in function  gen_server:terminate/6</div><div>   ancestors: [os_mon_sup,<0.49.0>]</div><div>   messages: []</div><div>   links: [<0.50.0>]</div><div>   dictionary: []</div><div>   trap_exit: true</div><div>   status: running</div><div>   heap_size: 377</div><div>   stack_size: 24</div><div>   reductions: 181</div><div> neighbours:</div><div><br></div><div>=SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===</div><div>    Supervisor: {local,os_mon_sup}</div><div>    Context:    child_terminated</div><div>    Reason:     {emfile,[{erlang,open_port,</div><div>                                 [{spawn,"/bin/sh -s unix:cmd 2>&1"},</div><div>                                  [stream]]},</div><div>                         {os,start_port_srv_loop,2}]}</div><div>    Offender:   [{pid,<0.21163.78>},</div><div>                 {name,memsup},</div><div>                 {mfa,{memsup,start_link,[]}},</div><div>                 {restart_type,permanent},</div><div>                 {shutdown,2000},</div><div>                 {child_type,worker}]</div><div><br></div><div><br></div><div>=CRASH REPORT==== 30-Jul-2010::14:23:27 ===</div><div> crasher:</div><div>   initial call: memsup:init/1</div><div>   pid: <0.21226.78></div><div>   registered_name: memsup</div><div>   exception exit: {emfile,[{erlang,open_port,</div><div>                                    [{spawn,"/bin/sh -s unix:cmd 2>&1"},</div><div>                                     [stream]]},</div><div>                            {os,start_port_srv_loop,2}]}</div><div>     in function  gen_server:terminate/6</div><div>   ancestors: [os_mon_sup,<0.49.0>]</div><div>   messages: []</div><div>   links: [<0.50.0>]</div><div>   dictionary: []</div><div>   trap_exit: true</div><div>   status: running</div><div>   heap_size: 377</div><div>   stack_size: 24</div><div>   reductions: 181</div><div> neighbours:</div><div><br></div><div>=SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===</div><div>    Supervisor: {local,os_mon_sup}</div><div>    Context:    child_terminated</div><div>    Reason:     {emfile,[{erlang,open_port,</div><div>                                 [{spawn,"/bin/sh -s unix:cmd 2>&1"},</div><div>                                  [stream]]},</div><div>                         {os,start_port_srv_loop,2}]}</div><div>    Offender:   [{pid,<0.21226.78>},</div><div>                 {name,memsup},</div><div>                 {mfa,{memsup,start_link,[]}},</div><div>                 {restart_type,permanent},</div><div>                 {shutdown,2000},</div><div>                 {child_type,worker}]</div><div><br></div><div><br></div><div>=CRASH REPORT==== 30-Jul-2010::14:23:27 ===</div><div> crasher:</div><div>   initial call: memsup:init/1</div><div>   pid: <0.21343.78></div><div>   registered_name: memsup</div><div>   exception exit: {emfile,[{erlang,open_port,</div><div>                                    [{spawn,"/bin/sh -s unix:cmd 2>&1"},</div><div>                                     [stream]]},</div><div>                            {os,start_port_srv_loop,2}]}</div><div>     in function  gen_server:terminate/6</div><div>   ancestors: [os_mon_sup,<0.49.0>]</div><div>   messages: []</div><div>   links: [<0.50.0>]</div><div>   dictionary: []</div><div>   trap_exit: true</div><div>   status: running</div><div>   heap_size: 377</div><div>   stack_size: 24</div><div>   reductions: 193</div><div> neighbours:</div><div><br></div><div>=SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===</div><div>    Supervisor: {local,os_mon_sup}</div><div>    Context:    child_terminated</div><div>    Reason:     {emfile,[{erlang,open_port,</div><div>                                 [{spawn,"/bin/sh -s unix:cmd 2>&1"},</div><div>                                  [stream]]},</div><div>                         {os,start_port_srv_loop,2}]}</div><div>    Offender:   [{pid,<0.21343.78>},</div><div>                 {name,memsup},</div><div>                 {mfa,{memsup,start_link,[]}},</div><div>                 {restart_type,permanent},</div><div>                 {shutdown,2000},</div><div>                 {child_type,worker}]</div><div><br></div><div><br></div><div>=CRASH REPORT==== 30-Jul-2010::14:23:27 ===</div><div> crasher:</div><div>   initial call: memsup:init/1</div><div>   pid: <0.21393.78></div><div>   registered_name: memsup</div><div>   exception exit: {emfile,[{erlang,open_port,</div><div>                                    [{spawn,"/bin/sh -s unix:cmd 2>&1"},</div><div>                                     [stream]]},</div><div>                            {os,start_port_srv_loop,2}]}</div><div>     in function  gen_server:terminate/6</div><div>   ancestors: [os_mon_sup,<0.49.0>]</div><div>   messages: []</div><div>   links: [<0.50.0>]</div><div>   dictionary: []</div><div>   trap_exit: true</div><div>   status: running</div><div>   heap_size: 377</div><div>   stack_size: 24</div><div>   reductions: 189</div><div> neighbours:</div><div><br></div><div>=SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===</div><div>    Supervisor: {local,os_mon_sup}</div><div>    Context:    child_terminated</div><div>    Reason:     {emfile,[{erlang,open_port,</div><div>                                 [{spawn,"/bin/sh -s unix:cmd 2>&1"},</div><div>                                  [stream]]},</div><div>                         {os,start_port_srv_loop,2}]}</div><div>    Offender:   [{pid,<0.21393.78>},</div><div>                 {name,memsup},</div><div>                 {mfa,{memsup,start_link,[]}},</div><div>                 {restart_type,permanent},</div><div>                 {shutdown,2000},</div><div>                 {child_type,worker}]</div><div><br></div><div><br></div><div>=SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===</div><div>    Supervisor: {local,os_mon_sup}</div><div>    Context:    shutdown</div><div>    Reason:     reached_max_restart_intensity</div><div>    Offender:   [{pid,<0.21393.78>},</div><div>                 {name,memsup},</div><div>                 {mfa,{memsup,start_link,[]}},</div><div>                 {restart_type,permanent},</div><div>                 {shutdown,2000},</div><div>                 {child_type,worker}]</div><div><br></div><div><br></div><div>=CRASH REPORT==== 30-Jul-2010::14:23:27 ===</div><div> crasher:</div><div>   initial call: riak_core_vnode:init/1</div><div>   pid: <0.175.0></div><div>   registered_name: []</div><div>   exception exit: {{badmatch,{error,emfile}},</div><div>                    [{bitcask_fileops,create_file_loop,3},</div><div>                     {bitcask,put,3},</div><div>                     {riak_kv_bitcask_backend,put,3},</div><div>                     {riak_kv_vnode,perform_put,3},</div><div>                     {riak_kv_vnode,do_put,7},</div><div>                     {riak_kv_vnode,handle_command,3},</div><div>                     {riak_core_vnode,vnode_command,3},</div><div>                     {gen_fsm,handle_msg,7}]}</div><div>     in function  gen_fsm:terminate/7</div><div>   ancestors: [riak_core_vnode_sup,riak_core_sup,<0.98.0>]</div><div>   messages: []</div><div>   links: [#Port<0.3521>,<0.100.0>]</div><div>   dictionary: []</div><div>   trap_exit: true</div><div>   status: running</div><div>   heap_size: 4181</div><div>   stack_size: 24</div><div>   reductions: 221476677</div><div> neighbours:</div><div><br></div><div>=SUPERVISOR REPORT==== 30-Jul-2010::14:23:27 ===</div><div>    Supervisor: {local,riak_core_vnode_sup}</div><div>    Context:    child_terminated</div><div>    Reason:     {{badmatch,{error,emfile}},</div><div>                 [{bitcask_fileops,create_file_loop,3},</div><div>                  {bitcask,put,3},</div><div>                  {riak_kv_bitcask_backend,put,3},</div><div>                  {riak_kv_vnode,perform_put,3},</div><div>                  {riak_kv_vnode,do_put,7},</div><div>                  {riak_kv_vnode,handle_command,3},</div><div>                  {riak_core_vnode,vnode_command,3},</div><div>                  {gen_fsm,handle_msg,7}]}</div><div>    Offender:   [{pid,<0.175.0>},</div><div>                 {name,undefined},</div><div>                 {mfa,</div><div>                     {riak_core_vnode,start_link,</div><div>                         [riak_kv_vnode,</div><div>                          570899077082383952423314387779798054553098649600]}},</div><div>                 {restart_type,temporary},</div><div>                 {shutdown,brutal_kill},</div><div>                 {child_type,worker}]</div><div><br></div><div><br></div><div>==> erlang.log.4 <==</div><div><br></div><div>=INFO REPORT==== 30-Jul-2010::14:23:27 ===</div><div>Spidermonkey VM host stopping (<0.143.0>)</div><div><br></div><div>=INFO REPORT==== 30-Jul-2010::14:23:27 ===</div><div>Spidermonkey VM host stopping (<0.141.0>)</div><div>{"Kernel pid terminated",application_controller,"{application_terminated,os_mon,shutdown}"}</div><div><br></div><div>Crash dump was written to: erl_crash.dump</div><div>Kernel pid terminated (application_controller) ({application_terminated,os_mon,shutdown})</div></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br><div><div>On Jul 30, 2010, at 4:10 PM, David Smith wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Given the amount of data Ken has inserted, I'd say there's a 90% chance this problem is related to a bug I fixed earlier this week:<div><br></div><div><a href="http://bitbucket.org/basho/bitcask/changeset/6a74d3aac4fb">http://bitbucket.org/basho/bitcask/changeset/6a74d3aac4fb</a></div>
<div><br></div><div>But without more information, it's hard to say. I presume, Ken, you are also seeing a vnode crash error before this happens and I'm betting it's emfile related.</div><div><br></div><div>D.<br>
<br><div class="gmail_quote">On Fri, Jul 30, 2010 at 3:06 PM, Alexander Sicular <span dir="ltr"><<a href="mailto:siculars@gmail.com">siculars@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
This may be another max file handler type of error. Or maybe even an oom thing if the key length is large.<br>
<div><div></div><div class="h5"><br>
<br>
On Jul 30, 2010, at 4:59 PM, Grant Schofield wrote:<br>
<br>
> I am not sure if you hit an already fixed bug in Bitcask or not. What version of Riak are you running on currently?<br>
><br>
> Grant Schofield<br>
> Developer Advocate<br>
> Basho Technologies<br>
><br>
> On Jul 30, 2010, at 1:28 PM, Ken Matsumoto wrote:<br>
><br>
>> Hi all,<br>
>><br>
>> I just tried to insert 1Billion data records.<br>
>> But I got the "write_lock" error after 12Million data.<br>
>> What is the reason and how should I avoid this?<br>
>> I use bitcask (default) backend and no parameters changed in config file.<br>
>> 1 record is just 70B text data.<br>
>><br>
>> Regards,<br>
>><br>
>> Ken.<br>
>><br>
>> --<br>
>> Ken Matsumoto<br>
>> VP / Research & Development<br>
>> Nomura Research Institute America, Inc.<br>
>> NRI Pacific<br>
>> 1400 Fashion Island Blvd., Suite 1010<br>
>> San Mateo, CA 94404, U.S.A.<br>
>><br>
>> PLEASE READ´╝ÜThis e-mail is confidential and intended for the named recipient only. If you are not an intended recipient, please notify the sender and delete this e-mail.<br>
>><br>
>><br>
>> _______________________________________________<br>
>> riak-users mailing list<br>
>> <a href="mailto:riak-users@lists.basho.com">riak-users@lists.basho.com</a><br>
>> <a href="http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com" target="_blank">http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com</a><br>
><br>
><br>
> _______________________________________________<br>
> riak-users mailing list<br>
> <a href="mailto:riak-users@lists.basho.com">riak-users@lists.basho.com</a><br>
> <a href="http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com" target="_blank">http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com</a><br>
<br>
<br>
_______________________________________________<br>
riak-users mailing list<br>
<a href="mailto:riak-users@lists.basho.com">riak-users@lists.basho.com</a><br>
<a href="http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com" target="_blank">http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com</a><br>
</div></div></blockquote></div><br></div>
_______________________________________________<br>riak-users mailing list<br><a href="mailto:riak-users@lists.basho.com">riak-users@lists.basho.com</a><br>http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com<br></blockquote></div><br></div></body></html>