Timeout when storing

Jim Adler jim.adler at comcast.net
Sun Oct 9 18:54:53 EDT 2011


Thanks David - I'll try that on my single-node instance, but I'm working another Riak issue on another thread. 


Jim 

----- Original Message -----
From: "David Smith" <dizzyd at basho.com> 
To: "jim adler" <jim.adler at comcast.net> 
Sent: Friday, October 7, 2011 7:02:01 AM 
Subject: Re: Timeout when storing 

Hi Jim, 

Sorry for the slow response -- email is like a running battle at times. :) 

How many partitions are you running? 

Also, take down the node and then remove any *.lock files. 

Thanks, 

D. 

On Mon, Oct 3, 2011 at 11:23 AM, <jim.adler at comcast.net> wrote: 
> About 90 out of 3000 are zero-bytes. 
> 
> Jim 
> 
> -----Original Message----- 
> From: riak-users-bounces at lists.basho.com 
> [mailto:riak-users-bounces at lists.basho.com] On Behalf Of David Smith 
> Sent: Monday, October 03, 2011 4:46 AM 
> To: Jim Adler 
> Cc: riak-users at lists.basho.com 
> Subject: Re: Timeout when storing 
> 
> Jim, 
> 
> If you look at your bitcask directories, do you have a large number of 
> zero-byte files, perchance? 
> 
> D. 
> 
> On Sat, Oct 1, 2011 at 1:58 PM, Jim Adler <jadler at intelius.com> wrote: 
>> After upgrading my single-node instance to 1.0, I'm still seeing the 
>> "timeout when storing" issue. Here are the changes I made based on 
>> everyone's suggestions (much appreciated!): 
>> 
>> - Ubuntu 11.04 (natty) 32-bit 
>> - Python client 1.3.0 
>> - /etc/riak/vm.args: -env ERL_MAX_PORTS 32768 
>> - /etc/default/riak: ulimit -n 32768 
>> 
>> Here's the /var/log/crash.log report: 
>> 
>> 2011-10-01 12:31:03 =ERROR REPORT==== 
>> ** State machine <0.3452.0> terminating 
>> ** Last event in was 
>> {'riak_vnode_req_v1',1136089163393944065322395631681798128560666312704 
>> ,{fsm,undefined,<0.3451.0>},{'riak_kv_put_req_v1',{<<"nodes">>,<<"user 
>> _id-17527747-info">>},{r_object,<<"nodes">>,<<"user_id-17527747-info"> 
>> >,[{r_content,{dict,4,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[], 
>> [],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[[<<"content-type">>,9 
>> 7,112,112,108,105,99,97,116,105,111,110,47,106,115,111,110],[<<"X-Riak 
>> -VTag">>,49,88,88,75,75,51,90,88,68,117,90,122,85,53,57,85,53,101,107, 
>> 89,115,110]],[[<<"index">>]],[],[[<<"X-Riak-Last-Modified">>|{1317,497 
>> 463,847242}]],[],[]}}},<<"{DATA 
>> DELETED}">>}],[],{dict,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[], 
>> [],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[[clean 
>> |true]],[]}}},undefined},51456853,63484716663,[coord]}} 
>> 
>> ** When State == active 
>> ** Data == 
>> {state,1136089163393944065322395631681798128560666312704,riak_kv_vnode 
>> ,{state,1136089163393944065322395631681798128560666312704,false,riak_k 
>> v_bitcask_backend,{state,#Ref<0.0.0.10359>,"11360891633939440653223956 
>> 31681798128560666312704",[{async_folds,true},[{vnode_vclocks,true},{in 
>> cluded_applications,[]},{add_paths,[]},{allow_strfun,false},{storage_b 
>> ackend,riak_kv_bitcask_backend},{legacy_keylisting,false},{reduce_js_v 
>> m_count,6},{js_thread_stack,16},{pb_ip,"0.0.0.0"},{riak_kv_stat,true}, 
>> {map_js_vm_count,8},{mapred_system,pipe},{js_max_vm_mem,8},{pb_port,80 
>> 87},{legacy_stats,true},{mapred_name,"mapred"},{stats_urlpath,"stats"} 
>> ,{http_url_encoding,on},{hook_js_vm_count,2}],{read_write,true}],11360 
>> 89163393944065322395631681798128560666312704,"/var/lib/riak/bitcask"}, 
>> {dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[] 
>> },{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<<35,9,254,249, 
>> 78,135,82,106>>,3000,1000,100,100,true,false},undefined,undefined,none 
>> ,undefined,<0.3454.0>,60000} 
>> ** Reason for termination = 
>> ** {bad_return_value,{error,{write_locked,emfile}}} 
>> 2011-10-01 12:31:03 =CRASH REPORT==== 
>> crasher: 
>> initial call: riak_core_vnode:init/1 
>> pid: <0.3452.0> 
>> registered_name: [] 
>> exception exit: {bad_return_value,{error,{write_locked,emfile}}} 
>> in function gen_fsm:terminate/7 
>> in call from proc_lib:init_p_do_apply/3 
>> ancestors: [riak_core_vnode_sup,riak_core_sup,<0.92.0>] 
>> messages: [{'EXIT',<0.3454.0>,shutdown}] 
>> links: [<0.96.0>] 
>> dictionary: [] 
>> trap_exit: true 
>> status: running 
>> heap_size: 6765 
>> stack_size: 24 
>> reductions: 160650 
>> neighbours: 
>> 2011-10-01 12:31:03 =SUPERVISOR REPORT==== 
>> Supervisor: {local,riak_core_vnode_sup} 
>> Context: child_terminated 
>> Reason: {bad_return_value,{error,{write_locked,emfile}}} 
>> Offender: 
>> [{pid,<0.3452.0>},{name,undefined},{mfargs,{riak_core_vnode,start_link 
>> ,undefined}},{restart_type,temporary},{shutdown,300000},{child_type,wo 
>> rker}] 
>> 
>> 2011-10-01 12:45:28 =ERROR REPORT==== 
>> Failed to merge 
>> 
> "/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/var/ 
> lib/riak/bitcask/605153021707326989568713251046585937826284568576/1315770213 
> .bitcask.data/var/lib/riak/bitcask/60515302170732698956871325104658593782628 
> 4568576/1316329673.bitcask.data/var/lib/riak/bitcask/60515302170732698956871 
> 3251046585937826284568576/1316330222.bitcask.data/var/lib/riak/bitcask/60515 
> 3021707326989568713251046585937826284568576/1316879145.bitcask.data/var/lib/ 
> riak/bitcask/605153021707326989568713251046585937826284568576/1316995340.bit 
> cask.data/var/lib/riak/bitcask/605153021707326989568713251046585937826284568 
> 576/1317493005.bitcask.data/var/lib/riak/bitcask/605153021707326989568713251 
> 046585937826284568576/1317495168.bitcask.data": 
>> {{badmatch,{error,emfile}},[{bitcask,'-merge1/3-lc$^0/1-1-',1},{bitcas 
>> k,'-merge1/3-lc$^0/1-1-',1},{bitcask,'merge1',3},{bitcask_merge_worker 
>> ,do_merge,1}]} 
>> 
>> 
>> -----Original Message----- 
>> From: David Smith [mailto:dizzyd at basho.com] 
>> Sent: Fri 9/30/2011 9:56 AM 
>> To: Jim Adler 
>> Cc: Sean Cribbs; riak-users at lists.basho.com 
>> Subject: Re: Timeout when storing 
>> 
>> IIRC, {error, emfile} indicates that the max # of ports (in the erlang 
>> VM) is being exceeded. Try bumping up ERL_MAX_PORTS in vm.args. 
>> 
>> D. 
>> 
>> On Thu, Sep 29, 2011 at 10:52 PM, Jim Adler <jadler at intelius.com> wrote: 
>>> Thanks Sean. I added the ulimit -n 10240 to /etc/default/riak, 
>>> restarted riak, but that didn't work. 
>>> 
>>> Fyodor Yarochkin suggested that the bitcask files could be corrupted, 
>>> but I wasn't sure which bitcask *.data or *.hint file to delete. Any 
>>> pointers? 
>>> 
>>> Here's the /var/log/riak/erlang.log: 
>>> 
>>> =ERROR REPORT==== 29-Sep-2011::20:27:42 === 
>>> ** State machine <0.369.0> terminating 
>>> ** Last event in was {riak_vnode_req_v1, 
>>> 
>>> 941983477185933521498468739836666790012612771840, 
>>> {fsm,undefined,<0.27704.1>}, 
>>> {riak_kv_put_req_v1, 
>>> {<<"nodes">>,<<"screen_name-psych_ic-info">>}, 
>>> 
>>> {r_object,<<"nodes">>,<<"screen_name-psych_ic-info">>, 
>>> [{r_content, 
>>> {dict,3,16,16,8,80,48, 
>>> 
>>> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, 
>>> {{[],[],[],[],[],[],[],[],[],[], 
>>> 
>>> [[<<"content-type">>,97,112,112,108,105,99,97, 
>>> 116,105,111,110,47,106,115,111,110], 
>>> 
>>> [<<"X-Riak-VTag">>,49,90,120,65,84,100,56,99,48, 
>>> 
>>> 80,86,99,111,122,71,79,108,90,70,97,53,87]], 
>>> [],[], 
>>> [[<<"X-Riak-Last-Modified">>| 
>>> {1317,353201,695471}]], 
>>> [],[]}}}, 
>>> <<"{DELETED DATA}">>}], 
>>> [{<<2,65,205,48>>,{1,63484572401}}], 
>>> {dict,1,16,16,8,80,48, 
>>> 
>>> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, 
>>> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[], 
>>> [[clean|true]], 
>>> []}}}, 
>>> undefined}, 
>>> 1174401,63484572401, 
>>> [{returnbody,true}]}} 
>>> ** When State == active 
>>> ** Data == 
>>> {state,941983477185933521498468739836666790012612771840, 
>>> riak_kv_vnode, 
>>> 
>>> {state,941983477185933521498468739836666790012612771840, 
>>> riak_kv_bitcask_backend, 
>>> {#Ref<0.0.0.3952>, 
>>> 
>>> "/var/lib/riak/bitcask/9419834771859335214984687398366667900126127718 
>>> 40"}, 
>>> {dict,0,16,16,8,80,48, 
>>> 
>>> {[],[],[],[],[],[],[],[],[],[],[],[],[], 
>>> [],[],[]}, 
>>> 
>>> {{[],[],[],[],[],[],[],[],[],[],[],[],[], 
>>> [],[],[]}}}, 
>>> false}, 
>>> undefined,none,60000} 
>>> ** Reason for termination = 
>>> ** {{badmatch,{error,emfile}}, 
>>> [{bitcask_fileops,create_file_loop,3}, 
>>> {bitcask,put,3}, 
>>> {riak_kv_bitcask_backend,put,3}, 
>>> {riak_kv_vnode,perform_put,3}, 
>>> {riak_kv_vnode,do_put,7}, 
>>> {riak_kv_vnode,handle_command,3}, 
>>> {riak_core_vnode,vnode_command,3}, 
>>> {gen_fsm,handle_msg,7}]} 
>>> 
>>> 
>>> 
>>> -----Original Message----- 
>>> From: Sean Cribbs [mailto:sean at basho.com] 
>>> Sent: Thu 9/29/2011 3:02 PM 
>>> To: Jim Adler 
>>> Cc: riak-users at lists.basho.com 
>>> Subject: Re: Timeout when storing 
>>> 
>>> Your environment has too few file handles. Retry starting riak after 
>>> setting `ulimit -n 1024` in the shell. Also see our wiki page about 
>>> this 
>>> issue: http://wiki.basho.com/Open-Files-Limit.html You may need to 
>>> set this limit specifically for the 'riak' user. 
>>> 
>>> Cheers, 
>>> 
>>> -- 
>>> Sean Cribbs <sean at basho.com> 
>>> Developer Advocate 
>>> Basho Technologies, Inc. 
>>> http://www.basho.com/ 
>>> 
>>> 
>>> _______________________________________________ 
>>> riak-users mailing list 
>>> riak-users at lists.basho.com 
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
>>> 
>>> 
>> 
>> 
>> 
>> -- 
>> Dave Smith 
>> Director, Engineering 
>> Basho Technologies, Inc. 
>> dizzyd at basho.com 
>> 
>> 
> 
> 
> 
> -- 
> Dave Smith 
> Director, Engineering 
> Basho Technologies, Inc. 
> dizzyd at basho.com 
> 
> _______________________________________________ 
> riak-users mailing list 
> riak-users at lists.basho.com 
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
> 
> 



-- 
Dave Smith 
Director, Engineering 
Basho Technologies, Inc. 
dizzyd at basho.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.basho.com/pipermail/riak-users_lists.basho.com/attachments/20111009/75ca5ec5/attachment.html>


More information about the riak-users mailing list