I think the key may lie here.... "<span style="font-family:Arial,'Liberation Sans','DejaVu Sans',sans-serif;font-size:14px;line-height:18px">,{checkout,</span><span style="font-family:Arial,'Liberation Sans','DejaVu Sans',sans-serif;font-size:14px;line-height:18px">false,5000}"</span><div>
<font face="Arial, Liberation Sans, DejaVu Sans, sans-serif"><span style="font-size:14px;line-height:18px"><br></span></font></div><div><font face="Arial, Liberation Sans, DejaVu Sans, sans-serif"><span style="font-size:14px;line-height:18px">Are you releasing your connections back to the pool?   Is your throughput greater than the system can handle due to limited connection pool sizes?  What is your ulimit set to (ulimit -n)  ... maybe you're running out of FD's?</span></font></div>
<div><font face="Arial, Liberation Sans, DejaVu Sans, sans-serif"><span style="font-size:14px;line-height:18px"><br></span></font></div><div><font face="Arial, Liberation Sans, DejaVu Sans, sans-serif"><span style="font-size:14px;line-height:18px">-mox<br>
</span></font><br><div class="gmail_quote">On Wed, Oct 10, 2012 at 10:31 PM, Mikhail Kuznetsov <span dir="ltr"><<a href="mailto:kuznetsov.m.yu@gmail.com" target="_blank">kuznetsov.m.yu@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><p style="text-indent:0px;letter-spacing:normal;text-align:left;font-variant:normal;font-style:normal;font-weight:normal;padding:0px;vertical-align:baseline;clear:both;line-height:18px;text-transform:none;font-size:14px;white-space:normal;margin:0px 0px 1em;font-family:Arial,'Liberation Sans','DejaVu Sans',sans-serif;word-wrap:break-word;border:0px;background-repeat:initial initial;word-spacing:0px">
I got nasty problem in production. We make connection pool with official erlang pb client. Everything works fine. To organize pool we use hottub (we try several,but that is simplest). Each connection used at least once in 3-5 minutes(production is not full loaded now).</p>
<p style="text-indent:0px;letter-spacing:normal;text-align:left;font-variant:normal;font-style:normal;font-weight:normal;padding:0px;vertical-align:baseline;clear:both;line-height:18px;text-transform:none;font-size:14px;white-space:normal;margin:0px 0px 1em;font-family:Arial,'Liberation Sans','DejaVu Sans',sans-serif;word-wrap:break-word;border:0px;background-repeat:initial initial;word-spacing:0px">
After several days riak server disconnect us. But socket process doesn't die, on any request it answers {error, disconnected}. So far I wrote pool workers checker, if it is_connected(Pid) return not true, we kill worker and pool create new one. I fired it every ten minutes. But it didn't help. It return true, but then I am making request I get {error, disconnected}. Only solution that work so far is pool full reinit if some worker return {error, disconnected}. It is very barbaric and may crash whole app.</p>
<p style="text-indent:0px;letter-spacing:normal;text-align:left;font-variant:normal;font-style:normal;font-weight:normal;padding:0px;vertical-align:baseline;clear:both;line-height:18px;text-transform:none;font-size:14px;white-space:normal;margin:0px 0px 1em;font-family:Arial,'Liberation Sans','DejaVu Sans',sans-serif;word-wrap:break-word;border:0px;background-repeat:initial initial;word-spacing:0px">
When I checked server logs I found many errors like this two: 2012-09-20 00:10:10.976 [error] <0.803.0>@riak_core_vnode:handle_info:510 296867520082839655260123481645494988367611297792 riak_kv_vnode worker pool crashed {timeout,{gen_server,call,[<0.819.0>,{work,<0.806.0>,{fold,#Fun,#Fun},{raw,59205031,<0.28969.11>}}]}} 2012-09-20 00:10:10.976 [error] <0.862.0>@riak_core_vnode:handle_info:510 365375409332725729550921208179070754913983135744 riak_kv_vnode worker pool crashed {timeout,{gen_fsm,sync_send_event,[<0.866.0>,{checkout,false,5000},5000]}}</p>
<p style="text-indent:0px;letter-spacing:normal;text-align:left;font-variant:normal;font-style:normal;font-weight:normal;padding:0px;vertical-align:baseline;clear:both;line-height:18px;text-transform:none;font-size:14px;white-space:normal;margin:0px 0px 1em;font-family:Arial,'Liberation Sans','DejaVu Sans',sans-serif;word-wrap:break-word;border:0px;background-repeat:initial initial;word-spacing:0px">
I guess that is real problem, but I think client connection should at least log something, get connection problem failures list or die. I got is_connected(Pid) = true</p><p style="border-bottom-width:0px;text-indent:0px;text-align:left;border-style:initial;font-weight:normal;clear:both;line-height:18px;margin-right:0px;font-size:14px;white-space:normal;border-right-width:0px;word-wrap:break-word;border-left-width:0px;word-spacing:0px;padding-left:0px;padding-right:0px;padding-top:0px;letter-spacing:normal;font-variant:normal;font-style:normal;margin-bottom:1em;border-color:initial;padding-bottom:0px;border-top-width:0px;vertical-align:baseline;text-transform:none;margin-left:0px;font-family:Arial,'Liberation Sans','DejaVu Sans',sans-serif;margin-top:0px">
How are you organizing connection pools which work 24/7? How you check pool workers or refresh them?</p><div>
<div style="word-wrap:break-word"><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<div><div><br></div></div></div></span></div></span></div>
</div>
<br></div><br>_______________________________________________<br>
riak-users mailing list<br>
<a href="mailto:riak-users@lists.basho.com">riak-users@lists.basho.com</a><br>
<a href="http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com" target="_blank">http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com</a><br>
<br></blockquote></div><br></div>