Complete Riak failure.

Dmitry Demeshchuk demeshchuk at gmail.com
Mon Aug 9 10:07:00 EDT 2010


The error you get means that too many sockets are opened
simultaneously. I guess there are two main possible reasons:

1. You are being spammed by someone
2. Your REST requests to Riak are somehow handled slower than new
requests come. So, the number of opened HTTP connections increase over
time and you start getting this message.

Some time ago I got exactly the same problem with YAWS (nothing
related to Riak though) and I decided to migrate to mochiweb and
optimized some of our server-side code to reduce the response delay.
And that helped.

On Mon, Aug 9, 2010 at 5:55 PM, Richard Heycock <rgh at roughage.com.au> wrote:
> I've been running riak for about a month and I'm using it as a
> persistent cache. Everything has been fine until tonight that is where
> it appears to have had a catastrophic failure. Basically when I make a
> request using the http interface I get a failure. I cannot be more
> specific than that as there is no response code back; I think it's
> getting a TCP reset ...
>
> ... That was about 10 minutes ago. When I tried to find out if I was
> getting a TCP reset I restarted th program and everything was fine.
>
> Anyway most of the errors look like this:
>
>
> =ERROR REPORT==== 9-Aug-2010::13:23:27 ===
> {mochiweb_socket_server,256,{acceptor_error,{error,accept_failed}}}
>
> =ERROR REPORT==== 9-Aug-2010::13:23:27 ===
>    application: mochiweb
>    "Accept failed error"
>    "{error,emfile}"
>
> And every so often I get:
>
> =ERROR REPORT==== 9-Aug-2010::13:23:56 ===
> ** State machine <0.17333.0> terminating
> ** Last event in was {riak_vnode_req_v1,
>                      182687704666362864775460604089535377456991567872,
>                      {fsm,undefined,<0.17335.0>},
>                      {riak_kv_put_req_v1,
>                       {<<"uris">>,
>                        <<"5ea5dc023dd73b5f711efde3d50b9e91da4c5027">>},
>                       {r_object,<<"uris">>,
>                        <<"5ea5dc023dd73b5f711efde3d50b9e91da4c5027">>,
>                        [{r_content,
>                          {dict,5,16,16,8,80,48,
>                           {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                           {{[],[],
>                             [[<<"Links">>]],
>                             [],[],[],[],[],[],[],
>                             [[<<"content-type">>,97,112,112,108,105,99,97,
>                               116,105,111,110,47,106,115,111,110],
>                              [<<"X-Riak-VTag">>,49,102,84,99,107,121,67,100,
>                               81,67,83,85,102,67,102,118,68,55,80,50,82,118]],
>                             [],[],
>                             [[<<"X-Riak-Last-Modified">>|
>                               {1281,360236,222890}]],
>                             [],
>                             [[<<"X-Riak-Meta">>]]}}},
>                          <<"{\"uri\":\"http://feedproxy.google.com/~r/time/politics/~3/SGIQlFYkIy8/0,8599,2006898,00.html\",\"download_date\":\"20100809132314\"}">>}],
>                        [{<<2,235,89,160>>,{1,63448579436}}],
>                        {dict,1,16,16,8,80,48,
>                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
>                           [[clean|true]],
>                           []}}},
>                        undefined},
>                       118194806,63448579436,
>                       [{returnbody,true}]}}
> ** When State == active
> **      Data  == {state,182687704666362864775460604089535377456991567872,
>                        riak_kv_vnode,
>                        {state,182687704666362864775460604089535377456991567872,
>                               riak_kv_bitcask_backend,
>                               {#Ref<0.0.0.23192>,
>                                "/var/lib/riak/bitcask/182687704666362864775460604089535377456991567872"},
>                               [],false},
>                        undefined,none}
> ** Reason for termination =
> ** {bad_return_value,{error,{write_locked,emfile}}}
>
>
> I've put a full log here:
>
>    http://stuff.roughage.com.au/riak-failure.log.gz
>
> It's nearly midnight here and I going to bed but if anyone can shed any
> light on this that'd cool.
>
> rgh
>
> --
> +61 (0) 410 646 369
> [e]:  rgh at roughage.com.au
> [im]: rgh at jabber.org
>
> You're worried criminals will continue to penetrate into cyberspace, and
> I'm worried complexity, poor design and mismanagement will be there to meet
> them - Marcus Ranum
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>



-- 
Best regards,
Dmitry Demeshchuk



More information about the riak-users mailing list