enotconn error

Luke Bakken lbakken at basho.com
Wed Oct 12 07:51:15 EDT 2016


Travis -

What are the client failures you are seeing? What Riak client library
are you using, and are you using the PB or HTTP interface to Riak?

The error message you provided indicates that the ping request
returned from Riak after haproxy closed the socket for the request.
One cause would be very high server load causing timeouts.
--
Luke Bakken
Engineer
lbakken at basho.com


On Wed, Sep 14, 2016 at 7:23 AM, Travis Kirstine
<tkirstine at firstbasesolutions.com> wrote:
> Hi all,
>
>
>
> I’m using haproxy with a riak 2.1.4 in a 5 node cluster.  I’m getting fairly
> consistent enotconn errors in riak which happen to coincide with client
> failures.  We’ve setup haproxy as recommended
> (https://gist.github.com/gburd/1507077) see below.  I’m running a leveldb
> backend with 9 GB max memory (I can go higher if needed).  I’m not sure at
> this point if I have a network issue or leveldb / riak issue.
>
>
>
> 2016-09-14 08:10:16 =CRASH REPORT====
>
>   crasher:
>
>     initial call: mochiweb_acceptor:init/3
>
>     pid: <0.28104.442>
>
>     registered_name: []
>
>     exception error:
> {function_clause,[{webmachine_request,peer_from_peername,[{error,enotconn},{webmachine_request,{wm_reqstate,#Port<0.6539192>,[],undefined,undefined,undefined,{wm_reqdata,'GET',http,{1,0},"defined_in_wm_req_srv_init","defined_in_wm_req_srv_init",defined_on_call,defined_in_load_dispatch_data,"/ping","/ping",[],defined_in_load_dispatch_data,"defined_in_load_dispatch_data",500,1073741824,67108864,[],[],{0,nil},not_fetched_yet,false,{0,nil},<<>>,follow_request,undefined,undefined,[]},undefined,undefined,undefined}}],[{file,"src/webmachine_request.erl"},{line,150}]},{webmachine_request,get_peer,1,[{file,"src/webmachine_request.erl"},{line,124}]},{webmachine,new_request,2,[{file,"src/webmachine.erl"},{line,69}]},{webmachine_mochiweb,loop,2,[{file,"src/webmachine_mochiweb.erl"},{line,49}]},{mochiweb_http,headers,5,[{file,"src/mochiweb_http.erl"},{line,96}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
>
>     ancestors: ['http://192.168.18.64:8098_mochiweb',riak_api_sup,<0.319.0>]
>
>     messages: []
>
>     links: [<0.325.0>,#Port<0.6539192>]
>
>     dictionary: []
>
>     trap_exit: false
>
>     status: running
>
>     heap_size: 987
>
>     stack_size: 27
>
>     reductions: 963
>
>   neighbours:
>
>
>
>
>
> # haproxy.cfg
>
> global
>
>     log         127.0.0.1 local2 info
>
>     chroot      /var/lib/haproxy
>
>     pidfile     /var/run/haproxy.pid
>
>     maxconn     256000
>
>     user        haproxy
>
>     group       haproxy
>
>     spread-checks       5
>
>     daemon
>
>     quiet
>
>     stats socket /var/lib/haproxy/stats
>
> defaults
>
>     log                     global
>
>     option                  httplog
>
>     option                  dontlognull
>
>     option                  redispatch
>
>     timeout connect         5000
>
>     maxconn                 256000
>
>
>
> frontend  main *:80
>
>     mode                    http
>
>     acl url_static       path_beg       -i /static /images /javascript
> /stylesheets
>
>     acl url_static       path_end       -i .jpg .gif .png .css .js
>
>
>
> backend static
>
>     balance     roundrobin
>
>     server      static 127.0.0.1:4331 check
>
>
>
> backend app
>
>     mode                    http
>
>     balance     roundrobin
>
>     server  wmts1riak 192.168.18.72:80 check
>
>     server  wmts2riak 192.168.18.73:80 check
>
>
>
> backend riak_rest_backend
>
>        mode               http
>
>        balance            roundrobin
>
>        option             httpchk GET /ping
>
>        option             httplog
>
>        server riak1 192.168.18.64:8098 weight 1 maxconn 1024  check
>
>        server riak2 192.168.18.65:8098 weight 1 maxconn 1024  check
>
>        server riak3 192.168.18.66:8098 weight 1 maxconn 1024  check
>
>        server riak4 192.168.18.67:8098 weight 1 maxconn 1024  check
>
>        server riak5 192.168.18.68:8098 weight 1 maxconn 1024  check
>
>
>
> frontend riak_rest
>
>        bind               *:8098
>
>        mode               http
>
>        option             contstats
>
>        default_backend    riak_rest_backend
>
>
>
> backend riak_protocol_buffer_backend
>
>        balance            leastconn
>
>        mode               tcp
>
>        option             tcpka
>
>        option             srvtcpka
>
>        server riak1 192.168.18.64:8087 weight 1 maxconn 1024  check
>
>        server riak2 192.168.18.65:8087 weight 1 maxconn 1024  check
>
>        server riak3 192.168.18.66:8087 weight 1 maxconn 1024  check
>
>        server riak4 192.168.18.67:8087 weight 1 maxconn 1024  check
>
>        server riak5 192.168.18.68:8087 weight 1 maxconn 1024  check
>
>
>
> frontend riak_protocol_buffer
>
>        bind               *:8087
>
>        mode               tcp
>
>        option             tcplog
>
>        option             contstats
>
>        mode               tcp
>
>        option             tcpka
>
>        option             srvtcpka
>
>        default_backend    riak_protocol_buffer_backend
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> riak-users mailing list
> riak-users at lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>




More information about the riak-users mailing list