Re: haproxy bug or wrong kernel settings?

Hi Craig,

On Wed, Jun 30, 2010 at 07:16:32PM +0200, Craig Craig wrote:
> It seems that somewhere over 2000 and between 4500 established connections, the problems start; I've not been able to determine
> the exact number, as I've changed the NAT to the server directly - it could handle the ~6600 connections without problems.
>
> When I was querying a server through haproxy (on the haproxy itself), I saw this huge lag:
>
> 1.) time printf "GET / HTTP/1.1\r\nhost: www.foo.de\r\nConnection: close\r\nCookie: -\r\n\r\n" | nc -v 192.168.92.11 8085 &>/dev/null
> real 0m19.976s
> user 0m0.000s
> sys 0m0.008s

Such very long times generally indicate huge packet drops or session drops. I'm seeing that you have nf_conntrack loaded on the server, are you absolutely sure that the session table never fills up ? You can check that with "dmesg". I'm asking because this is an extremely common issue. Just in doubt, you should check if you can disable it.

> My config:

a few comments below.

> ############# haproxy.cfg
> global
> user haproxy
> group haproxy
> maxconn 75000
> ulimit-n 192000

you can safely remove ulimit-n above, it's correctly computed from maxconn.

> log 127.0.0.1 local0
>
> defaults
> timeout client 300s
> timeout server 300s
> timeout queue 60s
> timeout connect 7s
> timeout http-request 10s
>
> backend backend_btg
> mode http
> balance hdr(X-Forwarded-For)
> option redispatch
> option httpchk HEAD / HTTP/1.1\r\nHost:\ www.foo.de
> server Sxxx 192.168.71.43:80 weight 100 maxconn 16384 check inter 10000 fall 2 rise 2
>
> frontend frontend_btg
> bind 0.0.0.0:8085
> mode http
> option httplog
> reqidel ^X-Forwarded-For:.*
> option forwardfor except 192.168.97.11
> log 127.0.0.1 local0
> capture request header Host len 192
> timeout client 1m

OK I see. You have no "maxconn" setting in your frontend. So it's limited to the default value (2000). You should set it slightly below the global maxconn setting (which is for the whole process).

(...)

> ################################################################################################
> ###### sysctl -a output:

(...)

your sysctls look correct overall.

Regards,
Willy Received on 2010/06/30 23:08