Re: weird tcp syn/ack problem

I should also mention that the lb is an m1small EC2 instance.

On Wed, Dec 2, 2009 at 4:47 PM, Lincoln <linxbetter#gmail.com> wrote:

> Hi, I'm running HAProxy as my load balancer and sometimes (but not all the
> time) clients experience an 11s delay. The delay is always about 11s when
> it happens.
>
> I used Wireshark to try and see what was happening (screenshot from the
> capture on the haproxy box attached).
>
> As you can see SYNs are retried over and over but not ACKed until for some
> reason WS, TSV, and TSER are not passed in the request. When this happens
> it always happens the same number of times and always takes the same
> duration before acknowledgement happens.
>
> Here is my config file (it's worth mentioning that 7 of the 10 aux servers
> are not in rotation - don't think that has anything to do with anything).
>
> Any ideas on what is going on here? I'm a relative novice at tuning tcp
> (my tuning script is at the very bottom of this email). Also, this happens
> regardless of whether there is any load on our site.
>
> Thanks,
> Lincoln
>
> Config file------
>
> global
> log 127.0.0.1 local0
> log 127.0.0.1 local1 notice
>
> maxconn 50000
> ulimit-n 150000
> chroot /var/lib/haproxy
> user haproxy
> group haproxy
> daemon
>
> nbproc 1 # Number of processing cores. Dual Dual-core Opteron is 4 cores
> for example.
>
> defaults
> log global
> mode http
> option httplog
> option dontlognull
> retries 3
> option redispatch
> option forwardfor
> option httpclose
> timeout check 2s
> timeout client 5s
> timeout server 5s
> timeout connect 2s
> timeout http-request 4s
> timeout queue 5s
>
> frontend www *:80
> mode http
> acl events.hotpotato.com hdr_end(host) -i events.hotpotato.com
> acl go_to_engine path_beg /engine/
> use_backend events_server if events.hotpotato.com
> use_backend events_server if go_to_engine
> default_backend nginxunicorn
>
> backend events_server
> mode http
> fullconn 12000
> option httpchk HEAD /hcheck
> reqirep ^([^\ ]*)\ /engine/(.*) \1\ /\2
> server leftevents left.int.hotpotato.com:5050 minconn 50 maxconn 1000
> check
> server rightevents right.int.hotpotato.com:5050 minconn 50 maxconn 1000
> check
> server arbiterevents arbiter.int.hotpotato.com:5050 minconn 50 maxconn
> 1000 check
> server aux01events aux01.int.hotpotato.com:5050 minconn 50 maxconn 1000
> check
> server aux02events aux02.int.hotpotato.com:5050 minconn 50 maxconn 1000
> check
> server aux03events aux03.int.hotpotato.com:5050 minconn 50 maxconn 1000
> check
> server aux04events aux04.int.hotpotato.com:5050 minconn 50 maxconn 1000
> check
> server aux05events aux05.int.hotpotato.com:5050 minconn 50 maxconn 1000
> check
> server aux06events aux06.int.hotpotato.com:5050 minconn 50 maxconn 1000
> check
> server aux07events aux07.int.hotpotato.com:5050 minconn 50 maxconn 1000
> check
> server aux08events aux08.int.hotpotato.com:5050 minconn 50 maxconn 1000
> check
> server aux09events aux09.int.hotpotato.com:5050 minconn 50 maxconn 1000
> check
> server aux10events aux10.int.hotpotato.com:5050 minconn 50 maxconn 1000
> check
>
> backend nginxunicorn
> mode http
> fullconn 6000
> option httpchk HEAD /check
> server left left.int.hotpotato.com:81 minconn 25 maxconn 500 check
> server right right.int.hotpotato.com:81 minconn 25 maxconn 500 check
> server arbiter arbiter.int.hotpotato.com:81 minconn 25 maxconn 500 check
> server aux01 aux01.int.hotpotato.com:81 minconn 25 maxconn 500 check
> server aux02 aux02.int.hotpotato.com:81 minconn 25 maxconn 500 check
> server aux03 aux03.int.hotpotato.com:81 minconn 25 maxconn 500 check
> server aux04 aux04.int.hotpotato.com:81 minconn 25 maxconn 500 check
> server aux05 aux05.int.hotpotato.com:81 minconn 25 maxconn 500 check
> server aux06 aux06.int.hotpotato.com:81 minconn 25 maxconn 500 check
> server aux07 aux07.int.hotpotato.com:81 minconn 25 maxconn 500 check
> server aux08 aux08.int.hotpotato.com:81 minconn 25 maxconn 500 check
> server aux09 aux09.int.hotpotato.com:81 minconn 25 maxconn 500 check
> server aux10 aux10.int.hotpotato.com:81 minconn 25 maxconn 500 check
>
>
>
> TCP Tuning script - this runs every time the box comes up:
>
> echo 3000 > /proc/sys/net/core/netdev_max_backlog
> echo 10000 > /proc/sys/net/core/somaxconn
> echo 1024 65023 > /proc/sys/net/ipv4/ip_local_port_range
> echo 15 > /proc/sys/net/ipv4/tcp_fin_timeout
> echo 15 > /proc/sys/net/ipv4/tcp_keepalive_intvl
> echo 5 > /proc/sys/net/ipv4/tcp_keepalive_probes
> echo 400000 > /proc/sys/net/ipv4/tcp_max_tw_buckets
> echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
> echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
> echo 10240 > /proc/sys/net/ipv4/tcp_max_syn_backlog
> echo 60000 > /proc/sys/net/ipv4/tcp_max_orphans
> echo 2 > /proc/sys/net/ipv4/tcp_synack_retries
>
Received on 2009/12/02 23:35