Re: weird tcp syn/ack problem

Hi,

On Wed, Dec 02, 2009 at 04:47:01PM -0500, Lincoln wrote:
> Hi, I'm running HAProxy as my load balancer and sometimes (but not all the
> time) clients experience an 11s delay. The delay is always about 11s when
> it happens.
>
> I used Wireshark to try and see what was happening (screenshot from the
> capture on the haproxy box attached).
>
> As you can see SYNs are retried over and over but not ACKed until for some
> reason WS, TSV, and TSER are not passed in the request. When this happens
> it always happens the same number of times and always takes the same
> duration before acknowledgement happens.
>
> Here is my config file (it's worth mentioning that 7 of the 10 aux servers
> are not in rotation - don't think that has anything to do with anything).
>
> Any ideas on what is going on here? I'm a relative novice at tuning tcp (my
> tuning script is at the very bottom of this email). Also, this happens
> regardless of whether there is any load on our site.

well, what you show is rather surprizing. It has nothing to do with your haproxy config since haproxy only comes into play when the system receives the ACK from the client. However, I find the case interesting enough to investigate it.

The fact that after some time the SYN is retransmitted without any option is caused by the client which finally attempts to remove some of its options (window scaling and timestamps) in the hope that it will finally establish (and it was right to try because it worked).

The question is why doesn't the system acknowledge thoe SYN packets ?

I see nothing wrong in your network settings BTW.

Hmmm, I have an idea. Would you happen to run your haproxy machine behind a firewall which does sequence number randomization (typically a cisco PIX, FWSM or an OpenBSD firewall) ? If so, what generally happens is that when a source port is reused early by a client and the session is still present in your system's table in TIME_WAIT state, then the second session's sequence number is random and can be less than the last sequence number of the previous session from the same port. The packet is then considered as invalid by the local TCP stack (which conforms to RFC793), but at least your stack should send an ACK back to the client (instead of a SYN/ACK), to indicate what sequence number it expects. The client can then send an RST and retry with a new SYN which will get accepted.

Since we don't see this ACK, it means that either something is preventing the SYN from reaching the stack or something is preventing the SYN/ACK from going out.

Are you running iptables on this machine ? If so, maybe your rules are too strict and either the faulty SYN or the SYN/ACK can't pass through. Check if you have a rule matching state INVALID and check the counter. From memories, I used to accept INVALID states on SYNs and a few other combinations to permit such situations to work, but it was in old times, so things might have changed since.

I see that your local system has tcp_timetamps disabled. Normally, it's the workaround for random sequence numbers. You just enable timestamps and the local stack applies the PAWS algorithm to correctly identify that the new packet is not from the old session. Netfilter also supports it, but I think it will only work for it if your local system emits timestamps. You can do that by echoing 1 into /proc/sys/net/ipv4/tcp_timestamps. It may very well fix the issue.

Please try that and keep us informed. And if your provider runs a firewall as described above, please tell them that their option is broken. At least on cisco it can be disabled on a per-rule basis now :-)

Regards,
Willy Received on 2009/12/02 23:42