Re: Slowness in establishing connections from ruby to Haproxy after upgrading from 1.4.8 to 1.4.18

Hi Tim,

On Mon, Dec 12, 2011 at 02:33:53PM -0500, Timothy Garnett wrote:
> Hi all,
>
> We've been using Haproxy for a couple of years now to load balance our
> external traffic and our internal backend services. We recently upgraded
> from version 1.4.8 to 1.4.18 and we noticed some connection initialization
> performance issues when connecting from Ruby's standard net::http library,
> specifically it takes about 200ms longer for the connection to establish
> then it used to. It's unclear if the issue comes from Ruby or Haproxy.
> We're interested if anyone has run into similar problems and if there are
> things we can do to help identify the issue for fixing.

Not specifically this one but 200ms sounds like a missing TCP_NODELAY on the client side. What I *suspect* (tcpdump on loopback could confirm) is that ruby sends HTTP request in small writes (eg: one write per line) and for this, disables TCP_NODELAY instead of sending many packets, but fails to enable it on the last packet.

Alternatively, something else is possible. Maybe it's not the connection time which is huge but the total response time due to similar issues the other way around. Please add "option http-server-close" to your defaults section to see if it changes anything. Right now your config is in tunnel mode which is not very good, although it should not cause this.

Another possibility that comes to mind, since you're sending data, is that ruby sends multiple small packets. Due to TCP, it needs an ACK every two packet. During a POST, haproxy does a "bulk" forward of data, which sends blocks as large as possible, without TCP_NODELAY, to save network resources. It's possible that this results in ACKs being delayed for incomplete packets then. Really, a tcpdump between ruby and haproxy would immensely help.

You can also test with "option http-no-delay" which is mainly used for chunked-encoding transfers, when either direction is sensible to the framing (which is a direct violation of HTTP spec).

If you happen to take a network capture, please use "tcpdump -s0" to capture full packets and save them to a file using "-w file.cap".

Regards,
Willy Received on 2011/12/13 07:42