Re: Slowness in establishing connections from ruby to Haproxy after upgrading from 1.4.8 to 1.4.18

Hi Willy,

Thanks very much for your response. It looks like it does have something to do with TCP_NODELAY.

Setting "option http-server-close" had no effect on this problem. However, if I set "option http-no-delay" in haproxy then the problem goes away. It also goes away if I patch Ruby's net::http library to set the the TCP_NODELAY flag on the socket.

(In case anyone google's this and is curious, this is how you can do so, at least in ruby 1.9.1)
class Net::HTTP
  def on_connect()
    @socket.io.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_NODELAY, 1)   end
end

I don't know if there's a good reason to favor one solution over the other.  I'll attach tcpdumps in a separate e-mail.

Thanks!
Tim

On Tue, Dec 13, 2011 at 1:42 AM, Willy Tarreau <w#1wt.eu> wrote:

> Hi Tim,
>
> On Mon, Dec 12, 2011 at 02:33:53PM -0500, Timothy Garnett wrote:
> > Hi all,
> >
> > We've been using Haproxy for a couple of years now to load balance our
> > external traffic and our internal backend services. We recently upgraded
> > from version 1.4.8 to 1.4.18 and we noticed some connection
> initialization
> > performance issues when connecting from Ruby's standard net::http
> library,
> > specifically it takes about 200ms longer for the connection to establish
> > then it used to. It's unclear if the issue comes from Ruby or Haproxy.
> > We're interested if anyone has run into similar problems and if there
> are
> > things we can do to help identify the issue for fixing.
>
> Not specifically this one but 200ms sounds like a missing TCP_NODELAY on
> the client side. What I *suspect* (tcpdump on loopback could confirm) is
> that ruby sends HTTP request in small writes (eg: one write per line) and
> for this, disables TCP_NODELAY instead of sending many packets, but fails
> to enable it on the last packet.
>
> Alternatively, something else is possible. Maybe it's not the connection
> time which is huge but the total response time due to similar issues the
> other way around. Please add "option http-server-close" to your defaults
> section to see if it changes anything. Right now your config is in tunnel
> mode which is not very good, although it should not cause this.
>
> Another possibility that comes to mind, since you're sending data, is that
> ruby sends multiple small packets. Due to TCP, it needs an ACK every two
> packet. During a POST, haproxy does a "bulk" forward of data, which sends
> blocks as large as possible, without TCP_NODELAY, to save network
> resources.
> It's possible that this results in ACKs being delayed for incomplete
> packets
> then. Really, a tcpdump between ruby and haproxy would immensely help.
>
> You can also test with "option http-no-delay" which is mainly used for
> chunked-encoding transfers, when either direction is sensible to the
> framing
> (which is a direct violation of HTTP spec).
>
> If you happen to take a network capture, please use "tcpdump -s0" to
> capture
> full packets and save them to a file using "-w file.cap".
>
> Regards,
> Willy
>
>
Received on 2011/12/14 00:01