Re: HAProxy and TIME_WAIT

Hi Daniel,

On Tue, Nov 29, 2011 at 06:10:46PM +0200, Daniel Rankov wrote:
> For sure TIME_WAIT connections are not an issue when thay keep information
> about sockets to clients, but when TIME_WAIT connections keep sockets bussy
> for your host where HAProxy is deployed to the backend the limit can be
> reached - it's defined by ip_local_port_range.
> Here is what I mean:
> Client -(1)-> HAProxy -(2)-> Webserver
> / it doesn't metter if the web server and haproxy are on the same server./
> I) client connects to haproxy
> socket is tooken - clientIP:random_port:haproxy_ip:haproxy_port
>
> II) haproxy connects to webserver
> socket is tooken haproxy_local_ip:random_port:backend_ip:backend_port
>
> III) client closes a conneciton to haproxy (1) in normal way -
> FIN/FIN-ACK/ACK
> this way we have one connections that goes from ESTABLISHED to TIME_WAIT
> state. we don't really care about this TIME_WAIT connection beacause the
> socket that is tooken is between a client and haproxy
> - clientIP:random_port:haproxy_ip:haproxy_port
>
> IV) haproxy closes the connection to backend (2) with FIN/FIN-ACK/ACK
> Now this ESTABLISHED connection goes to TIME_WAIT state. and the socket
> that is tooken is between the haporxy and backend server.

I agree on this point and this is why it does not happen :-)

Haproxy uses an RST to close the connection to the backend server precisely because of this, otherwise it would not work at all. You can strace it, you will notice that it does a setsockopt(SO_LINGER, {0}) before closing. In fact, you cannot even configure it not to do this because it would cause too much harm.

(...)
> I believe that the common architecture is that backend servers are
> phisically close to haproxy and are on high speed networks where no packet
> lost is expected. So we don't really need TIME_WAIT state here. It's not
> needed on localhost for sure.

When haproxy closes the connection to a server, we never need the TIME_WAIT anyway, because if it closes, it means it has nothing left to say to the server and is not interested in getting its response. So even if some data were lost, it would not be an issue.

For instance, one situation where you can observe this close is when you enable forceclose or http-server-close. You'll see that as soon as the last byte of payload is received, haproxy sends an RST to the server to release the connection so that another pending request may immediately reuse it.

Even if the RST was lost, a packet from the server would reach the haproxy machine and match no known connection, causing an RST in return.

That said, I completely agree with all your analysis and that's what has caused me a lot of gray hair when implementing the client-side keep-alive, precisely because I needed a way to make haproxy close the server connection without being affected by the TIME_WAIT on this side.

Best regards,
Willy Received on 2011/11/29 18:43