Re: HAProxy and TIME_WAIT

Ok, now I'm kind of stuck here.
Let me share you my observations on my really simple evirionment: for client I use wget on server with ip 192.168.2.30 haproxy is set on server with ip 192.168.2.38 and haproxy and web serer comunicate on 127.0.0.1. haproxy is in tcpmode. this is the monitored tcpdump for connection client to haproxy /just the closing connection part/ :
....
14:56:40.448210 IP 192.168.2.30.55867 > 192.168.2.38.443: . ack 7983 win 204 <nop,nop,timestamp 2554292865 2553505887> 14:56:40.448849 IP 192.168.2.30.55867 > 192.168.2.38.443: F 618:618(0) ack 7983 win 204 <nop,nop,timestamp 2554292866 2553505887> 14:56:40.449513 IP 192.168.2.38.443 > 192.168.2.30.55867: F 7983:7983(0) ack 619 win 62 <nop,nop,timestamp 2553505889 2554292866> 14:56:40.449656 IP 192.168.2.30.55867 > 192.168.2.38.443: . ack 7984 win 204 <nop,nop,timestamp 2554292867 2553505889>

and this is tcpdump for 127.0.0.1 /just the closing part again/ : ....
14:56:40.447887 IP 127.0.0.1.59302 > 127.0.0.1.8443: . ack 7983 win 386 <nop,nop,timestamp 2553505887 2553505886> 14:56:40.448914 IP 127.0.0.1.59302 > 127.0.0.1.8443: F 618:618(0) ack 7983 win 386 <nop,nop,timestamp 2553505888 2553505886> 14:56:40.449236 IP 127.0.0.1.8443 > 127.0.0.1.59302: F 7983:7983(0) ack 619 win 273 <nop,nop,timestamp 2553505888 2553505888> 14:56:40.449272 IP 127.0.0.1.59302 > 127.0.0.1.8443: . ack 7984 win 386 <nop,nop,timestamp 2553505889 2553505888>

So that showes me that the connections from haproxy to webserver are closed with FIN/FIN-ACK/ACK.
here is netstat -anpo | grep TIME:

tcp        0      0 127.0.0.1:59302             127.0.0.1:8443
 TIME_WAIT   -                   timewait (58.73/0/0)

is that the expected bahaviour ?

All the best !

2011/11/29 Willy Tarreau <w#1wt.eu>

> Hi Daniel,
>
> On Tue, Nov 29, 2011 at 06:10:46PM +0200, Daniel Rankov wrote:
> > For sure TIME_WAIT connections are not an issue when thay keep
> information
> > about sockets to clients, but when TIME_WAIT connections keep sockets
> bussy
> > for your host where HAProxy is deployed to the backend the limit can be
> > reached - it's defined by ip_local_port_range.
> > Here is what I mean:
> > Client -(1)-> HAProxy -(2)-> Webserver
> > / it doesn't metter if the web server and haproxy are on the same
> server./
> > I) client connects to haproxy
> > socket is tooken - clientIP:random_port:haproxy_ip:haproxy_port
> >
> > II) haproxy connects to webserver
> > socket is tooken haproxy_local_ip:random_port:backend_ip:backend_port
> >
> > III) client closes a conneciton to haproxy (1) in normal way -
> > FIN/FIN-ACK/ACK
> > this way we have one connections that goes from ESTABLISHED to TIME_WAIT
> > state. we don't really care about this TIME_WAIT connection beacause the
> > socket that is tooken is between a client and haproxy
> > - clientIP:random_port:haproxy_ip:haproxy_port
> >
> > IV) haproxy closes the connection to backend (2) with FIN/FIN-ACK/ACK
> > Now this ESTABLISHED connection goes to TIME_WAIT state. and the socket
> > that is tooken is between the haporxy and backend server.
>
> I agree on this point and this is why it does not happen :-)
>
> Haproxy uses an RST to close the connection to the backend server precisely
> because of this, otherwise it would not work at all. You can strace it, you
> will notice that it does a setsockopt(SO_LINGER, {0}) before closing. In
> fact, you cannot even configure it not to do this because it would cause
> too
> much harm.
>
> (...)
> > I believe that the common architecture is that backend servers are
> > phisically close to haproxy and are on high speed networks where no
> packet
> > lost is expected. So we don't really need TIME_WAIT state here. It's not
> > needed on localhost for sure.
>
> When haproxy closes the connection to a server, we never need the TIME_WAIT
> anyway, because if it closes, it means it has nothing left to say to the
> server and is not interested in getting its response. So even if some data
> were lost, it would not be an issue.
>
> For instance, one situation where you can observe this close is when you
> enable forceclose or http-server-close. You'll see that as soon as the
> last byte of payload is received, haproxy sends an RST to the server to
> release the connection so that another pending request may immediately
> reuse it.
>
> Even if the RST was lost, a packet from the server would reach the haproxy
> machine and match no known connection, causing an RST in return.
>
> That said, I completely agree with all your analysis and that's what has
> caused me a lot of gray hair when implementing the client-side keep-alive,
> precisely because I needed a way to make haproxy close the server
> connection
> without being affected by the TIME_WAIT on this side.
>
> Best regards,
> Willy
>
>
Received on 2011/11/30 14:56