Re: "Proxy reached system memory limit at sockets" after haproxy restart

From: Matthias Kellermann <mkellermann#net-com.de>
Date: Tue, 29 Apr 2008 11:33:43 +0200

Hi Willy,

thanks for your answer. Meanwhile I could figure out some more things regarding the full sockets.

Willy Tarreau wrote:

>> As you can see the maxsockets limit gets nearly hit. If I look deeper 
>> into the sockets with netstat I can see that most connections go out to 
>> the http nodes. Thats about >11000 sockets!

>
> What state are they in ? I would suspect FIN_WAIT_1 or 2, meaning
> that haproxy would have closed upon timeout, but the socket is maintained
> in the system as long as the client has not ACKed. I have already observed
> very long-lasting FIN_WAIT_2 sockets on OpenBSD with no obvious solution.
> Maybe Hugo Silva would have hints on the subject.

Right, most where (>99%) in FIN_WAIT_2 state. On the other site (Debian Linux nodes) sockets where in LAST_ACK state.

>> So I think haproxy does not close all the sockets correctly or gets 
>> somehow in trouble with the FreeBSD sockets system. As for the moment 
>> I'll try to raise the maxsockets and see if the problem disappears.

>
> If raising the sockets fixes the problem, it means that the sockets
> get purged late, and that you need to cover the time-frame between
> the close() and the expiration, times the connection rate. Also, are
> you sure that your timeouts are correctly set on haproxy ? That's
> particularly important for the client timeout.

That is exactly what I've done yesterday. I changed the following sysctl values:

net.inet.tcp.keepidle=10000
net.inet.tcp.keepintvl=5000

Now a dead unused socket will get closed more quickly (less than a minute). This works fine, there are only a few connections left in the FIN_WAIT_2 state.

Here are my timeout settings in haproxy.conf: clitimeout 150000
srvtimeout 30000
contimeout 2000

Could these settings be the problem? What settings would you recommend?

-Matthias Received on 2008/04/29 11:33