Re: "Proxy reached system memory limit at sockets" after haproxy restart

Hi Matthias,

sorry for the delay. I saw that you were investigating and had more ability than me to do so !

On Wed, Apr 23, 2008 at 08:40:04AM +0200, Matthias Kellermann wrote:
> I did some further research and I think I've found the reason for this
> strange behaviour. Maybe a haproxy bug under FreeBSD, I'm not sure.
>
> The sysctl variable "kern.ipc.numopensockets" shows the number of open
> sockets. "kern.ipc.maxsockets" shows the max. number of sockets allowed
> on the system.
>
> # sysctl kern.ipc.numopensockets
> kern.ipc.numopensockets: 11301
>
> # sysctl kern.ipc.maxsockets
> kern.ipc.maxsockets: 12328
>
> As you can see the maxsockets limit gets nearly hit. If I look deeper
> into the sockets with netstat I can see that most connections go out to
> the http nodes. Thats about >11000 sockets!

What state are they in ? I would suspect FIN_WAIT_1 or 2, meaning that haproxy would have closed upon timeout, but the socket is maintained in the system as long as the client has not ACKed. I have already observed very long-lasting FIN_WAIT_2 sockets on OpenBSD with no obvious solution. Maybe Hugo Silva would have hints on the subject.

> So I think haproxy does not close all the sockets correctly or gets
> somehow in trouble with the FreeBSD sockets system. As for the moment
> I'll try to raise the maxsockets and see if the problem disappears.

If raising the sockets fixes the problem, it means that the sockets get purged late, and that you need to cover the time-frame between the close() and the expiration, times the connection rate. Also, are you sure that your timeouts are correctly set on haproxy ? That's particularly important for the client timeout.

Regards,
Willy Received on 2008/04/29 11:22