Re: TCP Socket TIME_WAIT error Windows WCF

On 25 May 2010 16:21, Willy Tarreau <w#1wt.eu> wrote:

> Hi,
>
> Could you please recheck the state on the server ? I don't believe a minute
> that it runs "out of sockets" if they are in TIME_WAIT state because
> TIME_WAIT
> is a final state without data and the socket does not exist anymore in the
> process which previously hold it. You can easily have millions of them,
> they
> are harmless. OK this is Windows, but their TCP stack is not *that* bad.
>
> What may happen however is that if you stop and restart the service, then
> it
> cannot bind because of "address already in use", but I hope that IIS
> developers
> have handled this very common issue.
>

The actual error message is as follows:

Insufficient winsock resources available to complete socket connection initiation.

System.InsufficientMemoryException: Insufficient winsock resources available to complete socket connection initiation. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was.....

>
> > When removing the load balancer and having the requests go directly to
> the
> > web server there are only a maximum of 10 sockets in the TIME_WAIT state
> at
> > any one time.
>
> This is because when you connect directly, you maintain the connections for
> as long as possible, while via haproxy they are closed after every
> exchange.
>

The WCF windows client which connects directly to the server is configured to disable HTTP keepalives and cookies. Also the WCF service running on the web server is stateless. Does this make a difference or are you saying that HAProxy closes TCP/IP connections every time as opposed to HTTP connections? If so, is there a way to get HAProxy to not close the connection after every exchange?

Another thing I should mention is that when we tried the setup with NLB<http://en.wikipedia.org/wiki/Network_Load_Balancing_Services>(configured with "Multiple Host", "Affinity: none") we didn't see this problem. Would you happen to know if NLB closes connections on each request or keeps them open and reuses them?

> > It seems like there's some error in communication between HAProxy and
> > Windows/IIS7 where the sockets aren't closed properly.
>
> In fact it's the opposite. If you see them in the TIME_WAIT state, then
> they
> are properly closed. How did you conclude that Windows ran out of sockets ?
> Just because you can't connect anymore ?

See the error message above.

> Are you sure you don't have iptables
> loaded on your load balancer, which would have its state stable filled
> after
> a few thousand tests and which would refuse to let new connections pass ?
>

The setup is a default Debain Lenny install. The iptables firewall does not have any rules in it, although the firewall itself is not completely disabled.

Thank you

-- 
Srđan Đukić