HAProxy randomly returning 502s when balancing IIS

Hello all,

I have tried to diagnose our HAProxy install to see if we can get it solved. It seems that the HAProxy will randomly return 502s for existing items in the end servers.

The configuration is as follows:
Ubuntu Server 9.04 ( Linux valkyrie 2.6.28-15-server #49-Ubuntu SMP Tue Aug 18 20:09:37 UTC 2009 x86_64 GNU/Linux ) HAProxy 1.3.20 (compiled from source with `make TARGET=linux26 USE_PCRE=1` after `apt-get build-dep haproxy` )

HAProxy Config can be found here: http://pastebin.com/f9014ec9

I am using HAProxy to load balance a varnish install on two servers which share virtual Ips through spread and wackamole. HAProxy runs on the same server running varnish and then load balances the web servers.

The http path is:
(browser) -> [ (HAProxy:public) -> (Varnish) -> (HAProxy:varnish_front) ] -> (IIS Cluster)

Where everything between square brackets is (optimally) in the same host. Varnish will only talk with the local HAProxy, but HAProxy will fail over to the other varnish in the cluster (this allows for varnish to fail on a single host, which is necessary because varnish has a fairly long restart time).

All the servers are located in the same VLAN inside the same blade enclosure and average latency is 0.150ms, .026ms mdev.

A snippet of the log (filtered for 502 responses) can be found here: http://pastebin.com/f6db9a351

Previously (haproxy 1.3.15.5, the ubuntu default) the 502s ended with SH- but now they are SL--.

I have tried changing the settings of net.ipv4.tcp_frto and net.ipv4.tcp_sack to see if they change the behavior but they don't seem to fix anything.

For completeness' sake, here is the varnish defaults I am using: http://pastebin.com/d6cb1276e

Any help would be greatly appreciated.

--
Miguel