Re: haproxy alleatory fails on config reload

On Thu, May 29, 2008 at 06:00:18PM +0200, Pablo Escobar wrote:
> Hi,
>
> Many thanks for your answer Willy, really helpful as always :)
>
> After your indications I tried to increase the check interval and reduce the
> rise parameter and I have been testing during this week. For three days it
> hasn´t failed but today suddenly the 503 error came back.
>
> I havent found that the check column is growing in noany of the backends.
>
> I have a quite strange behaviour. If I directly connect to the haproxy
> listening on port 81 I can see that all my backeds are up but I get the 503
> error for around 10 -15 seconds after the reload. This happens randomly.
>
> Also if a backend is down when I do the reload I get the 503 on all my
> backend, not just the down backend. ¿is this the normal behaviour?

No, but it sounds to me like a network problem on the proxy's network interface. Having several backends down at start time means that some health-checks have failed at least once. The fact that you get them more often when you reload may simply be because during a reload you already have traffic on the interface, while if you start from scratch you don't have any traffic yet so the health-checks have more chances to pass.

This would also explain why you get them all down if you reload during a problem : if you see one backend down, it means the problem is happening again. Then you reload everything and get all checks to fail on startup. This could be caused by a duplex mismatch between the haproxy server and the switch (most often a 100 Mbps switch forced to 100full with a server set to autoneg).

Please check "ethtool -i eth0" on the server. I would not be surprized to see "100 Half" here. Also, "dmesg" might be showing links going UP and DOWN.

You may as well be using spanning tree on your local switches, with a failing link causing occasionnal STP recalculations during which no traffic can pass.

> Right now I have a reverse proxy using apache´s mod_proxy which sends all my
> inbound http traffic to the haproxy listening on the same machine on port 81.
> ¿maybe this can be affecting?

no, I see no reason for that to cause any problem.

> I am doing it in this way because having a
> apache´s vhost which processes all my http traffic let me apply mod_rewrite,
> mod_security and mod_cband to all my traffic.
>
> If you are interested I can send you my vhost config and my haproxy config.
>
> many thanks in advance for any help.
>
> Pablo.
>
> p.d. I havent forgot about the docs to get snmpd working with perl support.
> Sorry for the delay, late weeks I have been overloaded with work. I promise
> to send it this week.

No emergency, keep calm :-)
We're all used to disappear for weeks and re-appear here on the list. It seems we all have a ton of work :-)

Regards,
Willy Received on 2008/05/29 22:40