RE: Custom health checks

Not saying this is a good way or not, but one method is to do something like the following on the servers:

iptables -A INPUT -p tcp --dport 3306 --syn -j REJECT

when it wants to mark itself down. (replace 3306 with whatever port you want to flag down). Only matching on syn packets, so existing connections will continue.

iptables -D INPUT -p tcp --dport 3306 --syn -j REJECT

to return to normal.

I use this method for a soft down type script easy to run on an individual server, but could be used as part of a generic health type test that runs on each server. Implementation is left as an exercise for the reader (cron once a minute, some monitoring daemon or script that checks more frequently, etc...)

If you do it based on load, just be careful not to get yourself in a chain reaction situation where a server auto shuts down new connections coming in, causing the load of other servers to go up, which then shut themselves out too, and soon no servers are left to respond...

> -----Original Message-----
> From: Guy [mailto:wyldfury#gmail.com]
> Sent: Wednesday, November 25, 2009 6:04 AM
> To: haproxy#formilux.org
> Subject: Custom health checks
>
> Hi,
>
> I'm using haproxy to proxy various tcp traffic (MySQL, IMAP, POP3,
> SMTP).
>
> I've had a couple of situations now where a backend server reaches
> really high load and doesn't reject connections, but becomes
> incredibly slow to respond. Since the connections aren't rejected
> haproxy doesn't seem to realise that there is a problem with the
> server.
> I was thinking that some sort of daemon on the back end server could
> be used to respond to haproxy's health checks since haproxy has no
> real way of telling there's a problem since connections are just
> horribly slow.
>
> What would be the best way to use something like this in haproxy?
> httpchk and return http like responses from the daemon? Or something
> else?
> I figure this would also allow for more indepth checks than a simple
> tcp connection check depending on the service being tested.
>
> Any advice or alternative ideas would be great.
>
> Thanks
> Guy
>
> --
> Don't just do something...sit there!
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.5.425 / Virus Database: 270.14.69/2508 - Release Date:
> 11/24/09 19:37:00
Received on 2009/11/25 12:45