Re: How does haproxy handle dead servers?

Hi Jan,

On Fri, Jan 04, 2008 at 03:22:18PM +0100, Jan Miczaika wrote:
> Hello,
>
> we are writing a script which reports a server as dead (504 http status
> code) when one of a number of conditions occurrs. The server is
> overloaded, is currently running cron jobs etc. This script will be
> entered as the health check script in haproxy.
>
> We do have one question. Assume the server is full of heavy php
> instances and prefers to mark itself unavailable for new connections.
> The script sets itself to 504. Haproxy notices this and marks the server
> as dead, thereby not sending new connections to the server.
>
> What happens to the current connections, which are still open on the
> server? Does haproxy in some way close them, or can the client still
> receive his data?

No, the established connections are not closed. However, if you need persistence, you'll have a problem as the next request from a user attached to a down server will be redispatched to another server.

Another solution would be to use the new feature introduced in haproxy 1.3.14 : "http-check disable-on-404". This statement says that if a server returns 404 to a health check, it will immediately be disabled from load-balancing, but will still receive persistent connections which reference it. Only the load-balancing algorithm will evict it.

It was designed explicitly for smooth maintenance mode without kicking the users off of the server. Of course, if you return any other error (such as 504) then the server will be considered dead. So basically the right method with this option is to follow this scheme of HTTP responses :

• server OK -----> | <---- stopping -----> | <----- dead -----> 2xx or 3xx | 404 | 4xx or 5xx or nothing

It also permits some sort of flow control. If your script is able to detect overload, it may return only 404 for this situation so that the server will not receive new clients, but still process current ones. Once the load is OK, you can return 200 and it will get new clients again. One such script would look like this :

  if (!server_is_alive())

     return 504
  else if (server_overloaded)

     return 404
  else

     return 200

If you have the stats page enabled, a server in 404 mode will be reported as "NOLB", which means "UP" but without LB.

Regards,
Willy Received on 2008/01/05 15:38