Re: make haproxy notice that backend server ip has changed

On Fri, Aug 05, 2011 at 11:11:50PM +0300, Piavlo wrote:
> >It's not a matter of config option. You're supposed to run haproxy
> >inside a chroot. It will then not have access to the resolver.
> There are simple ways to make the resolver work inside chroot without
> making the chroot less secure.

I don't know any such simple way. If you're in a chroot, you have no FS access so you can't use resolv.conf, nsswitch.conf, nor even load the dynamic libs that are needed for that. The only thing you can do then is to implement your own resolver and maintain a second config for this one. This is not what I call a simple way.

> > I could ask the question the other direction : why try to resolve a
> >name to IP when a check fails, there is no reason why a server would
> >have its address changed without the admin being responsible for it.
> I don't agree that admin is supposed to be responsible for it directly
> at all.

So you're saying that you find it normal that a *server* changes its IP address without the admin's consent ? I'm sorry but we'll never reach an agreement there.

> Say backend server crashes/enters bad state - this is detected and new
> ec2 instance is automatically spawned and autoconfigured to
> replace the failed backend ec2 instance- which is optionally terminated.
> The /etc/hosts of all relevent ec2 instances is auto updated (or DNS
> with 60 seconds ttl is updated - by the way the 60 seconds ttl works
> great withing ec2). There is no admin person involved - all is done
> automatically.

That's what I'm explaining from the beginning : this *process* is totally broken and does not fit in any way in what I'd call common practices :

• a failed server is replaced with another server with a different IP address. It could very well have kept the same IP address. If servers in datacenters had their IP address randomly changed upon every reboot it would require many more men to handle them.
• you're not even shoked that something changes the /etc/hosts of all of your servers when any server crashes. That's something I would never accept either. Of course, the only reason for this stupidity is the point above.
• on top of that the DNS is updated every 60 seconds. That means that any process detecting the failure faster than the DNS updates will act based on the old IP address and possibly never refresh it. Once again, this is an ugly design imposed by the first point.

I'm sorry Piavlo, but I can't accept such mechanisms. They are broken from scratch, there is no other word. A server's admin should be the only person who decides to change the server's address. Once you decide to let stupid process change everything below you, you can't expect some software to guess things for you and to automagically recover from the mess.

> >Also, in your case it would not fix the issue : resolving when the
> >server goes down will bring you the old address, and only after
> >caches expires it would bring the new one.
> If /etc/hosts is updated locally the is no need to wait for cache
> expiration.

1. /etc/hosts is out of reach in a chroot
2. it's out of question to re-read /etc/hosts before every connection.
3. if you don't recheck before every connection, you can connect to the wrong place due to the time it takes to propagate changes.

> And if /etc/hosts is auto updated by appropriate tool - going one more
> step of restarting/reloading haproxy is not a problem at all - but this
> is what I want to avoid.

If you want to avoid this mess, simply configure your servers not to change address with the phases of the moon.

> If instead for example i could send a command to haproxy control socket
> to re-resolve all the names (or better just specific name) configured in
> haproxy - it would be much better - as since /etc/hosts is already
> updated it would resolve to correct ip address.

It could not because it's not supposed to be present in the empty chroot.

> BTW afaiu adding/removing backends/frontends dynamically on the fly
> through some api / socket - is not something that is ever planned to be
> supported in haproxy?

At the moment it's not planned because it requires to dynamically change limits that are set upon startup, such as the max memory and max FD number. Maybe in the future we'll be able to start with a configurable margin to add some servers, but that's not planned right now. Changing a server's address by hand might be much easier to implement though, eventhough it will obviously break some protocols (eg: RDP). But it could fit your use case

Regards,
Willy Received on 2011/08/05 22:42