Re: HAProxy + Heartbeat

Hi Daniel,

On Tue, Mar 23, 2010 at 10:06:35PM +0100, Daniel Storjordet wrote:
> Hi!
>
> HaProxy is great!
>
> We have 2 servers running VMWare ESXi, each having a CentOS with HaProxy
> and Windows 2008 virtual machines. The 2nd innstallasjon of CentOS with
> HaProxy has been lying dormant for awhile but now it is time to try to
> setup heartbeat so we can benefit from high availability. I have been
> using the this guide:
> http://www.howtoforge.com/high-availability-load-balancer-haproxy-heartbeat-debian-etch
>
> Our configuration differs however from the examples. Both our CentOS
> installations have 2 network cards. One for internet and one for local
> communication in between servers. The internet network adapter (eth2) is
> the one who has the shared internet ip, while the local (eth3) has the
> local ip.
>
> The questions is if I have to change some of the values of ha.cf from
> eth3 to eth2 (Current version below):
>
> keepalive 2
> deadtime 10
> udpport 694
> bcast eth3
> mcast eth3 225.0.0.1 694 1 0
> ucast eth3 192.168.100.2
> udp eth3
> logfacility local0
> node haproxy1
> node haproxy2

Well, I'm not sure whether you'll find a response here as this is purely a heartbeat question.

Anyway, I'd like to say that I'm amazed by the number of people who use heartbeat to get a redundant haproxy setup. It is not the best tool for *this* job, it was designed to build clusters, which is a lot different from having two redundant stateless network equipments. Network oriented tools such as keepalived or ucarp are the best suited for that task.

The difference between those two families is simple :

• a cluster-oriented product such as heartbeat will ensure that a shared resource will be present at *at most* one place. This is very important for shared filesystems, disks, etc... It is designed to take a service down on one node and up on another one during a switchover. That way, the shared resource may never be concurrently accessed. This is a very hard task to accomplish and it does it well.
• a network-oriented product such as keepalived will ensure that a shared IP address will be present at *at least* one place. Please note that I'm not talking about a service or resource anymore, it just plays with IP addresses. It will not try to down or up any service, it will just consider a certain number of criteria to decide which node is the most suited to offer the service. But the service must already be up on both nodes. As such, it is very well suited for redundant routers, firewalls and proxies, but not at all for disk arrays nor filesystems.

The difference is very visible in case of a dirty failure such as a split brain. A cluster-based product may very well end up with none of the nodes offering the service, to ensure that the shared resource is never corrupted by concurrent accesses. A network-oriented product may end up with the IP present on both nodes, resulting in the service being available on both of them. This is the reason why you don't want to serve file-systems from shared arrays with ucarp or keepalived.

The nature of the controls and changes also has an impact on the switch time and the ability to test the service offline. For instance, with keepalived, you can switch the IP from one node to another one in just one second in case of a dirty failure, or in zero delay in case of volunteer switch, because there is no need to start/stop anything. That also means that even if you're experiencing flapping, it's not a problem because even if the IP constantly moves, it moves between places where the service is offered. And since the service is permanently available on the backup nodes, you can test your configs there without impacting the master node.

So in short, I would not like to have my router/firewall/load balancer running on heartbeat, as well as I would not like to have my fileserver/ disk storage/database run on keepalived.

With keepalived, your setup above is trivial. Just configure two interfaces with their shared IP addresses, enumerate the interfaces you want to track, declare scripts to check the services if you want and that's all. If any interface fails or if haproxy dies, the IP immediately switches to the other node. If both nodes lose the same interface (eg: shared switch failure), you still have part of the service running on one of the nodes on the other interface.

Hoping this helps understanding the different types of architectures one might encounter,

Willy Received on 2010/03/25 22:38