Re: server sizing for haproxy

Hi Marc,

On Mon, Dec 03, 2007 at 02:53:21PM -0500, Marc wrote:
> Hi All,
>
> I've been using haproxy for a while (v1.2.16) and I'm noticing that my CPU
> usage for the haproxy process is around 70% during normal load with
> occasional spikes to nearly 100%. We're running on a basic desktop class PC
> right now with 1 Intel(R) Celeron(R) CPU 2.53GHz CPU and 768MB.
>
> We're at roughly 2000 concurrent connections right now and about 20Mb of
> bandwidth at peak.
>
> Is our hardware definately too small at this point or could something else
> be wrong?

Most likely you're just using select() or poll() which scale badly. I'm almost sure that all the CPU usage you observe is spent in system time and not in user time.

Under linux 2.6 you can enable epoll() which scales better. On FreeBSD/OpenBSD, you can use kqueue(), but that one is only in 1.3.

> Would upgrading to haproxy 1.3 help or hurt us?

It may help a lot, because several parts have been improved for performance. However, you must keep in mind that while I maintain stable branches out of the most recent versions, it is still considered development code. But I'd say that a bunch of people on the list happily use it on big production systems without a glitch. I'm considering switching 1.3 to stable very soon (1.3.15 or 1.3.16, depending on what remains to be merged).

> If you do think we're undersized on the hardware, any suggestions as to how
> much we'd need to 5-10x where we are now?

No, the number of concurrent connections is not hardware-related. It just depends on the available RAM. Going over 10k is quite trivial. Above it requires quite a lot of system tuning, but once you get into it, there's no problem scaling up to 40-50k concurrent connections (I did).

What normally takes your CPU time is the number of new connections you establish every second. With a CPU like yours, you should have no problem reaching 10-15k connections per second. But this alone requires an awful amount of system tuning. Also, if it is a linux, verify that netfilter is not installed on the same box, as it can cut your performance by half.

Hoping this helps,
Willy Received on 2007/12/03 22:28