Re: HAproxy performance on virtual sys. KVM

Hi Konrad,

On Mon, Jul 12, 2010 at 07:49:25PM +0200, Konrad wrote:
> The networking is slow - you're correct. I'll check it why.
>
> But generally I don't know why the stats using "ab" tool can be so low.
> If I add additional server with apache there is hardly any performance
> increase. This makes me worry.
> I got 3 apache servers on every - remotely tested - about 3-4k req/s.
> And with HAproxy about 2k req/s.

but as I understand it, in the first case (apache only), your test is :

  ab --[network]--> apache

In the second case, your test is :

  ab --[network]--> haproxy --[network]--> apache

So if you pass twice over the network and it's the bottleneck, it's perfectly expected that the performance cuts in half.

> I made also a configuration with virtualbox just for test (1 HA, 3 vm's
> with apache). Situation is similar.
> To every vm there is about 600 req/s. And using HAproxy about 400 req/s.

Huh, the numbers are even 5 times lower ! Was that on the same physical machine ? If so, that should ring some bell about the host's tuning. I think the last time I've seen that low numbers, it was on a Pentium 90. And even in a VM I don't recall that low numbers.

> Right now I'm not sure but I'll also check the load on cpu on vm's.

The CPU on the host is more important. The one in the VM can report anything between 0 and 100% depending on what the host lets it really do.

> Generally I tried different configurations of haproxy.conf. But as you
> can confirm the effect is not good.
> Could you please suggest me what - more or less - effect should I expect
> (in the upper connection schema - 3-4k req/s to 2k req/s)?

From memories, haproxy under lguest on my Atom 1.6 GHz does slightly more than 4000 req/s. So any decent machine with hardware virtualization should do quite more.

There are some important tests you should do :

• move ab to the VMs running apache and launch it locally to check the difference.
• repeat test above on all apache VMs simultaneously to see how the numbers go. Do they add or do they cumulate to the same level ? That will indicate the impact on the CPU of running apache in those VMs.
• move ab to the VM running haproxy, and start ab directly from there to one apache. Note the numbers and check CPU usage on the haproxy VM.
• run ab from the haproxy VM to the local haproxy, configured to address the same apache as above. Note the numbers. Also check how CPU spreads between ab and haproxy on the same machine. If either uses a full CPU or if both add to 100 or more percent, you'll have to do some math to see if what you observe is correct versus previous test.
• move ab to another VM and address haproxy (1 server still). Check all numbers (including host's CPU usage).
• do the same again, but repeat test #1 on the two unused apache VMs to see if the CPU usage of unused VMs has any impact on the global load.
• do the same again, but use ab from VM1 to address apache in VM2 and conversely, to see if the network usage of unused VMs has any impact on the global load.

I really suspect that the issue is making the packets pass between VMs multiple times, which adds up network traffic that the host cannot handle between VMs for whatever reasons.

Hoping this helps,
Willy Received on 2010/07/12 20:13