Re: HAproxy performance on virtual sys. KVM

From: Konrad <km#linuxict.com>
Date: Sat, 10 Jul 2010 11:25:46 +0200

Hi,

W dniu 10.07.2010 10:11, Willy Tarreau pisze:
> Hi Konrad,
>
> On Fri, Jul 09, 2010 at 12:07:05PM +0200, Konrad wrote:
>
>> Hello,
>>
>> I have a question about HAproxy configuration. I have 2 physical servers:
>> 1) ServerA: Debian Lenny 64bit Kernel : 2.6.26-2-amd64 CPU: AMD
>> Athlon(tm) 64 X2 RAM: 2GB
>> 2) ServerB: Debian Lenny 64bit Kernel: 2.6.26-2-amd64 CPU: AMD
>> Athlon(tm) 64 X2 RAM: 4GB
>>
>> On ServerA and ServerB I got 4 vm's:
>> 1) ServerA: vm1 (HAproxy+keepalived) vm11 (installed: Apache - standard
>> conf.) - MEM: 700MB Same OS as above
>> 2) ServerB: vm2 (HAproxy+keepalived) vm21 (installed: Apache - standard
>> conf.) - MEM: 700MB Same OS as above
>>
>> The network configuration for vm's is bridged - br0:
>>
>> haproxy:
>> global
>> log 127.0.0.1 local0
>> log 127.0.0.1 local1 notice
>> maxconn 4096
>> user haproxy
>> group haproxy
>> daemon
>> defaults
>> log global
>> mode http
>> option httplog
>> option dontlognull
>> retries 3
>> option redispatch
>> maxconn 2000
>> contimeout 5000
>> clitimeout 50000
>> srvtimeout 50000
>>
>> listen webfarm 192.168.0.111:80
>> mode http
>> stats enable
>> stats auth admin:111
>> balance roundrobin
>> cookie JSESSIONID prefix
>> option abortonclose
>> option httpclose
>> option forwardfor
>> option httpchk HEAD /check.txt HTTP/1.0
>> server webA 192.168.0.52:80 cookie A maxconn 512 check
>> server webB 192.168.0.62:80 cookie B maxconn 512 check
>>
>>
>> sysctl.conf (vm1, vm2)
>> net.ipv4.tcp_max_syn_backlog = 10240
>> net.ipv4.tcp_max_tw_buckets = 400000
>> net.ipv4.ip_nonlocal_bind=1
>> net.ipv4.tcp_fin_timeout=30
>> net.ipv4.tcp_tw_reuse=1
>> net.ipv4.tcp_max_orphans = 60000
>> net.ipv4.tcp_synack_retries = 3
>> net.core.somaxconn = 10000
>>
>> Test:
>> a) vm11 (ab -c 100 -n 10000 http://127.0.0.1/check.txt):
>> About: Requests per second: 5286.85 [#/sec] (mean)
>> b) from 3 server (ab -c 100 -n 10000 http://192.168.0.111/check.txt):
>> About: Requests per second: 986.80 [#/sec] (mean)
>>
>> I googled a lot and tuned the HAproxy configuration as above. Also the
>> system configs but the performance didn't increase (~ 1k to 5k quite poor).
>> I can send also any additional configurations if it's needed.
>>
> It would be worth checking from the host OS to the VMs too. Also,
> are you running iptables or anything stateful on the hosts ? I've
> never tried KVM yet, but even my (supposedly low-end) lguest performs
> a lot better on my netbook. Reason why I suppose something is
> blocking between your VMs.
>
> Also, with "ab", you could also inject on haproxy's stats page, it
> will simply return you 401 and you'll see if the req/s sensibly
> increases or not. That way you'll know if the bottleneck is between
> ab and haproxy, between haproxy and the servers, or balanced between
> the two sides.
>
> Regards,
> Willy
>
>

Firstly really thanks for the answer.

I have nowhere iptables enabled. Also mentioned in many posts "conntrack" isn't enabled anywhere.

I tested it now like this (same example test with ab: ab -c 100 -n 1000 http://IP_addr/check.txt):
vm5 (additional vm from which I made tests) -> vm11 (here is one of Apache conf.)
ab: ~ Requests per second: 2872.94 [#/sec] vm5 -> vm21 (here is one of Apache conf.) ab: ~ Requests per second: 3089.87 [#/sec] vm5 -> 192.168.0.111 (HA, connects to vm11 and vm21) ab: ~ Requests per second: 1677.04 [#/sec]

the haproxy.conf (changed for test):
server webA 192.168.0.52:80 weight 4 cookie A maxconn 255 check inter 3000 server webB 192.168.0.62:80 weight 24 cookie B maxconn 255 check inter 3000

The test "page" is empty file check.txt (tested with links shows "OK" on every configuration).
I changed also the sysctl.conf (vm1):

net.ipv4.tcp_synack_retries = 3
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1
net.core.rmem_max = 8738000
net.core.wmem_max = 6553600
net.ipv4.tcp_rmem = 8192 873800 8738000
net.ipv4.tcp_wmem = 4096 655360 6553600
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_tw_buckets = 360000

vm.min_free_kbytes = 65536
vm.swappiness = 0

I just can't find out why HA is in my test configuration a bit slow.

Best regards,
Konrad Received on 2010/07/10 11:25