Re: Configuration of Host OS for maximum speed.

On Thu, Apr 21, 2011 at 11:14:26AM +0200, Stijn Vanhoorelbeke wrote:
> Hi,
>
> I was aware of the "ulimit" parameter.
> I already set this ( as well as I increased the nofile parameter ).
> Any other important settings?

"ulimit -n" is automatically adjusted by haproxy from the maxconn value and the number of listeners and possible checks.

When you say that you can't go beyond 1000 concurrent users even if you add servers, what are you observing exactly ? Do you observe degraded response time, connection failures, etc... ? The logs are extremely important in such a situation and will precisely tell you what is happening.

In general, connection errors are caused by an incorrectly tuned network path between haproxy and the server (that includes the local network stack as well as any possible untuned netfilter). High response times are only caused by server processing time, which can include the time the server is waiting for shared resources (SQL, NFS, external WebServices via limited connection pools or bandwidth, servers running as VMs inside the same host, ...). The details can only be observed on the server side, obviously. Shared resources always have the effect of preventing scaling at one point.

High data transfer times are commonly caused by limited network bandwidth between the client and the server. You'll always see a bit of this because some clients will always have limited bandwidth. But if it happens with all clients, it's very likely that it's limited on your side (either the internet link, or another shared link on the server side).

If your haproxy is running inside a VM and you observe random times everywhere and nothing really explainable, first ensure that the host is not saturated. If you can't check for this (eg: running in a shared environment which sells 10 times the real capacity to their customers) then move to a real machine or somewhere else and retry.

2000 concurrent connections is something quite low, still it requires a bit of tuning or at least a few controls. A quick check would at least involve :

• dmesg | grep -i conntrack => check for any error such as "table is full"
• halog -q -e -tc < haproxy-log-file => check for error termination codes, you'll see if you have more timeouts, refused connections, etc...
• halog -q -e -st < haproxy-log-file => check for error statuses (mainly 5xx), it's possible that you'll notice that you have many 504 indicating your servers don't respond fast enough, or that they are themselves waiting for something else.

Regards,
Willy Received on 2011/04/22 07:38