Re: Running haproxy on the cluster nodes

From: Martin Goldman <martin#mgoldman.com>
Date: Tue, 11 Dec 2007 20:15:51 -0500

Thanks Willy. I got excited there for a second, but I tried making the suggested changes, and would you believe me if I said it didn't seem to help?

As you suggested, I updated the conf file with maxconn set to 100000, just so I could be sure that wouldn't be a bottleneck. I recompiled for linux26, and got the appropriate output this time:

martin#kramer:~/haproxy-1.3.13.1$ sudo /usr/sbin/haproxy -f /etc/haproxy.cfg -V
Available polling systems :

     sepoll : pref=400,  test result OK
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK

Total: 4 (4 usable), will use sepoll.
Using sepoll() as the polling mechanism.

I re-ran the apachebench, and the requests per second achieved were still lower than that of each of the web servers individually. I did two tests:

• 500-byte file, 1000 concurrent requests, 500,000 total requests: Individual node = 14,000 requests/second; Cluster = 13,100 requests/second
• 100 KB file, 100 concurrent requests, 50,000 total requests: Individual node = 780 requests/second; Cluster = 400 requests/second

I don't know if there's anything else you can think of, but I certainly appreciate all the ideas thus far.

Thanks,
Martin

On 12/11/07, Willy Tarreau <w#1wt.eu> wrote:
>
> Hi Martin,
>
> On Tue, Dec 11, 2007 at 08:28:49AM -0500, Martin Goldman wrote:
> > Thanks Willy, I better understand what you're saying now. I checked and
> > don't have the ip_conntrack module on my systems.
>
> OK, that's fine.
>
> > I ran haproxy with -V and
> > re-ran the same test, and the only output I got was what's below. I'm on
> > 1.3.13.1 but can upgrade to the newest version if that would be helpful.
>
> That's not required. It would provide you with new shiny features but
> possibly
> you don't need them anyway.
>
> > Just in case there's something outrageously wrong with my config, I'm
> > pasting that below too -- FYI, I removed a few things (such as logging)
> and
> > the performance improved somewhat, but is still worse than that of the
> > individual web servers. Please let me know if you have any other ideas.
> > Thanks for all the help!
> >
> > martin#kramer:~$ sudo /usr/sbin/haproxy -V -f /etc/haproxy.cfg
> > Available polling systems :
> > poll : pref=200, test result OK
> > select : pref=150, test result OK
> > Total: 2 (2 usable), will use poll.
> > Using poll() as the polling mechanism.
>
>
> OK, HERE is the problem: you are using poll. I think you compiled your
> haproxy with the default "generic" target, which is portable but not much
> optimized. You should use epoll() on kerner 2.6, as its CPU usage doesn't
> vary with the number of concurrent connections. In order to do this, just
> build like this :
>
> $ make clean
> $ make TARGET=linux26
>
> Then restart with -V to ensure that it correctly uses epoll() (or sepoll
> which is epoll relying on speculative I/O), and you should be fine.
>
> I see no maxconn in your config below. By default, I think the maxconn is
> somewhere around 1000 or 2000, I don't remember (because I don't use the
> default one). You should specify it, both in the global section and in the
> listen section. If your maxconn is too low, the client connections will
> wait
> for the socket in the system's queues first, then will be dropped when the
> queues are full. So it is very important that you set it.
>
> Also, you can (and should) set a per-server maxconn to slightly less than
> the MaxClients you configured on your apache processes. This will ensure
> that an apache server will never get overloaded. Excess connections will
> simply be queued or sent to the remaining available server(s).
>
> Otherwise, I see nothing wrong.
>
> Regards,
> Willy
>
>
Received on 2007/12/12 02:15