Re: Running haproxy on the cluster nodes

These are hosted dedicated servers, so I don't have much info on the hardware. But from looking at dmesg, it looks like I may have one Intel PCI GigE card (which is connected to the LAN) and and a Broadcom/Tigon3 PCI Express card (which is connected to the Internet):

e1000: 0000:08:02.0: e1000_probe: (PCI:33MHz:32-bit) [MAC_address]
e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
e1000: eth1: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex

eth0: Tigon3 [partno(BCM95751) rev 4201 PHY(5750)] (PCI Express)10/100/1000BaseT Ethernet [MAC address] tg3: eth0: Link is up at 100 Mbps, full duplex.

I googled the Intel card (model 82541PI according to lspci) and it is indeed a PCI 33/66MHz card, and the Broadcom (model BCM5751) is indeed a PCI Express card. Would having one of each type of card like this allow for the type of bus saturation that you describe below? I'm not sure if the hosting company will do it, but should I look into having the PCI card swapped out for another PCI Express?

Many thanks for all the explanation; as you can obviously tell, I'm not much of a hardware guy :)

-Martin

On 12/12/07, Willy Tarreau <w#1wt.eu> wrote:
>
> On Wed, Dec 12, 2007 at 03:36:56PM -0500, Martin Goldman wrote:
> > I did a few throughput tests using iperf between servers and
> consistently
> > got a result of about 725Mbps -- it's not 1000, but it's a lot more than
> 320
> > at least. Is that a reasonable test?
>
> The test is reasonable, but the result very much indicates a 32-bit, 33
> MHz
> PCI NIC ! The 320 Mbps average limit would be for the proxy, because the
> same data passes twice on the bus, and if it's a shared PCI bus for both
> NICs,
> you divide the maximum performance by two because the bus is already
> saturated.
>
> PCI-X NICs normally give very good results. Most often, on-board NICs are
> connected to 32-bit, 100 MHz PCI busses, which sets the limit to a
> theorical
> 3 Gbps in+out but practically just enough to saturate in+out at 1 Gbps in
> each direction.
>
> Recent machines using PCI-Express achieve much higher performance because
> the PCIe bus is point-to-point, and the lowest speed (x1) moves 2 Gbps in
> each direction, fairly enough for a 1 Gbps NIC. But sometimes you can
> enounter buggy chips such as some early Marvell 88e8053 with which I had
> a *lot* of problems. The best GigE NIC I found to date was the intel Pro
> 1000,
> both in PCI-X and PCIe flavours.
>
> > hdparm is reporting a "Timing buffered disk reads" value of about
> 50MB/sec
> > for my disks (which are SATA). So it seems like it might be reasonable
> for
> > the individual web servers to max out at 40-something MB/sec.
>
> Yes, possibly.
>
> > What I don't quite understand is, is haproxy actually hitting the disk?
>
> Not at all. It does not even know how to access any file on the disk once
> it has read its configuration and written its pid file.
>
> > If not, it would
> > seem the cluster should be able to handle more requests than the web
> servers
> > individually. Does that make sense?
>
> Yes, that makes a lot of sense. So this would mean that *at least* the
> machine
> you used for the load balancer is limited, because when the traffic passes
> through it, the performance is halved.
>
> You may want to check your BIOS. While I don't see any valid reason for
> it, it
> might be possible that you can set the bus frequencies there, or that you
> have
> some sort of "legacy mode" for PCI busses which would explain the lower
> performance.
>
> Regards,
> Willy
>
>
Received on 2007/12/12 23:32