Re: Running haproxy on the cluster nodes

You know, you're right, but I just realized there's another thing I failed to mention or even consider -- my cluster is actually listening on the LAN interface -- this works because the machine I'm running apachebench from is also on the LAN. So the internal NIC and the internal network itself is having to deal with that data twice -- once from the web server into the proxy, and then from the proxy back to the client. Maybe that's where the other half of my throughput is going?... hrmm...

On 12/12/07, Dan Zubey <dzubey#odysseyware.com> wrote:
>
> You're connected at 100Mbps and not 1Gbps on eth0.
>
> My two cents.
>
> -Dan
>
> Martin Goldman wrote:
> > These are hosted dedicated servers, so I don't have much info on the
> > hardware. But from looking at dmesg, it looks like I may have one Intel
> > PCI GigE card (which is connected to the LAN) and and a Broadcom/Tigon3
> > PCI Express card (which is connected to the Internet):
> >
> > ----------
> > e1000: 0000:08:02.0: e1000_probe: (PCI:33MHz:32-bit) [MAC_address]
> > e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
> > e1000: eth1: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex
> >
> > eth0: Tigon3 [partno(BCM95751) rev 4201 PHY(5750)] (PCI Express)
> > 10/100/1000BaseT Ethernet [MAC address]
> > tg3: eth0: Link is up at 100 Mbps, full duplex.
> > ----------
> >
> > I googled the Intel card (model 82541PI according to lspci) and it is
> > indeed a PCI 33/66MHz card, and the Broadcom (model BCM5751) is indeed
> > a PCI Express card. Would having one of each type of card like this
> > allow for the type of bus saturation that you describe below? I'm not
> > sure if the hosting company will do it, but should I look into having
> > the PCI card swapped out for another PCI Express?
> >
> > Many thanks for all the explanation; as you can obviously tell, I'm not
> > much of a hardware guy :)
> >
> > -Martin
> >
> > On 12/12/07, *Willy Tarreau* <w#1wt.eu <mailto:w#1wt.eu>> wrote:
> >
> > On Wed, Dec 12, 2007 at 03:36:56PM -0500, Martin Goldman wrote:
> > > I did a few throughput tests using iperf between servers and
> > consistently
> > > got a result of about 725Mbps -- it's not 1000, but it's a lot
> > more than 320
> > > at least. Is that a reasonable test?
> >
> > The test is reasonable, but the result very much indicates a 32-bit,
> > 33 MHz
> > PCI NIC ! The 320 Mbps average limit would be for the proxy, because
> the
> > same data passes twice on the bus, and if it's a shared PCI bus for
> > both NICs,
> > you divide the maximum performance by two because the bus is already
> > saturated.
> >
> > PCI-X NICs normally give very good results. Most often, on-board
> > NICs are
> > connected to 32-bit, 100 MHz PCI busses, which sets the limit to a
> > theorical
> > 3 Gbps in+out but practically just enough to saturate in+out at 1
> > Gbps in
> > each direction.
> >
> > Recent machines using PCI-Express achieve much higher performance
> > because
> > the PCIe bus is point-to-point, and the lowest speed (x1) moves 2
> > Gbps in
> > each direction, fairly enough for a 1 Gbps NIC. But sometimes you
> can
> > enounter buggy chips such as some early Marvell 88e8053 with which I
> had
> > a *lot* of problems. The best GigE NIC I found to date was the intel
> > Pro 1000,
> > both in PCI-X and PCIe flavours.
> >
> > > hdparm is reporting a "Timing buffered disk reads" value of about
> > 50MB/sec
> > > for my disks (which are SATA). So it seems like it might be
> > reasonable for
> > > the individual web servers to max out at 40-something MB/sec.
> >
> > Yes, possibly.
> >
> > > What I don't quite understand is, is haproxy actually hitting the
> > disk?
> >
> > Not at all. It does not even know how to access any file on the disk
> > once
> > it has read its configuration and written its pid file.
> >
> > > If not, it would
> > > seem the cluster should be able to handle more requests than the
> > web servers
> > > individually. Does that make sense?
> >
> > Yes, that makes a lot of sense. So this would mean that *at least*
> > the machine
> > you used for the load balancer is limited, because when the traffic
> > passes
> > through it, the performance is halved.
> >
> > You may want to check your BIOS. While I don't see any valid reason
> > for it, it
> > might be possible that you can set the bus frequencies there, or
> > that you have
> > some sort of "legacy mode" for PCI busses which would explain the
> lower
> > performance.
> >
> > Regards,
> > Willy
> >
> >
>
Received on 2007/12/13 00:04