Re: High-volume HAProxy deployment tips?

On Fri, Dec 19, 2008 at 10:40:42AM +0100, Krzysztof Oledzki wrote:
> On Fri, 19 Dec 2008, Willy Tarreau wrote:
>
> >Hi Krzysztof,
> >
> >On Fri, Dec 19, 2008 at 08:29:00AM +0100, Krzysztof Oledzki wrote:
> >>On Fri, 19 Dec 2008, Willy Tarreau wrote:
> >>
> >>>On Fri, Dec 19, 2008 at 12:13:18AM +0100, Benoit wrote:
> >>>>Willy Tarreau a écrit :
> >>>>>>As for us, we had to double the haproxy boxes to get past the 40Mbps
> >>>>>>of
> >>>>>>http traffic ....
> >>>>>>
> >>>>>
> >>>>>what type of traffic do you have and what type of network cards do you
> >>>>>have ?
> >>>>>The smallest machines I have (fanless AMD Geode 500 MHz, 2.5W of
> >>>>>average
> >>>>>power
> >>>>>consumption) already supports 100 Mbps and up to 2000 connections/s
> >>>>>with
> >>>>>a
> >>>>>dirt cheap VIA Rhine NIC. So there's definitely a problem with your
> >>>>>setup !
> >>>>>
> >>>>
> >>>>Server is an HP blade, with 4 Broadcom Corporation NetXtreme II BCM5708S
> >>>>port, bonded two by two.
> >>>>and it's a bi dual core Xeon 5150#2.66Ghz cpu with 4Gb of memory system
> >>>>dedicated to haproxy.
> >>>>
> >>>>The cpu load is low, very very low, however when at 40 or 50Mbps
> >>>>connection time start to go up to the roof
> >>>>( up to 3/4s sometimes, instead of 180ms of total processing time), and
> >>>>activating the backup server fixed this
> >>>>so it isn't of problem with the backends
> >>>
> >>>OK I understand better now. At a customer's, we've been using HP blades
> >>>with BNX2 NICs too, and we disabled them. The packet loss is terrible on
> >>>those crappy NICs.
> >>
> >>I can't confirm this. I have two BCM5708 NICs currently running on
> >>2.6.26.8:
> >>
> >># lspci|grep Eth
> >>03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708
> >>Gigabit Ethernet (rev 12)
> >>07:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708
> >>Gigabit Ethernet (rev 12)
> >>
> >>I have never experienced performance issues with these NICs and from my
> >>POV bnx2/tg3 NICs are much better than for example e1000 ones as there is
> >>no IPMI+VLAN problem with Broadcoms.
> >
> >That's getting interesting! What machine is this ?
>
> Dell PowerEdge 1950
>
> >Maybe those are only the
> >NICs in HP blades which are crappy ?
>
> Maybe.
>
> >Or maybe just their firmware ? Also,
> >what bandwidth are you pushing out ?
>
> My latest peak is at about 400Mb/s
>
> >I started seeing massive drops at about
> >the same bitrates as Benoit (about 40/50 Mbps).
>
> 40/50 is indeed very poor!
>
> >>Maybe this is a firmware issue? Have you tried upgrading it? Mine is
> >>4.4.1:
> >>
> >># ethtool -i eth0
> >>driver: bnx2
> >>version: 1.7.5
> >>firmware-version: 4.4.1 ipms 1.6.0
> >>bus-info: 0000:03:00.0
> >>
> >>AFAIK currently this is the latest available firmware.
> >
> >Clearly those are not the same versions I have here :
> >
> >1) machine on RHEL5 2.6.18-53:
> >
> ># /sbin/ethtool -i eth2
> >driver: bnx2
> >version: 1.5.11
> >firmware-version: 1.9.6
> >bus-info: 0000:05:00.0
> >
> >2) machine on RHEL5 updated to 2.6.18-92.1.17
> >driver: bnx2
> >version: 1.6.9
> >firmware-version: 1.9.6
> >bus-info: 0000:03:00.0
> >
> >I'm pretty sure I remember seeing a different firmware when booted on
> >2.6.25, which would imply the firmware is provided with the driver.
>
> AFAIK there are two firmwares: one stored at NIC and one uploaded by the
> driver. However, I'm not sure how and if they are correlated.
>
> >Are you sure you can upgrade it yourself ?
>
> Yes, I'm 100% sure. I have done it several times on both IBM and Dell
> servers.
>
> >If so, I'm interested in the
> >procedure, as I could suggest my customer to try it on one machine.
>
> The firmware should be available on vendor's website. For example:
>
> http://support.euro.dell.com/support/downloads/format.aspx?cs=RC1078552&l=pl&s=pad&deviceid=10264&libid=5&releaseid=R197246
> https://www-304.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-5070004&brandind=5000020

Thanks very much Krzysztof, I'll forward this information here.

Cheers,
Willy Received on 2008/12/19 10:46