Re: High-volume HAProxy deployment tips?

Hi Krzysztof,

On Fri, Dec 19, 2008 at 08:29:00AM +0100, Krzysztof Oledzki wrote:
> On Fri, 19 Dec 2008, Willy Tarreau wrote:
>
> >On Fri, Dec 19, 2008 at 12:13:18AM +0100, Benoit wrote:
> >>Willy Tarreau a écrit :
> >>>>As for us, we had to double the haproxy boxes to get past the 40Mbps of
> >>>>http traffic ....
> >>>>
> >>>
> >>>what type of traffic do you have and what type of network cards do you
> >>>have ?
> >>>The smallest machines I have (fanless AMD Geode 500 MHz, 2.5W of average
> >>>power
> >>>consumption) already supports 100 Mbps and up to 2000 connections/s with
> >>>a
> >>>dirt cheap VIA Rhine NIC. So there's definitely a problem with your
> >>>setup !
> >>>
> >>
> >>Server is an HP blade, with 4 Broadcom Corporation NetXtreme II BCM5708S
> >>port, bonded two by two.
> >>and it's a bi dual core Xeon 5150#2.66Ghz cpu with 4Gb of memory system
> >>dedicated to haproxy.
> >>
> >>The cpu load is low, very very low, however when at 40 or 50Mbps
> >>connection time start to go up to the roof
> >>( up to 3/4s sometimes, instead of 180ms of total processing time), and
> >>activating the backup server fixed this
> >>so it isn't of problem with the backends
> >
> >OK I understand better now. At a customer's, we've been using HP blades
> >with BNX2 NICs too, and we disabled them. The packet loss is terrible on
> >those crappy NICs.
>
> I can't confirm this. I have two BCM5708 NICs currently running on
> 2.6.26.8:
>
> # lspci|grep Eth
> 03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708
> Gigabit Ethernet (rev 12)
> 07:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708
> Gigabit Ethernet (rev 12)
>
> I have never experienced performance issues with these NICs and from my
> POV bnx2/tg3 NICs are much better than for example e1000 ones as there is
> no IPMI+VLAN problem with Broadcoms.

That's getting interesting! What machine is this ? Maybe those are only the NICs in HP blades which are crappy ? Or maybe just their firmware ? Also, what bandwidth are you pushing out ? I started seeing massive drops at about the same bitrates as Benoit (about 40/50 Mbps).

> Maybe this is a firmware issue? Have you tried upgrading it? Mine is
> 4.4.1:
>
> # ethtool -i eth0
> driver: bnx2
> version: 1.7.5
> firmware-version: 4.4.1 ipms 1.6.0
> bus-info: 0000:03:00.0
>
> AFAIK currently this is the latest available firmware.

Clearly those are not the same versions I have here :

1. machine on RHEL5 2.6.18-53:

# /sbin/ethtool -i eth2
driver: bnx2
version: 1.5.11
firmware-version: 1.9.6
bus-info: 0000:05:00.0

2) machine on RHEL5 updated to 2.6.18-92.1.17 driver: bnx2
version: 1.6.9
firmware-version: 1.9.6
bus-info: 0000:03:00.0

I'm pretty sure I remember seeing a different firmware when booted on 2.6.25, which would imply the firmware is provided with the driver. Are you sure you can upgrade it yourself ? If so, I'm interested in the procedure, as I could suggest my customer to try it on one machine.

> >Please run "ethtool -s eth0 | grep discard",
>
> It should be "ethtool -S eth0 | grep discard" (grand S)

Indeed, thanks for correcting this.

Regards,
Willy Received on 2008/12/19 09:51