Re: high connection rates

Sorry for the radio silence. I took some time off to go skiing.

On Sun, Jan 31, 2010 at 2:54 PM, Willy Tarreau <w#1wt.eu> wrote:
> On Sun, Jan 31, 2010 at 01:21:07PM -0800, David Birdsong wrote:
>> > Do you use TCP splicing ? You need a recent kernel (> 2.6.27.X, with
>> > X at least up to date with your distro). Then build haproxy with
>> > USE_LINUX_SPLICE=1.
>>
>> i just tried this and something is all wrong.  i had to disable after
>> traffic plummeted.
>
> That's not expected at all.

yes, i was surprised.

>
>> traffic comes to this tier if health checks pass, so loss of traffic
>> could be health checks timing on the client.  i just couldn't verify
>> that health checks were failing, so i dont know yet why traffic went
>> down.
>
> Well, splicing does not affect health checks, just data forwarding.
> Also, if it fails to allocate a pipe to move the data, it falls back
> to normal recv/send.
>
>> i dot have my tcp stats:
>> tcpabortonclose skyrocketed from 0/sec to 10k/sec
>> TWKilled went from 10k/sec/ to 30k/sec
>> many stats went down too, but traffic went from 320mbps to 80 mbps
>>
>> wracking my brain to figure out what was so bad about this test.  cpu
>> was nowhere near pegged, plenty of idle.
>>
>> i tested this on one server out of 3.  they are load balanced by a
>> server iron by least connections, the server iron dsr's to them so
>> they talk directly to end clients.  traffic dropped on all machines.
>
> It's unclear to me what machines you're talking about when you say
> "all machines", as I understand that you have 3 LBs behind your
> serveriron.

all of these components in the path of traffic just confused matters, so i simplified the path for purposes of testing.

but then i just straced haproxy while traffic was plummetting, i should have done this a long time ago:

splice(0x14, 0, 0xf, 0, 0xffffffffffffffff, 0x3) = -1 EINVAL (Invalid argument)

# ldd /usr/local/sbin/haproxy
	linux-vdso.so.1 =>  (0x00007fff017ff000)
	libc.so.6 => /lib64/libc.so.6 (0x00007ffe705dd000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ffe7094f000)

this is built from: haproxy-1.4-dev4

>> trying to gather more info..i may need to try again.
>
> before that, did you find any error or abnormal conditions in the
> logs ? Also, what haproxy version are you using ?
>
>> > You're saying you have two NICs, care to indicate a bit more about
>> > that ? Are they used in load balancing or one for input and the other
>> > one for output ?
>>
>> only 1 nic is active, if we can push line rate or close to it stably,
>> we will use ethernet bonding to allow failover.
>
> OK.
>
>> though now that you mention pushing a GigE on each, maybe separate
>> links to separate routers with 2 independent processes would be more
>> fault tolerant.
>
> Then you need your routers to provide that fault tolerance for you.
> Otherwise, using keepalived, it's possible to share multiple IPs
> between multiple machines and always have them all active at the
> same time.
>
> Regards,
> Willy
>
>
Received on 2010/02/12 02:52