Re: haproxy alleatory fails on config reload

Hi Pablo,

On Fri, May 30, 2008 at 02:22:02PM +0200, Pablo Escobar wrote:
> El Friday 30 May 2008 06:43:34 Willy Tarreau escribió:
> > On Fri, May 30, 2008 at 01:22:56AM +0200, Pablo Escobar wrote:
> > > Hi Krzysztof and Willy,
> > >
> > > I have compiled haproxy 1.3.15.1 for x86_64
> > >
> > > I found with dmesg some of this errors:
> > > ip_conntrack: table full, dropping packet.
> > >
> > > So I tougth I had found the problem but I was wrong. I doubled the
> > > ip_conntrack_max value and the dmesg error dissapeared but I still get
> > > the same problem. Around 5 reloads ok and then a wrong reload wich takes
> > > all my websites offline for EXACTLY 1 MIN.
> > >
> > > connecting to port 81 I see every backend up. connecting on port 80 I get
> > > 503 error on every backend. exactly when the stats web arrives to "uptime
> > > 1:01min" I get every backend up on port 80. I am sure the 1min is not
> > > random. I have tried it 2 times with the same result.
> >
> > 1 minute might be the time needed to expire old sessions from your
> > conntrack. You can try to reduce ip_conntrack_tcp_timeout_time_wait in
> > /proc/sys/net/ipv4/netfilter to see if this has any effect.
> >
> I have a value of 120 on ip_conntrack_tcp_timeout_time_wait. ¿would be safe to
> change it to 60?

Yes. Don't go below 20-30 though, otherwise you'll get some erroneous DROP logs due to late retransmits.

> also I have found a 60 value on "ip_conntrack_tcp_timeout_syn_recv"
>
> [root#haproxy]$>
> cat /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_syn_recv
> 60
>
> ¿maybe the problem is related to this?

I don't think so. It's related to incoming connections which are not ACKed. Typically needed in case of synflood.
>
> > Also, it is possible that your conntrack is buggy and does not initiate a
> > new session on a SYN reusing a same source port (once your source ports
> > wrap around).
> >
> > You should *really* unload conntrack to see if it makes any difference.
>
> Sorry for my ignorance but........¿If I unload this module could it make my
> haproxy machine stop working? This is the main machine on my web cluster
> (doing reverse proxy on every webserver) so if it stops working everything
> goes down. ¿is safe to remove the conntrack module? Right now I can´t do it
> because this hours is when I have most of the web traffic but if anyone can
> confirm that is safe to unload the module I will do some testing tonight.

The conntrack module is needed ONLY if you :

• have netfilter configured on the machine to do stateful firewalling
• are using NAT or transparent proxying through the TPROXY patch.

Regards,
Willy Received on 2008/05/30 14:55