Re: haproxy 1.3.14.1 on FreeBSD 7 -- a couple of issues

Hugo Silva wrote:
> Hi list,
>
> I've setup a couple of load balancers in production recently, and I'm
> running into some problems while using haproxy 1.3.14.1 on freebsd 7
> (the same thing happened with freebsd 6 and haproxy 1.2.X)
>
> The first issue is that should I enable the firewall (pf), for some
> reason that I have not yet determined, *lots* of packets will be
> considered invalid by the firewall (both on haproxy's firewall and the
> webservers), causing a tremendous amount of connection errors on
> haproxy, and all sorts of strange problems on the PHP application (ie
> clicking any link and the page simply not loading).
>
> It is not a state limit problem:
> states hard limit 262144
> src-nodes hard limit 10000
> frags hard limit 4096
> tables hard limit 1000
> table-entries hard limit 200000
>
> State Table Total Rate
> current entries 2312
> searches 318798877 215.8/s
>
> I have turned off the firewall on all but one webserver, and also on
> the load balancers, as no packets are being logged as dropped, yet the
> 'state-mismatch' counter on the firewalls.
That didn't make much sense, dinner and writing emails don't go well together :-)

I meant to say that no packets are being logged as dropped (with the firewall enabled), yet the 'state-mismatch' counter keeps increasing, and I am logging *everything* the firewall drops (not just SYN)

>
>
> As an example, here's one such counter from one of the webservers (who
> has the firewall disabled now)
> state-mismatch 240759 0.1/s
>
>
> Compare with the database server, which gets at least as many
> connections as all the webservers together:
> state-mismatch 1523 0.0/s
>
>
> Regarding this issue, any clarification on all possible ways haproxy
> would increment errors (stats page) on:
>
> - Req
> - Resp
> - Conn
>
> would be welcomed. Even on the load balancers and the webservers which
> have the firewall turned off, I am seeing lots (~3000 connections
> errors, ~10000 request errors in 7 hours) of errors. The one
> webserver that still has the firewall enabled has a lot more errors
> than all the others.
>
> I know the description is a bit vague, I am hoping someone else using
> *BSD and pf on an environment that gets lots of connections/s has also
> seen the same.
>
>
>
> The second issue is:
>
> Connect() failed for server backend/server: local address already in use.
>
> I have mailed Willy about this and he confirms he has also seen this
> on other BSDs, and also offered an explanation as to why it happens.
>
> I've only seen this issue with haproxy, which leads me to believe
> there's probably a "BSD way" of doing this. I know haproxy gets a lot
> more exposure on linux, which is why I am posting this issue here,
> maybe someone can offer some insight and/or a fix.
>
> This error tends to happen very frequently when there are lots of
> connections (but for instance it doesn't happen with varnish if the
> static server goes down and connections pile up - I am mentioning
> varnish because it does basically the same as haproxy in terms of
> connections - it contacts backend(s)).
>
> Thanks in advance!
>
> Best regards,
>
> Hugo
>
>
Received on 2008/01/21 23:03