haproxy 1.3.14.1 on FreeBSD 7 -- a couple of issues

From: Hugo Silva <hugo#barafranca.com>
Date: Mon, 21 Jan 2008 21:45:08 +0000

Hi list,

I've setup a couple of load balancers in production recently, and I'm running into some problems while using haproxy 1.3.14.1 on freebsd 7 (the same thing happened with freebsd 6 and haproxy 1.2.X)

The first issue is that should I enable the firewall (pf), for some reason that I have not yet determined, *lots* of packets will be considered invalid by the firewall (both on haproxy's firewall and the webservers), causing a tremendous amount of connection errors on haproxy, and all sorts of strange problems on the PHP application (ie clicking any link and the page simply not loading).

It is not a state limit problem:

states        hard limit   262144
src-nodes     hard limit    10000
frags         hard limit     4096
tables        hard limit     1000

table-entries hard limit 200000

State Table                          Total             Rate
  current entries                     2312
  searches                       318798877          215.8/s

I have turned off the firewall on all but one webserver, and also on the load balancers, as no packets are being logged as dropped, yet the 'state-mismatch' counter on the firewalls.

As an example, here's one such counter from one of the webservers (who has the firewall disabled now)

  state-mismatch                     240759            0.1/s

Compare with the database server, which gets at least as many connections as all the webservers together:

  state-mismatch                      1523            0.0/s

Regarding this issue, any clarification on all possible ways haproxy would increment errors (stats page) on:

• Req
• Resp
• Conn

would be welcomed. Even on the load balancers and the webservers which have the firewall turned off, I am seeing lots (~3000 connections errors, ~10000 request errors in 7 hours) of errors. The one webserver that still has the firewall enabled has a lot more errors than all the others.

I know the description is a bit vague, I am hoping someone else using *BSD and pf on an environment that gets lots of connections/s has also seen the same.

The second issue is:

Connect() failed for server backend/server: local address already in use.

I have mailed Willy about this and he confirms he has also seen this on other BSDs, and also offered an explanation as to why it happens.

I've only seen this issue with haproxy, which leads me to believe there's probably a "BSD way" of doing this. I know haproxy gets a lot more exposure on linux, which is why I am posting this issue here, maybe someone can offer some insight and/or a fix.

This error tends to happen very frequently when there are lots of connections (but for instance it doesn't happen with varnish if the static server goes down and connections pile up - I am mentioning varnish because it does basically the same as haproxy in terms of connections - it contacts backend(s)).

Thanks in advance!

Best regards,

Hugo Received on 2008/01/21 22:45