Re: long ip acl's

On Thu, Feb 25, 2010 at 06:03:14PM -0800, David Birdsong wrote:
> 2010/2/25 XANi <xani666#gmail.com>
> >
> > Hi
> > Dnia 2010-02-25, czw o godzinie 15:35 -0800, David Birdsong pisze:
> >
> > On Thu, Feb 25, 2010 at 12:48 AM, Willy Tarreau <w#1wt.eu> wrote:
> > > Hi David,
> > >
> > > On Wed, Feb 24, 2010 at 06:06:26PM -0800, David Birdsong wrote:
> > >> I'm autogenerating haproxy configs on some of our front ends and
> > >> appending a growing set of IP addresses that we'll ban.  Does this
> > >> scale well in haproxy?  Can I expect performance to drop as the list
> > >> grows and grows or is this implemented in a way that scales pretty
> > >> horizontally?
> > >
> > > Yes the performance will drop but not *that* much, because IP ACLs
> > > check is quite fast. Just put as many IPs per line as you can.
> > >
> > > I have plans to load IP ranges from a file and to perform dichotomic
> > > search on them (which will be even faster than tree search due to
> > > lower memory footprint). It would make it possible to load millions
> > > of IP addresses without a noticeable performance degradation. It's
> > > just not there yet.
> > >
> > > I also plan to add ACL matches for stickiness tables. That will allow
> > > us to check using ACLs if an address was already added to a table. We
> > > first have to relax the conditions in which an address can be inserted.
> > >
> > > How many IP addresses do you intend to load, and how many requests
> > > per second do you estimate ?
> > right now there are 20 or so, but this i've automated their addition
> > to the config file and was wondering if this was something i could
> > forget about -clearly not.
> >
> > we had a bad referrer list that nobody paid attention to and it grew
> > to like 4k. our home grown lighttpd module was killing lighttpd's
> > performance comparing all requests against a 4,000 referrer list.
> >
> > these are uploads(posts) so rate is quite low. less than 100/sec.
> >
> > U might try to use iptables + ipset instead. Tho according to manual there is 65535 IP limit per set. Also no need to restart haproxy for adding new IP's
> >
> yes, i've considered iptables would actually be simpler.

iptables + ipset is a fast solution. You can also build a tree of iptables rules which with a limited depth, which will always be fast to evaluate. Just be careful not to enable ip_conntrack (well at 100/s it will not be noticeable anyway).

Another solution to blacklist IP addresses is to add them as blackholes. Thus you benefit from the system's routing tables which are parsed very quickly and no incoming connection is created when its source is blackholed.

At least one site I know has had to use this recently because of a DDoS, and it worked pretty well.

Regards,
Willy Received on 2010/02/26 07:54