Re: haproxy & linux firewall (netfilter)

On Sat, Oct 20, 2007 at 04:03:38PM +0200, Krzysztof Oledzki wrote:
> >I assume that you put *one* source address per server entry.
>
> No, with each new connection haproxy gets next source address from above
> list. There are no source address defined per server.

OK, initially I didn't understand that you changed the code :-)

> >What is very strange is that linux uses random increments, so your ISNs
> >should not wrap in a matter of a few seconds.
>
> Good point. I need to investigate this.

netcat is very convenient for such tests. It's easy to bind it to a source port for consecutive tests while you run tcpdump in the background :  

  $ echo bla | nc -p 1234 192.168.1.2 80   $ echo bla | nc -p 1234 192.168.1.2 80

Also, please try this with tcp_timestamps enabled and disabled to see if it changes anything.

> >Maybe (I said *maybe*) linux completely randomizes the ISNs when timestamps
> >are enabled ? You may want to retry with timestamps disabled. Anyway, I
> >think it would be time to implement PAWS in netfilter :-/
>
> I agree but I do not feel brave enough to do it myself. :|

I can understand! At least if we can find a normal situation which triggers the problem very often and which fairly requires PAWS, we can check with the netfilter core team if it would not be a good opportunity to implement it.

> >Since I've already implemented it in another program, I know that when
> >you do this, you also need to manage the source ports yourself.
>
> Only when you want to implement it this way. In my solution it simply was:
>
> +struct source_addr_pool {
> + struct sockaddr_in addr;
> + struct source_addr_pool *next;
> +};
> +
> (...)
>
> struct proxy {
> (...)
> - struct sockaddr_in source_addr; /* the address to which we
> want to bind for connect() */
> + struct source_addr_pool *source_addr; /* pool of addresses to
> which we want to bind for connect() */
> + struct source_addr_pool *curr_sa; /* the address to which we
> want to bind for connect() */
> }
>
> (...)
> - if (bind(fd, (struct sockaddr *)&s->be->source_addr,
> sizeof(s->be->source_addr)) == -1) {
> + struct source_addr_pool *sap = s->be->curr_sa;
> +
> + s->be->curr_sa=sap->next?sap->next:s->be->source_addr;
> + if (bind(fd, (struct sockaddr *)&(sap->addr),
> sizeof(sap->addr)) == -1) {
>
> Of course this is only a short example, there are more places requiring
> changes.

The problem when you only assign addresses is that linux has only *one* list of free ports, regardless of the addresses. That means that if you do the following :

   bind(fd, {addr=192.168.1.1, port=0});    bind(fd, {addr=192.168.1.2, port=0});

The second port will never be the same as the first one. This results in absolutely no increase in the mixing of IP/ports, which is why you still get collisions. Check the number of TIME_WAITs on your system, it should never go beyond the number of source ports.

> <CUT>
> >I'm not really sure this is interesting to do. In your case, the bug is
> >between linux and the firewall which runs on it (netfilter). It's not
> >expected that if you enable timestamps exactly to fix this problem, it
> >makes the problem worse !
>
> OK. I am also not sure and that is why I have never pushed this.

OK.

Regards,
Willy Received on 2007/10/20 17:00