Re: TProxy 4

On 8/5/08, Willy Tarreau <w#1wt.eu> wrote:
>
> On Tue, Aug 05, 2008 at 12:21:09PM -0600, Caleb Anthony wrote:
> > > I really need to try those patches again :-)
> >
> >
> > I know what you mean. I'm not toally sure if this is a TProxy problem or
> > HAProxy problem. ;-)
>
> My real problem is that we still have no success report of Tproxy4, mostly
> by lack of necessity and tests.

Well I have something to report. I was able to get this (partly) working. Here are my steps:

Patched and recompiled my kernel with the TProxy patches. Patched and recompiled iptables with the TProxy patches. Ran the following netfilter / routing commands:

ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT

Added:
source 0.0.0.0 usesrc client
to my haproxy.cfg

Thats all that needs to be done on the HAProxy side to allow non-local binding.

> Attached is an strace of HAproxy, and a tcpdump of the traffic. In the
> > tcpdump 10.193.67.9 is the client, 10.193.67.17 is the server running
> > HAProxy, and 10.35.154.142 is the backend web server. I noticed that in
> the
> > tcpdump that the client to HAproxy looks ok, but the communication from
> > HAproxy to the backend server isn't working. HAProxy sends out 8 requests
> > with the spoofed cleint ip (10.193.67.9),
>
> OK this is excellent, because it proves that the foreign binding works.
>
> > but it seems that the backend web
> > server dosen't reply, or is trying to reply to the real client, and not
> > HAProxy.
>
> I'd bet it's the backend server which doesn't have its default gateway
> set to route through haproxy. I'm almost 99% sure! It absolutely needs
> to route the return traffic through haproxy so that the traffic is
> transformed again.

I tried configuring the default gateway on the backend web server to point to the HAProxy server, but that didn't help. So I ran Wireshark on the backend web server and found out that the backed web server is getting the request ok, but it's still sending the replies directly to the client and not the HAProxy server. I noticed this because of the destination MAC address it was trying to send to. So I added a static ARP entry on the backend web server using the client's IP address and the HAProxy servers MAC address, and that got it to work.

I verified it was working correctly by looking in the logs on the backend web server, and the logs did show the client's IP address and not the HAProxy servers address for each request.

This is great and all, but I'm sure this isn't how it's supposed to work. Other than setting MAC addresses by hand on each backend web server, how am I supposed to have the reply traffic get routed back through the HAProxy? It's almost as if the HAProxy would have to be inline with all traffic on the network (essentially a router), which isn't possible where I work.

So it looks like I'm out of luck, however you now have some proof that your TProxy code works, which is great.

I do have one additional question though. If HAProxy supported HTTP Keep-Alive I would be able to accomplish logging the correct client IP address by using the X-Forwarded-For header. I've seen in other postings that this is in the road map. Do you have any idea when that might be? Also, I'm open to any testing that you need done with it. Even if it means testing pre-alpha code. ;-)

Thanks for all your help.

> That causes HAProxy to return a 503 to the client. That tells me
> > that there is a problem with TProxy or my iptables config.
>
> Till there, the forward traffic looks OK on your haproxy box. You need
> to sniff on the backend server if you can't get more info out of the
> routing tables. It is also possible that you have some iptables enabled
> on the backend server, which blocks any non-local traffic ;-)
>
> > I've seen some
> > information on the web about setting up a bridge when using TProxy so
> that
> > the routing will be correct. As far as you know, is that still the case
> with
> > TProxy 4.1 and HAProxy?
>
> I have no idea. This might impact backwards traffic, but let's fix one
> problem at a time. When we get responses routed back through haproxy,
> if the traffic is not processed, we'll have to investigate further.
>
> > Great, I was trying to go as minimal as possible so that I could avoid
> any
> > potential problems with my config.
>
> That's a good choice.
>
> Regards,
> Willy
>
>
Received on 2008/08/06 20:26