Re: TProxy 4

On 8/4/08, Willy Tarreau <w#1wt.eu> wrote:
>
> Hello,
>
> On Mon, Aug 04, 2008 at 10:39:37AM -0600, Caleb Anthony wrote:
> > Hello all.
> >
> > For the past few days I have been trying to setup HAProxy 1.3.15.2 with
> > TProxy 4.1 so that my backend servers will see the original clients IP
> > address. This is mostly for logging purposes, and enabling
> X-Forwarded-For
> > isn't an option in my environment because I need HTTP Keep-Alive enabled
> on
> > my servers (turning it off with option httpclose breaks Integrated
> Windows
> > Authentication on IIS).
> >
> > Here is what I have so far:
> >
> > I patched my kernel with the latest patches from BalaBit. I originally
> tried
> > the tproxy-kernel-2.6.24-20080602-165651-1212418611.tar.bz2 patch with
> > kernel 2.6.24.5, and I also tried the
> > tproxy-kernel-2.6.25-20080519-165031-1211208631.tar.bz2 patch with kernel
> > 2.6.25.
>
> I really need to try those patches again :-)

I know what you mean. I'm not toally sure if this is a TProxy problem or HAProxy problem. ;-)

> I re-compiled HAproxy with USE_LINUX_TPROXY=1
> >
> > Lastly I patched and recompiled iptables 1.4.0 with the
> > tproxy-kernel-2.6.25-20080519-165031-1211208631.tar.bz2 patch. I'm not
> > entirely sure if this is really needed, other than to load the proper
> kernel
> > modules.
> >
> > Once all the patching was done, I tried loading the TProxy modules
> manually
> > with modprobe, and I also tried setting some iptables rules that also
> loaded
> > the kernel modules.
> >
> > I know the modules are loaded and working because /var/log/messages
> displays
> > the following after the modules are loaded:
> > kernel: NF_TPROXY: Transparent proxy support initialized, version 4.1.0
> > kernel: NF_TPROXY: Copyright (c) 2006-2007 BalaBit IT Ltd.
> >
> > I also set the following sysctl values (not sure if these are needed
> > either):
> > echo 1 > /proc/sys/net/ipv4/ip_forward
> > echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
> >
> > Also I added the following line to my haproxy.cfg:
> > source 0.0.0.0 usesrc clientip
>
> With tproxy v2, it was not possible to use source 0.0.0.0. Could
> you try to put your LB's address here ? You need to set up the
> address from which you can reach the backend server. Tproxy needed
> this in order to create its internal NAT tables.

I tried that, and no change.

> When a client connects, I can see the connection in the HAProxy debug
> > output, but the browser just hangs for a little bit, then eventually
> HAProxy
> > returns a HTTP 503 No server is available to handle this request.
> >
> > Here is the debug output from HAProxy:
> > Available polling systems :
> > sepoll : pref=400, test result OK
> > epoll : pref=300, test result OK
> > poll : pref=200, test result OK
> > select : pref=150, test result OK
> > Total: 4 (4 usable), will use sepoll.
> > Using sepoll() as the polling mechanism.
> > 00000000:Frontend.accept(0004)=0005 from [x.x.x.x:4851]
> > 00000000:Frontend.clireq[0005:ffff]: GET /test/ HTTP/1.1
> > 00000000:Frontend.clihdr[0005:ffff]: Accept: */*
> > 00000000:Frontend.clihdr[0005:ffff]: Accept-Language: en-us
> > 00000000:Frontend.clihdr[0005:ffff]: Accept-Encoding: gzip, deflate
> > 00000000:Frontend.clihdr[0005:ffff]: User-Agent: Mozilla/4.0 (compatible;
> > MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322;
> > InfoPath.1; .NET CLR 2.0.50727)
> > 00000000:Frontend.clihdr[0005:ffff]: Host: xxxxx
> > 00000000:Frontend.clihdr[0005:ffff]: Connection: Keep-Alive
> > 00000000:Frontend.clihdr[0005:ffff]: Cache-Control: no-cache
> > 00000000:Default.srvcls[0005:0006]
> > 00000000:Default.srvcls[0005:0006]
> > 00000000:Default.srvcls[0005:0006]
> > 00000000:Default.clicls[0005:0006]
> > 00000000:Default.srvcls[0005:0006]
> > 00000000:Default.closed[0005:0006]
> >
> > Removing "source 0.0.0.0 usesrc clientip" from my config makes HAProxy
> work
> > fine (except with no transparent proxy functionality of course).
>
> It would be nice if you could bring an strace output of the process during
> such a test, as well as a tcpdump trace of the interface with the server.
> That will tell us if the outgoing packets are correct (if they are there
> at all), and if the response traffic is correctly interpreted (if any at
> all).

Attached is an strace of HAproxy, and a tcpdump of the traffic. In the tcpdump 10.193.67.9 is the client, 10.193.67.17 is the server running HAProxy, and 10.35.154.142 is the backend web server. I noticed that in the tcpdump that the client to HAproxy looks ok, but the communication from HAproxy to the backend server isn't working. HAProxy sends out 8 requests with the spoofed cleint ip (10.193.67.9), but it seems that the backend web server dosen't reply, or is trying to reply to the real client, and not HAProxy. That causes HAProxy to return a 503 to the client. That tells me that there is a problem with TProxy or my iptables config. I've seen some information on the web about setting up a bridge when using TProxy so that the routing will be correct. As far as you know, is that still the case with TProxy 4.1 and HAProxy?

> Here is my haproxy.cfg:
>
> at first glance, it looks fine.

Great, I was trying to go as minimal as possible so that I could avoid any potential problems with my config.

> global
> > debug
> >
> > defaults
> > balance source
> > mode http
> > timeout client 60s
> > timeout connect 5s
> > timeout http-request 10s
> > timeout server 60s
> >
> > frontend Frontend
> > bind 0.0.0.0:80
> > default_backend Default
> >
> > backend Default
> > server Web_Server_1 xxxxx:80 source 0.0.0.0 usesrc client
> > Any help or direction would be appreciated.
> >
> > Thanks.
>
> Regards,
> Willy
>
> I really appreciate your time, thanks!