Re: httpchk - checking request body

Hello Anze.

On 13 March 2010 20:08, Anze <anzenews#volja.net> wrote:
>> Interesting. Have you run haproxy -d from the command line to check
>> the debugging output? "option log-health-checks" is also useful.
> I was running it from the command line (too) because logging was not working
> (rsyslogd's UDP support was disabled). That was the output I have sent in the
> first mails.

Ah, missed that email.

>> It's a half-close. HAProxy only sends one request, and then waits for
>> the response, so we might as well close the transmit channel.
> Are you sure nothing is being sent anymore through this socket? Never?

Not in my packet traces. HAProxy sends one request, receives the response from the real server, real server closes its channel, HAProxy replies with RST. Future tests bind a new socket.

>> When tcpdump'ing the problem system that led me to work on this, I
>> found that HAProxy was not closing the transmit channel properly. The
>> real servers were sending FIN to close their side, and HAProxy was
>> responding with RST. So I put in the half-close to make sure the
>> connection was cleanly shut down.
> I think this is not the proper way to deal with this, but as I said, I am no
> expert in sockets programming.

Neither am I - my background is in System/Network Admin. I'm also just learning the HAProxy code, so there may be a better way to ensure that the check connection is closed properly.

> I would say that haproxy still tries to sent RST but can't - which leads to an
> error somewhere.

The RST should not be sent - that's a response to an already-closed socket. However, HAProxy is not signalling that the connection should be closed by sending a FIN.

>> It's interesting that this has caused problems in your system. As well
>> as running HAProxy in debug mode, would you be able to do a tcpdump on
>> the system running haproxy? If you could do that both with and without
>> the half-close, that would be very useful.
> Hmmm, I am a bit out of depths here. I would love to help out, but please send
> me more specific instructions on what you need.
>
> What arguments should I use with tcpdump so it helps you? Also, if you don't
> mind, I'd like to send it privately to you and Willy (so I don't need to clean
> up too much private / security sensitive info from it).

I have no problem with you sending it to me directly.

My usual tcpdump invocation is:

   tcpdump -i any -s 0 -w <output_file>
Substitute an appropriate filename for <output_file>. You won't want to leave this running for too long, as it listens on all interfaces and captures the whole of every packet. I'd suggest starting the tcpdump, then start HAProxy, wait for a few health checks to be sent, then stop the tcpdump with Ctrl-C.

>> If required, I can send you a version of checks.c with debugging code
>> in it - that will dump a lot of data on the checks if you run it in
>> debug.
> Sure, bring it on. ;)

I'll send that directly to you, rather than the list...

>> Those two will not check the content - I'll need to trawl through the
>> config code to check whether it defaults to checking the HTTP return
>> code in these cases.
> I have tested GET with the fixed version (no half-closed socket) and it works.

Interesting. I'll test that locally.

Thanks for your help debugging this.

Nick.

-- 
Nick Chalk.

Loadbalancer.org Ltd.
Phone: +44 (0)870 443 8779
http://www.loadbalancer.org/