Re: Getting error 502 Bad Gateway

Hi Joseph,

On Sat, Nov 29, 2008 at 06:25:47PM -0500, Joseph Hardeman wrote:
> Hi Everyone,
>
> We are having some issues with our haproxy servers, when I go to my web
> servers and restart them, I take them out of haproxy before restarting
> my apache servers, but afterwards some people are getting the "502 Bad
> Gateway The server returned an invalid or incomplete response" error.

When you say you're taking them out of haproxy, how do you proceed ? I suspect two possibilities :

• you do a hot restart, and some slow sessions are still bound to the server through the old process. When you stop them, the error is forwarded to the users.
• you do a cold restart but don't wait long enough for the check to detect that the servers are dead. You have inter 5000, so you need at least 5 seconds for the first check to complete. By default, haproxy considers the servers are UP when starting, so that it does not return 503 to all clients. But if you wait long enough for it to know that the server is indeed stopped, you should be OK.

> It doesn't happen every time, I could be jumping around in the web site
> and then have the "502 Bad Gateway" thrown to us.
>
> I have gone into my haproxy.conf and set "cookie SERVERID rewrite"
> originally we had it "cookie SERVERID insert indirect nocache" and I had
> "capture cookie ASPSESSION len 32" which I have now removed.

This should be unrelated, you can safely keep the insert mode.

> Even after making these changes, we are still getting the errors. Can
> someone suggest a fix or a way around this issue? I am putting in the
> config below.

You can use "fastinter" on server lines to check for transitions faster. For instance, with "fastinter 1000", the servers will be checked every second when detected not fully up, which is the situation on start-up. It would ensure that within one second, your server is detected as down an no more traffic will be sent to it.

Before shutting your server, you should also check its logs to ensure that it's not used anymore.

> Also, I upgraded to 1.3.15.6 from 1.3.15.5 and we are still getting the
> 502 errors. I can go directly to the server and get the web page with
> no problem.

Also, please provide some extracts of the haproxy logs exhibiting the 502 errors, along with the line containing "server is down" and "proxy started", so that we can find how this is happening. You can blank your client's IP address and even the request if you want, as that's useless in this case.

Regards,
Willy Received on 2008/12/03 06:58