Re: A (Hopefully Not too Generic) Question About HAProxy

On Mon, May 17, 2010 at 11:11 PM, Hank A. Paulson < hap#spamproof.nospammail.net> wrote:

> On 5/17/10 10:24 PM, Willy Tarreau wrote:
>
>> On Mon, May 17, 2010 at 07:42:03PM -0700, Hank A. Paulson wrote:
>>
>>> I have some sites running a similar set up - Xen domU, keepalived,
>>> fedora not RHEL and they get 50+ million hits per day with pretty
>>> fast response. you might want to use the "log separate errors" (sp?)
>>> option and review those 50X errors carefully, you might see a pattern
>>> - do you have http-close* in all you configs? That got me weird, slow
>>> results when I missed it once.
>>>
>>
>> Indeed, that *could* be a possibility if combined with a server maxconn
>> because connections would be kept for a long time on the server (waiting
>> for either the client or the server to close) and during that time nobody
>> else could connect. The typical problem with keep-alive to the servers in
>> fact. The 503 could be caused by requests waiting too long in the queue
>> then.
>>
>
> My example was just to assure Chin Yin that haproxy on xen should be able
> to handle his current load depending, of course, on the glassfish servers.
>
> I meant some kind of httpclose option
> (httpclose/forceclose/http-server-close/etc) turned on regardless of
> keep-alive status - you know, like you are always reminding people :)
>
> I noticed when I forgot it on a section (that was not keepalive related) it
> caused wacky results - hanging browsers, images/icons/css not showing up,
> etc. Obviously it should not affect single requests like you would assume
> Akamai would be sending, it was a pure guess.
>
>

Thank you everyone for your feedback. I really appreciate your help.

Sorry for taking so long to respond. I had to get permission from my director to post some of the log data and our haproxy configuration file. I also had to hide a bit more of the configuration than was suggested because of concerns about making the issues we're encountering too public. I hope you understand.

From my research on HAProxy and high availability websites in general, it seemed to me that compared to other websites, our traffic volume is actually rather light. In addition to how we have configured HAProxy for our infrastructure, I'm definitely also taking a look at our application servers and our content as well.

I started looking at the log files and the HAProxy configuration file more closely today.

I attached the (poorly) cleaned HAProxy configuration file. Looking at it, I can already see that the httpclose option isn't consistently included in all the sections, both the frontend and the backend. I will make sure this option is in all sections. Should I also add this to the global settings for HAProxy? Is it okay if this option is listed more than once in a section (I noticed that this happened a couple of times)?

>

>> Chin Yin, Xani was right, please take a look at your logs. Also, sending
>> us your config would help a lot. Replace IP addresses and passwords with
>> "XXX" if you want, we'll comment on the rest. BTW you should tell your
>> admin that 1.3.21 has an annoying bug which makes it crash when connecting
>> to the stats socket. Thus, this reduces your possibilities of debugging
>> it.
>> When you have some time, you should upgrade it to 1.3.22 or later (1.3.24)
>> which fix a small number of remaining bugs.
>>
>> example stats page screenshot attached.
>>>
>>
>> Nice stats Hank :-)
>>
>
> That is just the page frames (mostly) not including images, css, js, static
> icons or any other "stuff" but neither is it just for one day, it is longer.
>
>

I have already reported to my director to let him know that we really need to upgrade to 1.3.22 or later.

As for the logs, it seems that I'll need to look at the configuration for HAProxy a bit more to make some adjustments first. A few months back, I know I saw messages indicating the status of server (e.g. 3 active, 2 backup). I also see messages when the HAProxy configuration was reloaded or when HAProxy was restarted. I no longer see these status messages in the log files. I recall that the system administrator who initially configured HAProxy mentioned that he removed the logging of some inter-server traffic to make the log file sizes smaller. I'm wondering if he also removed these status messages as well.

Again, thank you all for your help and suggestions. C.Y.

>
>> Cheers,
>> Willy
>>
>>
>