Re: A (Hopefully Not too Generic) Question About HAProxy

2010/5/17 XANi <xani666#gmail.com>

> Dnia 2010-05-17, pon o godzinie 14:45 -0700, Chih Yin pisze:
>
> Hi,
>
>
>
> Please excuse me if the information contained in this email is a bit
> generic. I'm not the regular administrator, but I've been given the task of
> troubleshooting some issues with my website. If more details are needed,
> I'll gladly go look for additional information.
>
>
>
> Currently, my website is experiencing a lot of errors and slowness.
> The server errors I see in the HAProxy log file are mainly 503 errors. The
> slowness of page loads for my website can be as long as minutes. I'm trying
> to determine if anyone have had any similar issues with using HAProxy as a
> high availability load balancer.
>
>
>
> HAProxy 1.3.21
>
> CentOS running on Citrix XenServer
>
> HP blades
>
>
>
> There are actually almost 100 virtual servers running on the blades. A
> good many of the virtual servers are application servers running Glassfish.
> There are a few servers dedicated to CAS for authentication and access. I
> have three servers running Rsyslogd for writing HAProxy log data to file. A
> NetApp filer is used for storage.
>
>
>
> Currently, the website gets about:
>
>
>
> 73,000 pageviews a day
>
> 32,000 unique visitors a day
>
> 46,000 visit a day
>
>
>
> 3,000 pageviews a hr
>
> 1,300 unique visitors a hr
>
> 1,000 visit a hr
>
>
>
> I am using Akamai to help manage content delivery.
>
>
>
> One of the things Akamai is reporting to me is that they are having
> difficulty requesting content that needs to be refreshed. Akamai tries up
> to 4 times to get the content with a 2 second timeout to update content
> whose TTL has expired. After the 4th time, Akamai looks to their own cache
> before returning a 503 error to the user if the content is not available in
> the cache.
>
>
>
> Recently, I've noticed that Akamai is encountering an increasingly
> large number of 503 and 404 errors from my website. I've traced the 404
> errors to missing images, but I'm not sure what the cause of the 503 errors
> could be. I had some external resources help me verify that they are able
> to retrieve the content from the Glassfish application servers even when
> HAProxy is reporting the 503 errors.
>
>
>
> One thing I did notice about the HAProxy configuration is that there
> are actually three servers running HAProxy with identical configurations.
> One serves as the primary high availability load balancer while the other
> two act as failovers. The keep-alive daemons are configured to accomodate
> that setup.
>
>
>
> From this generic description, is there something in the way this
> architecture is set up or in the configuration of HAProxy that may be
> causing the 503 errors to be reported to Akamai? As I mentioned, when an
> external resource makes a request for the same content directly from the
> application server, the same errors do not appear to occur.
>
>
> 503 would (usually) mean haproxy sees backends as DOWN, look for msg about
> server goin up/down in haproxy logs.
> Or, if thats not a case, grep haproxy logs for those 503 errors (make sure
> ure using http log mode), then go to section 8 in
> http://haproxy.1wt.eu/download/1.3/doc/configuration.txt and try to
> determine what was exact reason for error and/or post few examples here,
> your haproxy config (with "sensitive" information removed ofc ;) )
>
> Having some kind of monitoring, or at least stats page active is also very
> helpful.
>

Hi Mariusz,

  Thank you for the link. I'm reviewing the documentation now. Hopefully, it'll provide some additional insights for me. I do see in my configuration that HAProxy is using httplog, but that's only specified in the front end.  Is that correct?

  The status page for HAProxy is active, and I don'e believe we're seeing servers reported as being down during the times when we encounter the 503 errors. We are using zabbix to monitor our virtual servers with mixed results. I'm currently not involved with the configuration of zabbix, so I cannot comment on why we don't always get notifications during a time of increased numbers of 503 errors.

  I've attached a (hopefully) cleaned copy of our HAProxy configuration file in a separate email. I didn't want to spam the list with multiple attachments of the same thing.

Thank you for your help,
C.Y.

> --
> Mariusz Gronczewski (XANi) <xani666#gmail.com>
> GnuPG: 0xEA8ACE64http://devrandom.pl
>
>
Received on 2010/05/19 00:58