Date: Mon, 17 May 2010 19:42:03 -0700
I have some sites running a similar set up - Xen domU, keepalived, fedora not
RHEL and they get 50+ million hits per day with pretty fast response. you
might want to use the "log separate errors" (sp?) option and review those 50X
errors carefully, you might see a pattern - do you have http-close* in all you
configs? That got me weird, slow results when I missed it once.
example stats page screenshot attached.
On 5/17/10 2:45 PM, Chih Yin wrote:
> Hi,
>
> Please excuse me if the information contained in this email is a bit
> generic. I'm not the regular administrator, but I've been given the
> task of troubleshooting some issues with my website. If more details
> are needed, I'll gladly go look for additional information.
>
> Currently, my website is experiencing a lot of errors and slowness.
> The server errors I see in the HAProxy log file are mainly 503 errors.
> The slowness of page loads for my website can be as long as minutes.
> I'm trying to determine if anyone have had any similar issues with
> using HAProxy as a high availability load balancer.
>
> HAProxy 1.3.21
> CentOS running on Citrix XenServer
> HP blades
>
> There are actually almost 100 virtual servers running on the blades.
> A good many of the virtual servers are application servers running
> Glassfish. There are a few servers dedicated to CAS for authentication
> and access. I have three servers running Rsyslogd for writing HAProxy
> log data to file. A NetApp filer is used for storage.
>
> Currently, the website gets about:
>
> 73,000 pageviews a day
> 32,000 unique visitors a day
> 46,000 visit a day
>
> 3,000 pageviews a hr
> 1,300 unique visitors a hr
> 1,000 visit a hr
>
> I am using Akamai to help manage content delivery.
>
> One of the things Akamai is reporting to me is that they are having
> difficulty requesting content that needs to be refreshed. Akamai tries
> up to 4 times to get the content with a 2 second timeout to update
> content whose TTL has expired. After the 4th time, Akamai looks to
> their own cache before returning a 503 error to the user if the content
> is not available in the cache.
>
> Recently, I've noticed that Akamai is encountering an increasingly
> large number of 503 and 404 errors from my website. I've traced the 404
> errors to missing images, but I'm not sure what the cause of the 503
> errors could be. I had some external resources help me verify that they
> are able to retrieve the content from the Glassfish application servers
> even when HAProxy is reporting the 503 errors.
>
> One thing I did notice about the HAProxy configuration is that there
> are actually three servers running HAProxy with identical
> configurations. One serves as the primary high availability load
> balancer while the other two act as failovers. The keep-alive daemons
> are configured to accomodate that setup.
>
> From this generic description, is there something in the way this
> architecture is set up or in the configuration of HAProxy that may be
> causing the 503 errors to be reported to Akamai? As I mentioned, when
> an external resource makes a request for the same content directly from
> the application server, the same errors do not appear to occur.
>
> thanks,
> C.Y.
>
>
