[haproxy] Help tracking down a possible HAProxy outage...

From: hernan <hernan.silberman#gmail.com>
Date: Fri, 28 Nov 2008 15:59:52 -0800

Hey folks, I need some help analyzing an application outage this afternoon.  Here's what I see in my haproxy log. Client IP addresses have been obscured and I've added my comments in [[]]'s:

---

Nov 28 16:21:18 127.0.0.1 haproxy[21447]: 0.0.0.0:4408[28/Nov/2008:16:21:18.451] http app/<NOSRV> 0/0/0/45/151 200 484 - - ----
1999/1999/290/0 0/0 "POST /mysite/graniteamf/amf HTTP/1.1" Nov 28 16:21:18 127.0.0.1 haproxy[21447]: 0.0.0.0:50333[28/Nov/2008:16:21:18.603] http app/<NOSRV> 0/0/0/23/110 200 484 - - ----
1999/1999/290/0 0/0 "POST /mysite/graniteamf/amf HTTP/1.1" Nov 28 16:21:18 127.0.0.1 haproxy[21447]: 0.0.0.0:52577[28/Nov/2008:16:21:18.713] http app/<NOSRV> 0/0/0/42/146 200 485 - - ----
1999/1999/290/0 0/0 "POST /mysite/graniteamf/amf HTTP/1.1" Nov 28 16:21:18 127.0.0.1 haproxy[21447]: 0.0.0.0:50602[28/Nov/2008:16:21:18.860] http app/<NOSRV> 0/0/0/79/135 200 7836 - - ----
1999/1999/290/0 0/0 "POST /mysite/graniteamf/amf HTTP/1.1"

[[ Looks like either no requests came for several minutes here which is
unlikely, or haproxy stopped accepting connections at this point. ]]

Nov 28 16:29:42 127.0.0.1 haproxy[21447]: 0.0.0.0:51556[28/Nov/2008:16:29:42.355] smartfox sf/<NOSRV> -1/1/0/-1/61 0 92 - - ----
2000/0/0/0 0/0 "<BADREQ>"

[[ The above log record is the only sign of life in the server and users
called en masse to complain about a service outage. ]]

[[ HAproxy was restarted at this point. ]

Nov 28 16:34:39 127.0.0.1 haproxy[4403]: Proxy smartfox started.
Nov 28 16:34:39 127.0.0.1 haproxy[4403]: Proxy http started.
Nov 28 16:34:39 127.0.0.1 haproxy[4403]: Proxy registration started.
Nov 28 16:34:39 127.0.0.1 haproxy[4403]: Proxy sf started.
Nov 28 16:34:39 127.0.0.1 haproxy[4403]: Proxy static started.
Nov 28 16:34:39 127.0.0.1 haproxy[4403]: Proxy app started.

[normal healthy operation after the restart] Nov 28 16:34:39 127.0.0.1 haproxy[4403]: 0.0.0.0:50523[28/Nov/2008:16:34:39.908] http app/<NOSRV> 0/0/0/1/26 200 447 - - ----
3/3/3/0 0/0 "POST /mysite/graniteamf/amf HTTP/1.1" Nov 28 16:34:39 127.0.0.1 haproxy[4403]: 0.0.0.0:50522[28/Nov/2008:16:34:39.895] http app/<NOSRV> 4/0/1/38/68 200 485 - - ----
3/3/3/0 0/0 "POST /mysite/graniteamf/amf HTTP/1.1"

---

All of the individual services behind HAProxy are monitored closely and the only alerts I received were those that tested the application through HAProxy. There didn't appear to be any network issues getting to the HAProxy host. On the HAProxy host there were no OS errors in the system logs and the machine looks effectively idle and healthy as usual.

HAProxy's maxconn is set really high, I don't think we were even close to the limit. HAProxy version info is "HA-Proxy version 1.3.14.6 2008/06/21" running on Linux (CentOS 5).

Let me know if you can suggest additional things to look for in my HAProxy configuration. I've been looking at the host and I don't see anything that strikes me as out of the ordinary. I can share my haproxy.cfg if needed.

thanks!
Hernan Received on 2008/11/29 00:59