Re: high cpu utilization

Thanks for continuing to help me trouble-shoot here. Wasn't able to upgrade the kernel today as we had issues with failover. Will try again next saturday morning.

You asked:
What's the approximative session rate and data rate ?

We're peaking at about 30Mbps. There are two types of data going through haproxy:
- HTTP - 90% of requests have maybe 30 bytes of data outside of the HTTP
overhead. These 90% requests are AJAX polling calls that are *held open on the server for 20 seconds* before returning a response that effectively says "no messages waiting for you".
- TCP - there's a lot of data going through here at high rates. I'd
estimate that this makes up 20Mbps of the 30. Yesterday for about 60 minutes we didn't have the TCP traffic going through here. We were bypassing the load balancer during that period. The CPU usage of haproxy remained at the same 100% level during that period. These TCP sessions are persistent, btw. There are about 100 of them and once the sockets are established they stick around for days at a clip.

You asked about the size of the machine: Dual Core Pentium E2160, 1.8 GHz, 1MB Cache, 800MHz Front Side Bus 2GB RAM Glad to hear the configuration sounds ok and that health-checks are not an issue at all.

Here's the stats output from strace:
% time seconds usecs/call calls errors syscall

------ ----------- ----------- --------- --------- ----------------
 93.05    3.276951        1146      2859           poll
  1.56    0.054937          12      4532         8 send
  1.29    0.045360           5      8370        40 recv
  1.04    0.036664           9      3941           close
  0.68    0.023890           7      3235      1314 accept
  0.67    0.023423          12      1938      1926 connect
  0.47    0.016556           9      1926           socket
  0.37    0.013019           7      1877           shutdown
  0.36    0.012678           3      3847           setsockopt
  0.30    0.010513           3      3847           fcntl64
  0.21    0.007548           3      2860           gettimeofday
  0.00    0.000103           4        28           getsockopt
------ ----------- ----------- --------- --------- ----------------
100.00    3.521642                 39260      3288 total