RE: Reducing I/O load of logging

> I stopped logging so much in haproxy, but I get the same thing if I
> grep the nginx logs on this server: haproxy's mongrel backend checks
> start failing. I've noticed it only happens when using httpchk (or at
> least it happens much, much more quickly).
>
> Here's an iostat I ran -- the first two are during the grep on the
> nginx logs; the last one is after I finished:

The iostat looks ok.

Cut-n-past the following (or run from a script) so we can get a better idea of the box's general load and to see if they turn up anything:

cat /proc/interrupts
free
netstat --inet -n | awk '{ print $6 }' | sort | uniq -c ulimit -a
vmstat 1 10 & ( sleep 5 ; grep whatever /var/log/messages >/dev/null ) cat /proc/interrupts
echo lsof count `lsof | wc -l`

What type of disk subsystem do you have? Given how it chokes when doing a grep, it almost sounds like you might have a faulty driver. You do realize 8 cores is overkill for this, unless you are running other stuff on the box. The two checks on the interrupts is to see if something (especially disk I/O) is generating too many as we need to look at the difference. Received on 2009/02/13 03:44