Re: Reducing I/O load of logging

Sorry, forgot to answer the disk question. I *think* this has 6 10k rpm drives in a raid 10. It's a dell running FC7.

On Thu, Feb 12, 2009 at 8:20 PM, Michael Fortson <mfortson#gmail.com> wrote:

> Here's the result:http://pastie.org/387928
>
> This box used to run everything (much of which has now been moved to other
> clusters). If I can't get it to behave it'll be doing nothing soon :)
>
> log/messages isn't large enough to trigger a misbehavior, but hopefully
> it'll show something... I can't really do it on the nginx log (which is
> massive) because I always have to kill that before enough backend tests flip
> over to cause a site outage.
>
>
>
>
>
>
>
> On Thu, Feb 12, 2009 at 6:44 PM, John Lauro <john.lauro#covenanteyes.com>wrote:
>
>> > I stopped logging so much in haproxy, but I get the same thing if I
>> > grep the nginx logs on this server: haproxy's mongrel backend checks
>> > start failing. I've noticed it only happens when using httpchk (or at
>> > least it happens much, much more quickly).
>> >
>> > Here's an iostat I ran -- the first two are during the grep on the
>> > nginx logs; the last one is after I finished:
>>
>> The iostat looks ok.
>>
>> Cut-n-past the following (or run from a script) so we can get a better
>> idea
>> of the box's general load and to see if they turn up anything:
>>
>> cat /proc/interrupts
>> free
>> netstat --inet -n | awk '{ print $6 }' | sort | uniq -c
>> ulimit -a
>> vmstat 1 10 & ( sleep 5 ; grep whatever /var/log/messages >/dev/null )
>> cat /proc/interrupts
>> echo lsof count `lsof | wc -l`
>>
>> What type of disk subsystem do you have? Given how it chokes when doing a
>> grep, it almost sounds like you might have a faulty driver. You do
>> realize
>> 8 cores is overkill for this, unless you are running other stuff on the
>> box.
>> The two checks on the interrupts is to see if something (especially disk
>> I/O) is generating too many as we need to look at the difference.
>>
>>
>>
>
Received on 2009/02/13 05:23