Re: Reducing I/O load of logging

WOW. Make sure you have - in front of the filename?
>

This makes quite a difference!

Before:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdb 0.00 303.60 0.00 736.46 0.00 4160.24 11.30 0.73 1.00 0.98 72.53

After:
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sdb 0.00 12.29 0.00 3.00 0.00 122.28 40.80 0.01 4.33 3.00 0.90

CPU usage has doubled, however. To 2%.

Thanks John!
James

2009/2/11 John Lauro <john.lauro#covenanteyes.com>

> It shouldn't be too hard for a machine to keep up with logging. How are
> you logging? standard syslog? Make sure you have - in front of the
> filename? How connections per second are you logging?
>
>
>
> Haven't done it with Haproxy, but have with other things that generate tons
> of logs…
>
>
>
> what you could do is dump the logs (don't forget the – as part of the file
> name) to /dev/shm/ (assuming linux), and then rotate the logs once a minute
> and process them… That way, you will not have any disk I/O from the logs,
> but would increase memory requirements.
>
>
>
>
>
>
>
> *From:* James Brady [mailto:james.colin.brady#gmail.com]
> *Sent:* Wednesday, February 11, 2009 12:14 PM
> *To:* haproxy#formilux.org
> *Subject:* Reducing I/O load of logging
>
>
>
> The machine we run HAProxy on is approaching its limits in terms of disk
> I/O due to our debug-level logging.
>
>
>
> The CPU is barely used, memory is no problem at all - the bottleneck will
> soon be the rate at which we can log to disk. Because the machine is more
> than adequate in all other areas, we'd really like to reduce the I/O load of
> logging.
>
>
>
> We are using the debug log output to tell us the response time and status
> code for various URLs - this is an essential task we can't do without.
>
>
>
> Is there any way to get this information without logging and
> post-processing every request? Can HAProxy dump out averaged statistics like
> this? Can we somehow reduce the I/O load by just logging the exact fields
> we're interested in?
>
>
>
> Many thanks!
>
> James
>
Received on 2009/02/11 19:44