RE: Reducing I/O load of logging

If you prefix the filename with a -, that tells syslog not to flush the data to disk immediately (which is only good in case of a kernel crash, as you might not miss a log entry leading to it, ha!), and it is inefficient forcing a sync every log entry when you have a log entry more at least every few seconds, and much more so when you have hundreds or thousands a second.

Note, this isn't exactly standard in all versions of syslog. Do a "man syslog.conf" to make sure it applies to your OS, but generally is standard with Linux, and other operating systems have others ways to allow for normal buffering.

So, instead of a line like:
*.debug;mail.none;authpriv.none;cron.none /var/log/messages Use the following:
*.debug;mail.none;authpriv.none;cron.none -/var/log/messages

How are you doing your log rotates? That seems strange that it would cause problems, as that should be a relatively light process, unless you are doing something like compressing the logs when you rotate. It should be just be a mv and then send a -HUP to syslogd. Of course, if a simple grep is causing issues, then it sounds like you definitely have some sort of resource contention problem.

> -----Original Message-----
> From: Michael Fortson [mailto:mfortson#gmail.com]
> Sent: Wednesday, February 11, 2009 8:38 PM
> To: John Lauro
> Cc: James Brady; haproxy#formilux.org
> Subject: Re: Reducing I/O load of logging
>
> Hi John,
>
> Can you explain what you mean by making sure there's a dash in the
> name? Is this for something like syslog via /var/log/messages?
>
> We've had some major issues when logrotate kicks in, knocking haproxy
> offline (it reports most of the back-end servers as down and starts
> throwing 503s). It's a busy site with 80 servers configured in the
> backend. Reducing logrotate to 3 hour intervals shortened the
> downtimes but it still happened. It does the same thing if I try to
> grep the log.
>
> I've had to turn off logrotate for now...
>
>
>
>
>
> On Wed, Feb 11, 2009 at 10:01 AM, John Lauro
> <john.lauro#covenanteyes.com> wrote:
> > It shouldn't be too hard for a machine to keep up with logging. How
> are you
> > logging? standard syslog? Make sure you have - in front of the
> filename?
> > How connections per second are you logging?
> >
> >
> >
> > Haven't done it with Haproxy, but have with other things that
> generate tons
> > of logs.
> >
> >
> >
> > what you could do is dump the logs (don't forget the - as part of the
> file
> > name) to /dev/shm/ (assuming linux), and then rotate the logs once a
> minute
> > and process them. That way, you will not have any disk I/O from the
> logs,
> > but would increase memory requirements.
> >
> >
> >
> >
> >
> >
> >
> > From: James Brady [mailto:james.colin.brady#gmail.com]
> > Sent: Wednesday, February 11, 2009 12:14 PM
> > To: haproxy#formilux.org
> > Subject: Reducing I/O load of logging
> >
> >
> >
> > The machine we run HAProxy on is approaching its limits in terms of
> disk I/O
> > due to our debug-level logging.
> >
> >
> >
> > The CPU is barely used, memory is no problem at all - the bottleneck
> will
> > soon be the rate at which we can log to disk. Because the machine is
> more
> > than adequate in all other areas, we'd really like to reduce the I/O
> load of
> > logging.
> >
> >
> >
> > We are using the debug log output to tell us the response time and
> status
> > code for various URLs - this is an essential task we can't do
> without.
> >
> >
> >
> > Is there any way to get this information without logging and
> > post-processing every request? Can HAProxy dump out averaged
> statistics like
> > this? Can we somehow reduce the I/O load by just logging the exact
> fields
> > we're interested in?
> >
> >
> >
> > Many thanks!
> >
> > James
Received on 2009/02/12 04:36