Hack 54 Steering syslog

Make syslog work harder, and spend less time looking through huge log files

The default syslog installation on many distributions doesn't do a very good job of filtering classes of information into separate files. If you see a jumble of messages from sendmail, sudo, bind, and other system services in /var/log/messages, then you should probably review your /etc/syslog.conf.

There are a number of facilities and priorities on which syslog can filter. For easy reference, here they are:

Facilities	Priorities
auth auth-priv cron daemon kern lpr mail news syslog user uucp local0 - local7	debug info notice warning err crit alert emerg

Note that applications decide for themselves at what facility and priority to log (and the best applications let you choose), so they may not always be logged as you expect. Here's a sample /etc/syslog.conf that attempts to shuffle around what gets logged where:

auth.warning /var/log/auth
mail.err /var/log/maillog
kern.* /var/log/kernel
cron.crit /var/log/cron
*.err;mail.none /var/log/syslog
*.info;auth.none;mail.none /var/log/messages

#*.=debug /var/log/debug

local0.info /var/log/cluster
local1.err /var/log/spamerica

All of the above lines will log the specified priority (or higher) to the respective file. The special priority none tells syslog not to bother logging the specified facility at all. The local0 through local7 facilities are supplied for use with your own programs, however you see fit. For example, the /var/log/spamerica file fills with local1.err (or higher) messages that are generated by our spam processing job. It's nice to have those messages separate from the standard mail delivery log (in /var/log/maillog.)

That commented *.=debug line is useful when debugging daemonized services. It tells syslog to specifically log only debug priority messages of any facility and generally shouldn't be running (unless you don't mind filling your disks with debug logs). Another approach is to log debug information to a fifo. This will make debug logs take up no space but will disappear unless a process is watching it. To log to a fifo, first create it in the filesystem:

# mkfifo -m 0664 /var/log/debug

Then amend the debug line in syslog.conf to include a | like this:

*.=debug |/var/log/debug

Now debug information is constantly logged to the fifo, and can be viewed with a command like less -f /var/log/debug. This is also handy to set up if you want a process to constantly watch all system messages and perhaps notify you via email when a critical system message is seen. Try making a fifo called /var/log/monitor, and add a rule like this to your syslog.conf:

*.* |/var/log/monitor

Now every message (at every priority) is passed to the /var/log/monitor fifo, and any process watching it can react accordingly, all without taking up any disk space.

54.1 Mark Who?

Do you notice a bunch of lines like this in /var/log/messages?

Dec 29 18:33:35 catlin -- MARK --
Dec 29 18:53:35 catlin -- MARK --
Dec 29 19:13:35 catlin -- MARK --
Dec 29 19:33:35 catlin -- MARK --
Dec 29 19:53:35 catlin -- MARK --
Dec 29 20:13:35 catlin -- MARK --
Dec 29 20:33:35 catlin -- MARK --
Dec 29 20:53:35 catlin -- MARK --
Dec 29 21:13:35 catlin -- MARK --

These are generated by the mark functionality of syslog, as a way of "touching base" with the system, so that you can theoretically tell if syslog has unexpectedly died. Most times, this only serves to fill your logfiles, and unless you are having problems with syslog, you probably don't need it. To turn this off, pass the -m 0 switch to syslogd (after first killing any running syslogd), like this:

# killall syslogd; /usr/sbin/syslogd -m 0

54.2 Remote Logging

Modern versions of syslogd disable the ability to receive logs from remote machines and for good reason. Without safeguards in place, it is entirely possible for a random miscreant to fill up your disk with bogus syslog messages, as there is no host authentication available in syslog. Still, it is very handy to have a centralized syslog, particularly when dealing with a cluster of machines.

To turn on remote reception on the master, start syslogd with -r:

# /usr/sbin/syslogd -m 0 -r

On each of the machines you'd like to log from, add this to the syslog.conf:

*.* @master.syslog.host.com

(Naturally, with your own hostname or IP address after the @.) It is a very good idea to protect the syslog port on your syslog server. It listens on UDP port 514 and is easily filtered with this iptables command:

# iptables -A INPUT -t filter -p udp --dport 514 -s ! $LOCAL_NET -j DROP

where $LOCAL_NET is the network (or host) that you would like to receive sylog messages from (e.g., 192.168.1.0/24 or florian.nocat.net).

Once your system logs are better organized, it becomes much easier to find the information you're looking for. Even then, system logs can get a bit overwhelming. For even more help dealing with log files, check out [Hack #75].