Hack 75 Colorized Log Analysis in Your Terminal

View your log files in an xterm window in full, living color

If you find yourself slowly going cross-eyed while looking at line after line of system logs, then you should consider using tools to help you organize your logs. While a properly configured syslog (as shown in [Hack #54]) goes a long way toward logfile sanity, it can still be a bit overwhelming to sift through a multi-megabyte /var/log/messages looking for patterns.

Just as a colorized ls can help identify types of files at a glance, a colorized grep is a handy tool for making patterns leap out of a sea of grey lines. There are a number of X applications that will do this for you, but why not make it easy to view your colorized logs from the command line?

Save this as ~/bin/rcg (short for Regex Colored Glasses):

#!/usr/bin/perl -w
use strict;
use Term::ANSIColor qw(:constants);

my %target = ( );

while (my $arg = shift) {
my $clr = shift;

if(($arg =~ /^-/) | (!$clr)) {
print "Usage: rcg [regex] [color] [regex] [color] ...\n";
exit;
}

#
# Ugly, lazy, pathetic hack here
#
$target{$arg} = eval($clr);
}

my $rst = RESET;

while(<>) {
foreach my $x (keys(%target)) {
s/($x)/$target{$x}$1$rst/g;
}
print;
}

rcg is a simple filter that uses Term::ANSIColor to colorize arbitrary regexs, specified on the commandline. It is intended to help visually slog through log files.

You must pass rcg an even number of command line parameters. The odd terms specify the regex, the even terms specify the color.

Suppose you wanted anything with the word sendmail in your messages log to show up magenta, instead of grey:

$ rcg sendmail MAGENTA < /var/log/messages | less -r

The less -r is optional but handy (as it displays the intended colors in less, instead of the ESC characters.)

You can use any arbitrary regex as an odd term:

$ rcg '\d+\.\d+\.\d+\.\d+' GREEN < /var/log/maillog

Or chain colors together:

$ tail -50 /var/log/messages | rcg WARNING 'BOLD . YELLOW . ON_RED' 

You can specify any number of regex/color code pairs on a single commandline. This is where teeny shell scripts or aliases would come in handy (one for messages, one for firewall logs, one for Apache).

See the Term::ANSIColor docs for the full list of colors and combinations.

Some other useful strings:

\w+=\S+

Variables, such as TERM=xyz

\d+\.\d+\.\d+\.\d+

Probably an IP address

^(J|F|M|A|S|O|N|D)\w\w (\d|)\d

Might be a date

\b\d\d:\d\d:\d\d\b

Possibly the time

.*last message repeated.*

Makes this "BOLD . WHITE"

Use your imagination, but be warned: color params are just eval( )'d. Theoretically, many valid Perl expressions can be substituted for regexes or colors; exploiting this, uh, feature is left as an exercise to the reader. You probably shouldn't be running arbitrary rcg lines as root, unless you wrote them yourself. Also note that colorization is applied in arbitrary order, so it's not possible to guarantee the behavior of overlapping regexes.