The stock Unix syslogd suffers from a number of deficiencies, and is best replaced with a newer daemon such as syslog-ng. These notes show how syslog-ng can improve a logging infrastructure. The following list of changes are the intended improvements over stock logging.
- More reliable logging over the Transmission Control Protocol (TCP).
- No log file rotation: save to archive location by default.
- Improved log entries (year in date, facility and priority information).
These notes are based on the author’s experience managing a heterogeneous educational environment with around 200 systems logging on average 175,000 log entries daily to a central loghost; smaller sites or even single hosts should benefit from syslog-ng. The 1.6 release of syslog-ng is covered here, being run on a variety of Unix operating systems including FreeBSD, Mac OS X, OpenBSD, and RedHat Linux.
Future improvements to these notes may include the encryption of the TCP connections between the clients and the loghost systems, either via stunnel or a Virtual Private Network (VPN).
Be sure to use an up-to-date version of syslog-ng; older versions may not support newer features listed here, such as template statements and similar.
See the syslog-ng Frequently Asked Questions (FAQ) for more documentation.
Various scripts exist to feed data from syslog-ng into a database, such as sqlsyslogd.
Configuring syslog-ng
The syslog-ng daemon may be difficult to configure for those not experienced with the syslog-ng.conf configuration file. The following examples show how to configure and run syslog-ng in a variety of situations.
The following network overview diagram illustrates the various hosts discussed below. The central loghost accepts logs from various network devices: reliable TCP from the mail server and grid cluster master, and User Datagram Protocol (UDP) logs from the firewalls chickenwire and screendoor. Additionally, the cluster master grid proxies for a number of cluster nodes, which in turn log via UDP to grid. A stand alone laptop configuration is also covered.
High volume log servers should run or have access to caching Domain Name System (DNS) services. If all the systems on a network are under a configuration management system, Internet Addresses could be used instead of DNS records for logging.
Central Loghost Setup
A central loghost accepts log data from various network devices, ideally scans the data for problems, and saves the logs for future reference. Firewalling and bandwidth control should be used on the loghost to restrict access and prevent runaway log sources from drowning out other hosts.
Firewalls and network routers are likely sources of excessive logging, though I have seen a Linux desktop peak at 313 log messages per second due to a failing disk; there were multiple errors from the kernel and filesystem preventing the "last message repeated" limit from taking effect. Solutions to this problem include running multiple loghost to spread out the load, and using bandwidth control to prevent a single host from dominating.
If syslog-ng has been compiled with tcp_wrappers support, entries for syslog-ng may need to be made in the preferences file for tcp_wrappers. The following assumes a firewall is blocking unwanted connections to syslog-ng.
syslog-ng: ALL
The syslog-ng.conf configuration file uses blocks (options, source, filter, destination, and log) that together specify options, and how log entries are gathered, matched, and routed. See syslog-ng.conf for a complete example server configuration.
- Options
- Sources
- Filters
- Destinations
- Logs
The following options use a special logs group for all log files; being a member of the logs group is required to view the data. Some sites may want to chain hostnames, and push that data into a different column under a database; I leave the chain out and preserve the original fully qualified hostname in my setup. I also ignore the time set by the client, which is more likely to be wrong than on a properly managed loghost.
options {
group(logs);
dir_group(logs);
perm(0640);
dir_perm(0750);
create_dirs(yes);
log_fifo_size(4096);
use_fqdn(yes);
keep_hostname(yes);
chain_hostnames(no);
stats(3600);
bad_hostname("gconfd");
};
Using a custom logs user and group allows log analysis software to be run as the logs user, instead of the superuser.
Certain software has buggy log implementations, such as Gnome. To avoid this problem, use the bad_hostname or similar kluges to ignore or deal with such logs. This may need to be done in the configuration for the syslog-ng client.
I use a single source statement to join all incoming logs into a single group, though multiple source statements can be specified. The following is suitable for use on RedHat Linux, with an additional TCP source on port 5000. Kernel logs are forwarded by the klogd daemon.
The default max-connections limit for TCP will need to be increased on most loghost systems.
source local {
unix-stream("/dev/log");
udp();
tcp(ip(0.0.0.0) port(5000) max-connections(300));
internal();
};
Use filter blocks to match or exclude logs. The following shows two example filters.
filter notdebug {
level(info...emerg);
}
filter notmail {
not facility(mail);
}
The destination block is used to send logs somewhere, whether to a file, remote host, or program. Where to store the files on a loghost or whether to upload them to a database will be highly dependent on the needs of the site in question. For logging to files, I recommend changing the default log format to use a complete timestamp and include the facility and priority information.
Use the R_* macros when constructing filenames to log to. These macros contain the received timestamp of the log message, as opposed to the source timestamp (S_* macros, listed under src/macros.c), which could be wrong due to misconfiguration. If logging to a database, log both fields, so that comparisons can be done to look for clock skew or other problems.
destination messages {
file("/var/log/archive/messages/$R_YEAR/$R_MONTH/$R_YEAR-$R_MONTH-$R_DAY"
template("$ISODATE <$FACILITY.$PRIORITY> $HOST $MSG\n")
template_escape(no)
);
};
The above example also avoids the need to do log file rotation, as syslog-ng will create the new files based on the current time. This approach requires new scripts to keep track of and run checks on the log archives, as there is no rotation event to run a script from.
Solaris systems may append additional details after the program name that repeat the facility and priority information. To disable this, set msgid=0 in /kernel/dev/log.conf and reboot, or run echo log_msgid/W0 | adb -kw .
Finally, use log blocks to join source to destination with optional filter specifications.
log {
source(local);
filter(notdebug);
filter(notmail);
destination(messages);
};
Log Proxy Server
A proxy host can simply forward all logs on to the central loghost, or additionally do local checks on and storage of the incoming data to reduce the load on the central loghost. Using proxies scales well, as different groups can have different logging setups, and a proxy can be configured not to forward to the central loghost for security or performance reasons if needed.
- Options
- Sources
- Destinations
The proxy host does not need to log locally, but will need to maintain (or chain) the hostname.
options {
chain_hostnames(no);
keep_hostname(yes);
use_fqdn(yes);
log_fifo_size(1000);
};
Like a central loghost, the proxy listens for both UDP and TCP connections in addition to the usual local sources.
source local {
unix-stream("/dev/log");
udp();
tcp(ip(0.0.0.0) port(5000));
internal();
};
To forward all logs onto the central loghost using TCP, use the following destination configuration.
destination loghost {
tcp("loghost.example.org" port (5000));
};
log {
source(local);
filter(notdebug);
destination(loghost);
};
Client Configuration
Example syslog-ng.conf for client systems.
Forward Only
To log reliably from a critical host such as a mail server, but where no local filesystem logs are maintained, use the following, which only accepts network logs over the localhost interface, and sends almost all logs to loghost.example.org. This example uses log sources suitable to FreeBSD 4.
options {
use_fqdn(yes);
log_fifo_size(8192);
};
source local {
unix-dgram("/var/run/log");
file("/dev/klog");
udp(ip(127.0.0.1) port(514));
internal();
};
filter notdebug {
level(info...emerg);
}
destination loghost {
tcp("loghost.example.org" port (5000));
};
log {
source(local);
filter(notdebug);
destination(loghost);
};
Mac OS X
With everything else important running syslog-ng, my laptop running the default Apple syslog setup no longer uses the same log format as the other loghost, does not support the tools I use, nor does it use log file paths I am used to. This section covers replacing syslogd on Mac OS X 10.3 with syslog-ng and related notes.
syslog-ng (and the required libol library) compile from source. To replace the default syslogd, edit /etc/rc and start syslog-ng instead. Also, update the /etc/{daily,weekly,monthly} periodic scripts to not send a HUP signal to syslogd, and maybe also disable log file rotation. Logs are sent via SOCK_DGRAM to /var/run/syslog instead of the usual /var/run/log or /dev/log, so the following source statement should work for Mac OS X.
options {
dir_perm(0755);
perm(0644);
create_dirs(yes);
use_fqdn(yes);
log_fifo_size(4096);
stats(3600);
};
source local {
unix-dgram("/var/run/syslog");
udp(ip(127.0.0.1) port(514));
internal();
};
source s_kernel {
file("/dev/klog" log_prefix("kernel: "));
};
For a laptop, I use a single set of archive files so that all logs go to a single file each day. To reference the logs for the day, a /var/log/everything symlink is maintained via a periodic script.
destination everything {
file("/var/log/archive/$R_YEAR/$R_MONTH/$R_YEAR-$R_MONTH-$R_DAY"
template("$ISODATE <$FACILITY.$PRIORITY> $HOST $MSG\n")
template_escape(no)
);
};
log { source(local); source(s_kernel); destination(everything); };
filter emerg { level(emerg); };
destination users { usertty("*"); };
log { source(local); filter(emerg); destination(users); };
A laptop moved between different networks may have a different hostname; to update the logging to use the current hostname, edit the /System/Library/SystemConfiguration/Kicker.bundle/Contents/Resources/set-hostname shell script to include code to restart syslog-ng. Also test that logging is working, and update any convenience symbolic links.
This sort of customization can also be managed by a configuration utility like CFEngine.
if [ -f /var/run/syslog-ng.pid ]; then
kill -HUP `head -1 /var/run/syslog-ng.pid`
# test that logging is working
LOGSTAMP="syslog restart test, date=`date`, pid=$$"
logger -i -t logtest -- "$LOGSTAMP"
sleep 5
LOGFILE=`date +%/var/log/archive/%Y/%m/%Y-%m-%d`
if ! fgrep -- "$LOGSTAMP" $LOGFILE > /dev/null; then
logger -i -t logtest -p syslog.alert -- \
"log check failed, LOGSTAMP not found: file=$LOGFILE"
echo "LOGSTAMP not found in $LOGFILE" | mail -s "log check failed" root
fi
ln -sf $LOGFILE /var/log/everything
fi
Apple’s sys/syslog.h header file references several non-standard facility codes; will need to see how syslog-ng handles these.
#define LOG_NETINFO (12<<3) /* NetInfo */
#define LOG_REMOTEAUTH (13<<3) /* remote authentication/authorization */
#define LOG_INSTALL (14<<3) /* installer subsystem */
Olivier Scherler has contributed the following untested code, which should be added to src/syslog-names.c, "at the end of struct sl_name sl_facilities[], just before { NULL, -1 }". The code should allow syslog-ng to recognize the custom Apple logging facilities.
/* Mac OS X Specific: netinfo, remoteauth & install */
#ifdef LOG_NETINFO
{ "netinfo", LOG_NETINFO },
#endif
#ifdef LOG_REMOTEAUTH
{ "remoteauth", LOG_REMOTEAUTH },
#endif
#ifdef LOG_INSTALL
{ "install", LOG_INSTALL },
#endif
/* /Mac OS X Specific */
Windows
Windows clients can use custom programs such as Snare Agent to send logs to a syslog-ng server. Snare Agent includes a MSWinEventLog tag before each message. Match this tag with a program filter in syslog-ng.conf, and use flags(final) to route the logs from Windows into a custom file before any subsequent destination.
filter windows {
program(MSWinEventLog);
};
destination windows {
file("/var/log/archive/windows/$R_YEAR/$R_MONTH/$R_YEAR-$R_MONTH-$R_DAY"
template("$ISODATE <$FACILITY.$PRIORITY> $HOST $MSG\n")
template_escape(no)
);
};
log {
source(local); filter(windows); destination(windows);
flags(final);
};
Logfile Rotation
The above configurations for syslog-ng recommend the use of logging directly to files based on /var/log/archive/mail/$YEAR/$MONTH/$YEAR-$MONTH-$DAY instead of the usual /var/log/maillog or /var/log/mail.log seen with stock syslogd. This requires changes to the handling of logfile rotation, as there is no longer any rotation of the logfiles. Instead, both the creation of convenience symbolic links and custom handling of logfile analysis will need to be done.
To preserve conventional logfile names, and to provide easy access to the logfiles for the current day, symbolic links will need to be created and updated daily to point at the current logfile. For instance, /var/log/messages could be changed via the following code called from a cron(8) job.
ln -sf `date +%/var/log/archive/messages/%Y/%m/%Y-%m-%d` /var/log/messages
Additionally, the script should also contain a logger statement to generate a log, and additional code should be written to ensure the generated log is logged to the expected file. While more complex, this provides a daily check that logging is actually working. Such sanity tests could instead be integrated into a custom script for monitoring software, such as Big Brother or Nagios.
If links for multiple files are being maintained, use a loop of some sort that associates the destination and target in an easy to modify list.
(
cd /var/log || exit 1
while read symlink target; do
[ -z $target ] && target=$symlink
ln -sf `date +%archive/$target/%Y/%m/%Y-%m-%d` $symlink
done <<'EOF'
cron
maillog mail
messages
EOF
)
Other alternatives including parsing logfile locations directly from the syslog-ng.conf preferences file, or have configuration software like CFEngine update the links.
control:
syslog_ng_file = ( ExecResult(/bin/date "+%Y/%m/%Y-%m-%d") )
syslog_ng_dir = ( "/var/log/archive" )
links:
/var/log/maillog ->! $(syslog_ng_dir)/mail/$(syslog_ng_file) ifelapsed=59
/var/log/messages ->! $(syslog_ng_dir)/messages/$(syslog_ng_file) ifelapsed=59
Logfile Analysis
Standard logfile analysis relies on periodic rotation of the current logfile to trigger analysis with various programs, such as swatch or logwatch. With no logfile rotation being done, a new approach to logfile analysis is required. The method outlined here uses index data to record checksums of previously seen files, and calls various analysis software if the logfile in question has not been seen before.
Advantages of this approach include easy support for new analysis software, as the seen index can also include a custom tag for each form of analysis software. New logfile analysis programs can use a new tag, which allows previous logfiles to be processed by the new software, as they have not been seen in association with the tag before. Also, should a previously seen logfile be changed for some reason, analysis will be repeated on the file, which may be a clue that corruption or a security problem has occurred.
Disadvantages include the need to calculate checksums and maintain the index data on the logfiles, along with housekeeping problems related to the retiring old logfile entries and avoiding tagging active logfiles for analysis. The checksum calculations may be too expensive should there be large amounts of data on slower hardware.
This method works for both logfiles saved to archive locations, or logfiles rotated, as the checksum does not include the changing filename. Exclusions will need to be done if older logfiles are compressed, which changes the checksum, though an easier solution might be to save compressed logfiles to a different area of the filesystem.
The core of this method is provided by my unseen script, which maintains an index file of checksums, and reports to standard out names of files it has not seen before, allowing other scripts to then be run based on the output. It also has options to ignore active logfiles and retire old index entries. Additional documentation on the required wrapper scripts will be done later.
More information on log analysis with swatch is available.
