HEADS UP: Logging changes: clog deprecated, plain text logs + newsyslog rotation in next snapshots
Maintaining clog patches and dealing with clog file format/output is cumbersome and not a strict requirement now that we have moved beyond the age of embedded/nanobsd.
I have made changes to remove clog from the system and from here on, the log files will be plain text, with default rotation configured at the previous set log sizes.
A few notes:
Log rotation is checked frequently (once per minute now, might change before release)
Rotated logs are compressed with
bzip2by default, this may eventually be configurable
Log display/searching/etc in the GUI includes content from rotated logs automatically
Existing log files ARE NOT converted on upgrade, since we will no longer be including the
- On upgrade, log files will be reset to empty plain text files.
The intent is to have more GUI settings to allow fine-tuning rotation size/timing but that is not yet finished.
For package authors:
- The package
loggingmethods are still there and should be used if possible (see haproxy & tinc for examples), good for basic logging+rotation+gui display under Status > Package Logs
- There are compatibility functions for
return_clog()since those were renamed to change
log. If your package uses these functions, please start using the new names on 2.5.0, or put in a version/function test to choose the appropriate one.
- For more advanced logging needs for packages:
syslogconfig files can be placed in
/var/etc/syslod.d/with unique filenames -- these will be loaded, but not touched by pfSense
newsyslogconfig files can be placed in
/var/etc/newsyslod.conf.d/with unique filenames -- these will be loaded, but not touched by pfSense
- The package
The changes are working for me locally but there will undoubtedly be some hiccups, especially with rotation and figuring out the exact parameters needed for each daemon being logged.
In addition to being a lot easier for other utilities to work with logs, this is also easier for packages to setup their own log settings and rotation needs, and hopefully puts an end to situations where package logs end up filling drives.
cloghas been retired, there are a few things that might take some getting used to for those who have been accustomed to working with clog.
First, when a log reaches the maximum size, it is rotated and compressed, and a fresh log file is created. To read log messages from before the rotation, look in the older log files. When using the GUI, this is all automatic -- The GUI pages look back into the rotated log files automatically when displaying entries and even when searching. This is handled transparently and the user does not need to know how the logs are stored.
From the shell, users can look at the first log file without any special handling. Text utilities such as
grep, and so on will work as expected. To look in the compressed copies, there are shell utilities which match the chosen compression type for basic operations. For example, the default compression type is bzip2. For
cat, there is
bzgrep. The main compression binary can also be used, with output sent to stdout:
bzip2 -dc ipsec.log.0.bz2-- The exact binaries depend on the chosen compression type, but they all have similar programs available in the shell.
To make this easier, I just committed a script called
pfSlogcatwhich checks the chosen compression type and dumps all log files in the correct order, so users do not need to do all this by hand.
Following along with the log is also a little different. When
clog -fwas run it would dump the entire log and then follow the new output indefinitely. Now the
tail -Fcommand can follow the output, but it starts by only outputting a few lines rather than the whole file. In most cases this is what users expect.
When using tail, rather than using
tail -f, use
tail -Fso that if the log is rotated while it is being followed by the user,
tailcan detect the change and move to the new log.
Another note: If you are using zfs, then /var probably already has lz4 compression enabled so you can disable log compression if you like.
: zfs get compression zroot/var NAME PROPERTY VALUE SOURCE zroot/var compression lz4 inherited from zroot