HEADS UP: Logging changes: clog deprecated, plain text logs + newsyslog rotation in next snapshots

[jimp]

Maintaining clog patches and dealing with clog file format/output is cumbersome and not a strict requirement now that we have moved beyond the age of embedded/nanobsd.

https://redmine.pfsense.org/issues/8350

I have made changes to remove clog from the system and from here on, the log files will be plain text, with default rotation configured at the previous set log sizes.

A few notes:

• Log rotation is checked frequently (once per minute now, might change before release)

• Rotated logs are compressed with bzip2 by default, this may eventually be configurable

• Log display/searching/etc in the GUI includes content from rotated logs automatically

• Existing log files ARE NOT converted on upgrade, since we will no longer be including the clog binary.

  • On upgrade, log files will be reset to empty plain text files.

• The intent is to have more GUI settings to allow fine-tuning rotation size/timing but that is not yet finished.

• For package authors:

  • The package logging methods are still there and should be used if possible (see haproxy & tinc for examples), good for basic logging+rotation+gui display under Status > Package Logs
  • There are compatibility functions for dump_clog(), dump_clog_no_table(), and return_clog() since those were renamed to change clog to log. If your package uses these functions, please start using the new names on 2.5.0, or put in a version/function test to choose the appropriate one.
  • For more advanced logging needs for packages:
    • Additional syslog config files can be placed in /var/etc/syslod.d/ with unique filenames -- these will be loaded, but not touched by pfSense
    • Additional newsyslog config files can be placed in /var/etc/newsyslod.conf.d/ with unique filenames -- these will be loaded, but not touched by pfSense

The changes are working for me locally but there will undoubtedly be some hiccups, especially with rotation and figuring out the exact parameters needed for each daemon being logged.

In addition to being a lot easier for other utilities to work with logs, this is also easier for packages to setup their own log settings and rotation needs, and hopefully puts an end to situations where package logs end up filling drives.

[jimp]

Now that clog has been retired, there are a few things that might take some getting used to for those who have been accustomed to working with clog.

First, when a log reaches the maximum size, it is rotated and compressed, and a fresh log file is created. To read log messages from before the rotation, look in the older log files. When using the GUI, this is all automatic -- The GUI pages look back into the rotated log files automatically when displaying entries and even when searching. This is handled transparently and the user does not need to know how the logs are stored.

From the shell, users can look at the first log file without any special handling. Text utilities such as cat, grep, and so on will work as expected. To look in the compressed copies, there are shell utilities which match the chosen compression type for basic operations. For example, the default compression type is bzip2. For cat, there is bzcat. For grep there is bzgrep. The main compression binary can also be used, with output sent to stdout: bzip2 -dc ipsec.log.0.bz2 -- The exact binaries depend on the chosen compression type, but they all have similar programs available in the shell.

To make this easier, I just committed a script called pfSlogcat which checks the chosen compression type and dumps all log files in the correct order, so users do not need to do all this by hand.

Following along with the log is also a little different. When clog -f was run it would dump the entire log and then follow the new output indefinitely. Now the tail -F command can follow the output, but it starts by only outputting a few lines rather than the whole file. In most cases this is what users expect.
When using tail, rather than using tail -f, use tail -F so that if the log is rotated while it is being followed by the user, tail can detect the change and move to the new log.

[jimp]

Another note: If you are using zfs, then /var probably already has lz4 compression enabled so you can disable log compression if you like.

: zfs get compression zroot/var 
NAME       PROPERTY     VALUE     SOURCE
zroot/var  compression  lz4       inherited from zroot