Log Rotation Issue

Matt_Sharpe

So the Web UI for PFsenses state that log files have a "Log Rotation Size (Bytes) of 512000" = 500KiB. This combined with the Log Retention Count should mean no log outgrows 500KiB x 7.

We currently have a logging issue in another post (https://forum.netgate.com/topic/185283/carp-ha-logging-for-inactive-wan)

But it's still begging the question, why am I seeing logs with a size of several gigabytes when the settings above should be in play?

Is this a bug?

OR are the defaults not in use by default, unless another setting like compression is enabled? Disabled by default ?

stephenw10

Package logs may be handled differently. Which logs are you seeing that in?

Matt_Sharpe

@stephenw10 the log that I'm seeing outgrow the configuration is the system logs.

stephenw10

Hmm, OK, that's not right!

If you resave the log settings does it start rotating as expected?

That is a ludicrously huge log though. Especially if that's really over only 3hrs.

It's possible to configure the individual logs as well as the global values. The system log may have been setup to keep a far larger value.

Gertjan

@Matt_Sharpe

115 G in one day ?
Daily rotating seems to work, but what is logging that much ??

If the rotating process really would rotate as soon as the file size goes over half a Mbytes, then for 115 Gbytes it will have to rotate 230 times a day, or about 10 times per hour.
Evey six minutes pfSense whould have to shift the files, compress the newest one. Kick the syslogger process to signal the rotation ....
That's .... "not good".
If the drive is an SSD : it will not have a long lasting live .....

Again : whats logging that much ? and maybe its time make it less verbose.

Matt_Sharpe

@Gertjan the logging issue is outlined in another post. This issue only occurs on the passive node of a HA setup. The WAN interface is setup with private addresses and the Public DFG can only be contacted from the master node. Working on it, but the logs fill up on the passive node due to the arp issues via the CARP not allowing comms on passive.

Matt_Sharpe

@Gertjan @stephenw10 ideally, I'd like more options for excluding certain log entries, but it doesn't seem to be possible :(

stephenw10

With files that large and filling that quickly you might be hitting a compression issue. The system cannot compress the logs fast enough and ends up with a CPU core jammed trying to do it continually.

Try disabling log compression if it's still enabled.

Matt_Sharpe

@stephenw10 I don't have compression enabled on the impacted devices.

stephenw10

Hmm, do you have newsyslog in the crontab?

[2.7.2-RELEASE][admin@t70.stevew.lan]/root: cat /etc/crontab | grep syslog
*/1	*	*	*	*	root	/usr/sbin/newsyslog

Matt_Sharpe

@stephenw10 yes I do:

jimp

What is in the newsyslog config file?

cat /var/etc/newsyslog.conf.d/pfSense.conf

Matt_Sharpe

@jimp Output:

jimp

That all looks good... Not sure why it isn't rotating then. What happens if you run /usr/sbin/newsyslog yourself from a shell prompt? Does it produce any errors?

Also you may be able to cut down some of the log spam by checking System > Advanced, Networking tab, Suppress ARP Messages.

Matt_Sharpe

@jimp Tried enabling that setting, but the system.log.0 file is still growing quickly with the same messages:

jrey

@Matt_Sharpe

Some reading material.

https://forum.netgate.com/topic/149298/arpresolve-can-t-allocate-llinfo-for-192-168-100-1/22

https://forum.netgate.com/topic/153775/no-link-and-flood-of-arpresolve-can-t-allocate-llinfo-for-x-y-z-w-on-ix0

jimp

You could also try setting a tunable for net.link.ether.arp.log_level=0 (default is 6)

Or disable state sync since those messages are likely from that.

Matt_Sharpe

@jimp This setting appears to have stopped the log filling up.

Another query would be, is it possible to move the /var/log partition to a dedicated disk in PFsenses?

jimp

@Matt_Sharpe said in Log Rotation Issue:

Another query would be, is it possible to move the /var/log partition to a dedicated disk in PFsenses?

It's possible in FreeBSD but there isn't any supported way to do it in pfSense. Someone familiar how disks/partitions works in FreeBSD should be able to set it up without too much trouble, but it may also take some manual adjustments in the code since pfSense has to make a lot of assumptions in that area.