Logs Issue
-
I upgraded a unit from 2.6.0 to 2.7.0 and have noticed that sometimes when I go to check the logs it says there are no firewall logs. Seems odd. And the CPU is at 100%. The System Activity page shows roughly 17 instances of bzip2. Looks like maybe it is rotating the logs.
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 0 root -60 - 0B 1472K CPU1 1 928:53 52.98% [kernel{if_io_tqg_1}] 44982 root 97 0 22M 10M RUN 3 1:36 21.39% bzip2 -f /var/log/filter.log.5 87668 root 97 0 22M 10M RUN 3 0:28 21.19% bzip2 -f /var/log/filter.log.2 87043 root 97 0 22M 10M RUN 0 5:07 20.75% bzip2 -f /var/log/filter.log.0 17072 root 97 0 22M 10M RUN 2 3:22 20.75% bzip2 -f /var/log/filter.log.4 25675 root 97 0 22M 10M RUN 0 0:17 20.46% bzip2 -f /var/log/filter.log.1 27496 root 96 0 22M 10M RUN 3 4:33 20.36% bzip2 -f /var/log/filter.log.1 69721 root 97 0 22M 10M RUN 3 2:53 19.38% bzip2 -f /var/log/filter.log.6 80218 root 96 0 22M 10M RUN 0 0:30 19.29% bzip2 -f /var/log/filter.log.1 56783 root 96 0 22M 10M RUN 0 6:32 19.19% bzip2 -f /var/log/filter.log.5 22719 root 96 0 22M 10M RUN 2 0:19 17.68% bzip2 -f /var/log/filter.log.0 45007 root 95 0 22M 10M RUN 2 1:51 17.38% bzip2 -f /var/log/filter.log.4 51721 root 95 0 22M 10M RUN 2 3:51 17.29% bzip2 -f /var/log/filter.log.3 67307 root 95 0 22M 10M CPU0 0 1:30 16.46% bzip2 -f /var/log/filter.log.6 62735 root 94 0 22M 10M CPU3 3 0:32 14.70% bzip2 -f /var/log/filter.log.0 32044 root 92 0 13M 3576K RUN 2 147:08 11.18% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid 81496 root 63 0 145M 50M accept 2 1:48 8.98% php-fpm: pool nginx (php-fpm) 65678 root 97 0 22M 10M RUN 1 5:54 8.50% bzip2 -f /var/log/filter.log.6 95836 root 97 0 22M 10M RUN 1 3:09 8.06% bzip2 -f /var/log/filter.log.5 99551 root 92 0 22M 10M RUN 1 0:05 7.76% bzip2 -f /var/log/filter.log.3 80854 root 96 0 22M 10M RUN 1 4:05 7.37% bzip2 -f /var/log/filter.log.2Logs don't seem overly large:
-rw------- 1 root wheel 208K Oct 6 17:29 filter.log -rw------- 1 root wheel 0B Oct 6 17:29 filter.log.0.bz2 -rw------- 1 root wheel 599K Oct 6 17:28 filter.log.1 -rw------- 1 root wheel 0B Oct 6 17:29 filter.log.1.bz2 -rw------- 1 root wheel 4.5K Oct 6 17:25 filter.log.2.bz2 -rw------- 1 root wheel 4.8M Oct 6 17:25 filter.log.3 -rw------- 1 root wheel 0B Oct 6 17:28 filter.log.3.bz2 -rw------- 1 root wheel 6.2K Oct 6 17:21 filter.log.4.bz2 -rw------- 1 root wheel 8.0K Oct 6 17:21 filter.log.5.bz2 -rw------- 1 root wheel 15K Oct 6 17:25 filter.log.6.bz2Any ideas how to settle this down?
-
@Stewart what are your log rotation settings? Looks like the files are rapidly rotating, and it’s compressing the same file more than once. What kind of disk is it?
Since you upgraded I assume it’s not running ZFS, which has built in compression…
-
Yup, it's rotating too fast and cannot compress them fast enough. Increase the log size and/or how much is logged. Try disabling log compression if you have the disk space.
-
-
@Stewart Try setting log compression to None. If the logs are growing too fast and/or the CPU or I/O can't keep up it can get stuck perpetually compressing files which slows down the compression of other log files, etc.
-
Yup you've got 46G available so try just disabling compression.
-
I've disabled compression but I'm not sure why I need to. The logs shouldn't rotate until 512K in size and then it should compress pretty quickly. I'm not logging very much so I'm not sure why they are rotating so quick but when it rotates it could sit there compressing and using the CPU at 100% for at least 30 minutes. I've not seen this behavior before.
-
It's a known issue with log rotation. Some of those firewall logs are rotating at ~1min intervals which I would class as quickly rotating.
Though I can't actually find a bug for it right now.
