newsyslog cron frequency, hanging bzip2 processes, scan log files

[rb625]

I've encountered a problem on 21.05.1 and its predecessor running on two SG1100:

Throughput will slow, I'll check the GUI and see the CPU is at 100%. System Activity or ssh top will show several bzip2 processes that are hung. This takes several days/weeks to happen, seemingly only when ipsec.log needs to rollover. I think I have logging level a little higher than default on ipsec.

I am running the stock system + Zabbix agent 5.2, arpwatch, ipsec-profile-wizard, mailreport, pfBlockerNG, RRD_Summary, Service_Watchdog. mailreport and service_watchdog have nothing configured.

I have an ipsec mobile VPN server running. This works well.

I see there is a newsyslog cron job running every minute. Could this be too fast, given that I have some extra packages running? And thus cause another invocation before the previous minute's one is finished?

I am now trying disabling log compression in GUI.

Eventually, if this hanging process issue is solved, I would like to modify the cronjob from "/usr/sbin/newsyslog" to "/root/scanlogfile.php; /usr/sbin/newsyslog". See any issue with that? I want to scan ipsec.log and auth.log and get an sms on every connect or login. (VPN is for admin only)

Any thoughts?

[bmeeks]

There are two pfSense bug reports related to this, although neither matches exactly your case.

Here is the first one: https://redmine.pfsense.org/issues/12012. The fix for it is slated to be released with the upcoming 22.01 version of pfSense+ (which your SG-1100 will run).

The second one is specifically for ZFS file systems, but might have some applicability to your case: https://redmine.pfsense.org/issues/12011. This bug report, unfortunately, has the fix posted to Netgate's private GitHub repo for pfSense+, so you can't see the code changes.

You could install the Cron package on pfSense+ and then see if it offers you the opportunity to tweak the execution time of the newsyslog task. Of course any change you make might get overwritten with a reboot. It is very likely to get overwritten with any update or reinstall.

[SteveITS]

@rb625 said in newsyslog cron frequency, hanging bzip2 processes, scan log files:

disabling log compression

There have been a few posts recently about bzip processes. Similar to the Redmine issue, per the 21.09->22.01 release notes:
"The best practice is to disable Log Compression for rotation of System Logs manually for not only existing ZFS installations, but also for any system with slower CPUs. This setting can be changed under Status > System Logs on the Settings tab."
The storage on the 1100/2100 isn't that fast.

[rb625]

Steve and BMeeks, thanks for the info. Based on that, I'll stick with no log compression. These are just SOHO routers, not much traffic.

I will try upping the syslog cron job to 2 minutes and increasing log size to 1M. That will give me time for invoking my log scan utility.

My assumption is the cron period has to be short enough that a log can't grow x% beyond the max size during the period. If a log was increasing at 100 KB per minute, that would allow 20% beyond the max size of 1MB.

[rb625]

It appears this problem is related to https://redmine.pfsense.org/issues/12095#note-4. I'm now seeing ipsec.log entries like this:
Nov 28 13:16:38 3816 charon[15846]: 02[CFG] C_GetSlotInfo failed: SLOT_ID_INVALID
Nov 28 13:16:38 3816 charon[15846]: 02[CFG] error in C_WaitForSlotEvent: GENERAL_ERROR
Nov 28 13:16:38 3816 charon[15846]: 02[CFG] error in C_WaitForSlotEvent: GENERAL_ERROR

Despite setting the log size to 1MB, the ipsec logs only are about 120 MB each. The setting seems to work for other log files, limiting them to 1MB.

ps -aux reports that pcscd is running:
ps -aux | grep pcscd
root 13808 0.0 4.8 61540 48264 u0- S Sun13 1:55.24 /usr/local/sbin/pcscd
root 69967 0.0 0.2 11012 2388 0 S+ 10:34 0:00.01 grep pcscd

My CPU usage is being maxed out due to this. The bug report mentions there is a patch. Where can I get it and does it apply for 21.05.2?

BTW, I don't think that pc/sc support is a bad idea, if completely and stably implemented, so you could get a USB smart card and put the necessary certificates on that. I have used NitroKey and Yubikey products for that in high-security projects. This is a fair amount of work to get working, then test, then verify it's actually secure.

[SteveITS]

@rb625 Note 10 there links (which is unfortunately formatted as crossed out) to https://redmine.pfsense.org/issues/11933#note-7. Use the system patches package to install. Yes it needs to be done for 21.05.02. The patch was supposed to be in 21.09 which seems to be skipped in favor of 22.01. I'm a bit disappointed Netgate didn't put out a 21.05.03.

[rb625]

Steve,
Thanks for the advice. Patch seems to work, pcscd is no longer running.