crontab changes

[jrey]

So the fun started at 10:34pm with a syslog alert to my phone from the log gathering system - syslog being flooded by the netgate.

100's of log records per second being forwarded,
everything from constant
Reloading filter
Restarting OpenVPN tunnels/interfaces (I don't use OpenVPN)
Restarting IPsec tunnels (I don't have any IPSec tunnels)
filterdns Adding Action: x (type messages)
filterdns merge_config: configuration reload
constant (on the minute updates the cron has started /use/sbin/newsyslog)
limiter queues running out of space (these have been working fine with no issues)

during the next hour until about 11:30pm the system was what I would consider unstable.

No huge spikes in memory or CPU, so from that point of view everything looked "normal"

This morning I went to check the crontab, wanted to verify the run cycle of various jobs, because it certainly looked like cron was wild last night

Here is the crontab after the patch for the periodic tasks
I had grabbed a copy for reference at that time. the lines had been commented out, no issues since.

This morning however that crontab looks like this.

All the default system entries are gone and clearly the file was modified 3 minutes before the syslog flooding started.

There was no notice of anything being updated that I can see.
There are no new system patches listed or installed. Just the one from

From ff715efce5e6c65b3d49dc2da7e1bdc437ecbf12 Mon Sep 17 00:00:00 2001
From: jim-p jimp@netgate.com
Date: Wed, 22 Feb 2023 15:32:45 -0500
Subject: [PATCH] Do not try to retain FreeBSD default cron jobs. Fixes #14016

So the question is what made the unannounced change to the crontab at 10:31pm ?

and clearly it has all the default FreeBSD entries are gone.

Why did this change cause the system to let loose and become a nightmare for the next hour or so?

except for the on the minute logging that /usr/sbin/newsyslog is running and "normal" syslog volume, the system seems to back to normal.

[SteveITS]

@jrey The capital letters at the bottom, and no blank lines, are after the patch and reboot as I recall. Did your restart at that time?

[jrey]

@steveits
yes - the system was restarted at the time the patch was installed.

Verified then the changes (it commented out the 3 lines for the base periodic)
and the system has been running fine since then until whatever happened at 10:31pm . There was no restart last night either.

last one was actually
4 Days 13 Hours 32 Minutes 23 Seconds
ago (and that was me moving some stuff around)

I was just contemplating restarting it around 11:30 last night when it just settled back down, so I left it. been fine since.

The memory when I looked last night wasn't significantly different than normal, but there was a slightly larger bump at 00:15 ( I was long gone again by this point) -- but still nothing to get overly concerned about
the first small bump on this below is 22:05
the next larger but small bump is over 00:05->00:10

Memory (rather boring)

Processor (also rather boring)

something was clearly going on however with extremely elevated and what I would call random syslogging, (ie the restarting over and over of say OpenVPN, IPSec etc, which I am not using), poor dashboard performance and seemingly all triggered by the change to the crontab last night and whatever caused that.

[SteveITS]

@jrey said in crontab changes:

system was restarted at the time the patch was installed.
Verified then the changes (it commented out the 3 lines for the base periodic)

Hmm, that's not what I got from the patch. I looked after the restart. It recreated crontab (which is normal in pfSense) and the last two lines were changed as in the patch:

-		$crontab_contents .= "# " . gettext("If possible do not add items to this file manually.") . "\n";
-		$crontab_contents .= "# " . gettext("If done so, this file must be terminated with a blank line (e.g. new line)") . "\n";
+		$crontab_contents .= "# " . gettext("DO NOT EDIT THIS FILE MANUALLY!") . "\n";
+		$crontab_contents .= "# " . gettext("Use the cron package or create files in /etc/cron.d/.") . "\n";

@jrey said in crontab changes:

what made the unannounced change to the crontab at 10:31pm

I don't have a good answer for you, but to me it looks like something triggered a "run the code to recreate crontab" and it did what it was supposed to.

[SteveITS]

@jrey I went to pull one for you. This router booted at about 11:40 pm so this was just about 20 minutes after...

#
# pfSense specific crontab entries
# Created: March 4, 2023, 12:01 am
#
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin

1,31	0-5	*	*	*	root	/usr/bin/nice -n20 adjkerntz -a
1	3	1	*	*	root	/usr/bin/nice -n20 /etc/rc.update_bogons.sh
1	1	*	*	*	root	/usr/bin/nice -n20 /etc/rc.dyndns.update
*/60	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot
30	12	*	*	*	root	/usr/bin/nice -n20 /etc/rc.update_urltables
1	0	*	*	*	root	/usr/bin/nice -n20 /etc/rc.update_pkg_metadata
1	0	*	*	*	root	/bin/pkill -HUP -F /var/run/bandwidthd.pid
*/1	*	*	*	*	root	/usr/sbin/newsyslog
1	3	*	*	*	root	/etc/rc.periodic daily
15	4	*	*	6	root	/etc/rc.periodic weekly
30	5	1	*	*	root	/etc/rc.periodic monthly
0	*/2	*	*	*	root	/etc/rc.backup_rrd.sh
0	*/1	*	*	*	root	/etc/rc.backup_logs.sh
0	10	*	*	5	root	/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php dcc >> /var/log/pfblockerng/extras.log 2>&1
*/5	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php-cgi -f /usr/local/pkg/suricata/suricata_check_cron_misc.inc
*/1	*	*	*	*	root	/usr/bin/nice -n20 /sbin/pfctl -q -t snort2c -T expire 900
23	7,19	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php-cgi -f /usr/local/pkg/suricata/suricata_check_for_rule_updates.php
45	6	*	*	*	root	/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php cron >> /var/log/pfblockerng/pfblockerng.log 2>&1
#
# DO NOT EDIT THIS FILE MANUALLY!
# Use the cron package or create files in /etc/cron.d/.
#

Its HA pair was booted about 25 minutes later and is stamped "Created: March 4, 2023, 12:10 am."

[jrey]

@steveits

just as an additional point of reference

the syslog machine was recording 444 entries / second at highest point (and averaging around 350/second) during this time frame. which is what tripped the panic alarms to my phone. It was certainly a wild hour.

Normally the netgate is responsible for <10 / second (even with the dashboard running)

just going to throw this out there. Perviously in the crontab there was a FreeBSD entry for newsyslog and one in the pfSense section for /usr/sbin/newsyslog.
with the default FreeBSD one now removed and out of the way, is it possible that with only the pfSense section entry remaining, that "something" as I said above just "let loose"

doesn't answer the why did the crontab change at that time, but it might shed light onto the flooding syslog.

Everything really just looks normal again this morning, with no restart or other changes of any kind.

[jrey]

@steveits

Thanks, so I think what we are saying is that when the patch was applied and system restarted, it doesn't appear to have done what was expected)

(I never really looked that close at what it was doing specifically, but yes I agree is didn't rewrite a new file.)

Then after the reboot 4 days ago, it also did not modify the file.

there has been no restart in the past 4 days and certainly not one last night.

"but to me it looks like something triggered a "run the code to recreate crontab" and it did what it was supposed to."
agreed - but after pervious restarts and many days after the patch originally being installed. (and at such a seemingly random time)

beware the Ides of March (or late Friday night in this case)

Thanks again for your feedback

[SteveITS]

@jrey I only looked because I was expecting it to comment out the periodic daily line but it didn’t. Then it rewrote the file at boot, still I commented, so I pulled up the patch details. no memory spike the next day so it must have worked.

Perhaps crontab write triggers again at other criteria?