Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    crontab changes

    Scheduled Pinned Locked Moved General pfSense Questions
    8 Posts 2 Posters 611 Views 2 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J Offline
      jrey
      last edited by

      So the fun started at 10:34pm with a syslog alert to my phone from the log gathering system - syslog being flooded by the netgate.

      100's of log records per second being forwarded,
      everything from constant
      Reloading filter
      Restarting OpenVPN tunnels/interfaces (I don't use OpenVPN)
      Restarting IPsec tunnels (I don't have any IPSec tunnels)
      filterdns Adding Action: x (type messages)
      filterdns merge_config: configuration reload
      constant (on the minute updates the cron has started /use/sbin/newsyslog)
      limiter queues running out of space (these have been working fine with no issues)

      during the next hour until about 11:30pm the system was what I would consider unstable.

      No huge spikes in memory or CPU, so from that point of view everything looked "normal"

      This morning I went to check the crontab, wanted to verify the run cycle of various jobs, because it certainly looked like cron was wild last night

      Here is the crontab after the patch for the periodic tasks
      I had grabbed a copy for reference at that time. the lines had been commented out, no issues since.

      Screen Shot 2023-03-04 at 9.47.22 AM.png

      This morning however that crontab looks like this.
      Screen Shot 2023-03-04 at 10.12.31 AM.png

      All the default system entries are gone and clearly the file was modified 3 minutes before the syslog flooding started.

      There was no notice of anything being updated that I can see.
      There are no new system patches listed or installed. Just the one from

      From ff715efce5e6c65b3d49dc2da7e1bdc437ecbf12 Mon Sep 17 00:00:00 2001
      From: jim-p jimp@netgate.com
      Date: Wed, 22 Feb 2023 15:32:45 -0500
      Subject: [PATCH] Do not try to retain FreeBSD default cron jobs. Fixes #14016

      So the question is what made the unannounced change to the crontab at 10:31pm ?

      and clearly it has all the default FreeBSD entries are gone.

      Why did this change cause the system to let loose and become a nightmare for the next hour or so?

      except for the on the minute logging that /usr/sbin/newsyslog is running and "normal" syslog volume, the system seems to back to normal.

      S 1 Reply Last reply Reply Quote 0
      • S Offline
        SteveITS Rebel Alliance @jrey
        last edited by

        @jrey The capital letters at the bottom, and no blank lines, are after the patch and reboot as I recall. Did your restart at that time?

        Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
        When upgrading, allow 10-15 minutes to reboot, or more depending on packages, and device or disk speed.
        Upvote ๐Ÿ‘ helpful posts!

        J 1 Reply Last reply Reply Quote 0
        • J Offline
          jrey @SteveITS
          last edited by

          @steveits
          yes - the system was restarted at the time the patch was installed.

          Verified then the changes (it commented out the 3 lines for the base periodic)
          and the system has been running fine since then until whatever happened at 10:31pm . There was no restart last night either.

          last one was actually
          4 Days 13 Hours 32 Minutes 23 Seconds
          ago (and that was me moving some stuff around)

          I was just contemplating restarting it around 11:30 last night when it just settled back down, so I left it. been fine since.

          The memory when I looked last night wasn't significantly different than normal, but there was a slightly larger bump at 00:15 ( I was long gone again by this point) -- but still nothing to get overly concerned about
          the first small bump on this below is 22:05
          the next larger but small bump is over 00:05->00:10

          Memory (rather boring)
          Screen Shot 2023-03-04 at 10.52.44 AM.png

          Processor (also rather boring)

          Screen Shot 2023-03-04 at 10.59.21 AM.png

          something was clearly going on however with extremely elevated and what I would call random syslogging, (ie the restarting over and over of say OpenVPN, IPSec etc, which I am not using), poor dashboard performance and seemingly all triggered by the change to the crontab last night and whatever caused that.

          S 2 Replies Last reply Reply Quote 0
          • S Offline
            SteveITS Rebel Alliance @jrey
            last edited by

            @jrey said in crontab changes:

            system was restarted at the time the patch was installed.
            Verified then the changes (it commented out the 3 lines for the base periodic)

            Hmm, that's not what I got from the patch. I looked after the restart. It recreated crontab (which is normal in pfSense) and the last two lines were changed as in the patch:

            -		$crontab_contents .= "# " . gettext("If possible do not add items to this file manually.") . "\n";
            -		$crontab_contents .= "# " . gettext("If done so, this file must be terminated with a blank line (e.g. new line)") . "\n";
            +		$crontab_contents .= "# " . gettext("DO NOT EDIT THIS FILE MANUALLY!") . "\n";
            +		$crontab_contents .= "# " . gettext("Use the cron package or create files in /etc/cron.d/.") . "\n";
            

            @jrey said in crontab changes:

            what made the unannounced change to the crontab at 10:31pm

            I don't have a good answer for you, but to me it looks like something triggered a "run the code to recreate crontab" and it did what it was supposed to.

            Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
            When upgrading, allow 10-15 minutes to reboot, or more depending on packages, and device or disk speed.
            Upvote ๐Ÿ‘ helpful posts!

            1 Reply Last reply Reply Quote 0
            • S Offline
              SteveITS Rebel Alliance @jrey
              last edited by

              @jrey I went to pull one for you. This router booted at about 11:40 pm so this was just about 20 minutes after...

              #
              # pfSense specific crontab entries
              # Created: March 4, 2023, 12:01 am
              #
              SHELL=/bin/sh
              PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
              
              1,31	0-5	*	*	*	root	/usr/bin/nice -n20 adjkerntz -a
              1	3	1	*	*	root	/usr/bin/nice -n20 /etc/rc.update_bogons.sh
              1	1	*	*	*	root	/usr/bin/nice -n20 /etc/rc.dyndns.update
              */60	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot
              30	12	*	*	*	root	/usr/bin/nice -n20 /etc/rc.update_urltables
              1	0	*	*	*	root	/usr/bin/nice -n20 /etc/rc.update_pkg_metadata
              1	0	*	*	*	root	/bin/pkill -HUP -F /var/run/bandwidthd.pid
              */1	*	*	*	*	root	/usr/sbin/newsyslog
              1	3	*	*	*	root	/etc/rc.periodic daily
              15	4	*	*	6	root	/etc/rc.periodic weekly
              30	5	1	*	*	root	/etc/rc.periodic monthly
              0	*/2	*	*	*	root	/etc/rc.backup_rrd.sh
              0	*/1	*	*	*	root	/etc/rc.backup_logs.sh
              0	10	*	*	5	root	/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php dcc >> /var/log/pfblockerng/extras.log 2>&1
              */5	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php-cgi -f /usr/local/pkg/suricata/suricata_check_cron_misc.inc
              */1	*	*	*	*	root	/usr/bin/nice -n20 /sbin/pfctl -q -t snort2c -T expire 900
              23	7,19	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php-cgi -f /usr/local/pkg/suricata/suricata_check_for_rule_updates.php
              45	6	*	*	*	root	/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php cron >> /var/log/pfblockerng/pfblockerng.log 2>&1
              #
              # DO NOT EDIT THIS FILE MANUALLY!
              # Use the cron package or create files in /etc/cron.d/.
              #
              

              Its HA pair was booted about 25 minutes later and is stamped "Created: March 4, 2023, 12:10 am."

              Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
              When upgrading, allow 10-15 minutes to reboot, or more depending on packages, and device or disk speed.
              Upvote ๐Ÿ‘ helpful posts!

              J 1 Reply Last reply Reply Quote 0
              • J Offline
                jrey
                last edited by

                @steveits

                just as an additional point of reference

                the syslog machine was recording 444 entries / second at highest point (and averaging around 350/second) during this time frame. which is what tripped the panic alarms to my phone. It was certainly a wild hour.

                Normally the netgate is responsible for <10 / second (even with the dashboard running)

                just going to throw this out there. Perviously in the crontab there was a FreeBSD entry for newsyslog and one in the pfSense section for /usr/sbin/newsyslog.
                with the default FreeBSD one now removed and out of the way, is it possible that with only the pfSense section entry remaining, that "something" as I said above just "let loose"

                doesn't answer the why did the crontab change at that time, but it might shed light onto the flooding syslog.

                Everything really just looks normal again this morning, with no restart or other changes of any kind.

                1 Reply Last reply Reply Quote 0
                • J Offline
                  jrey @SteveITS
                  last edited by

                  @steveits

                  Thanks, so I think what we are saying is that when the patch was applied and system restarted, it doesn't appear to have done what was expected)

                  (I never really looked that close at what it was doing specifically, but yes I agree is didn't rewrite a new file.)

                  Screen Shot 2023-03-04 at 12.30.50 PM.png

                  Then after the reboot 4 days ago, it also did not modify the file.

                  there has been no restart in the past 4 days and certainly not one last night.

                  "but to me it looks like something triggered a "run the code to recreate crontab" and it did what it was supposed to."
                  agreed - but after pervious restarts and many days after the patch originally being installed. (and at such a seemingly random time)

                  beware the Ides of March (or late Friday night in this case)

                  Thanks again for your feedback

                  S 1 Reply Last reply Reply Quote 0
                  • S Offline
                    SteveITS Rebel Alliance @jrey
                    last edited by

                    @jrey I only looked because I was expecting it to comment out the periodic daily line but it didnโ€™t. Then it rewrote the file at boot, still I commented, so I pulled up the patch details. no memory spike the next day so it must have worked.

                    Perhaps crontab write triggers again at other criteria?

                    Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                    When upgrading, allow 10-15 minutes to reboot, or more depending on packages, and device or disk speed.
                    Upvote ๐Ÿ‘ helpful posts!

                    1 Reply Last reply Reply Quote 0
                    • jimpJ jimp moved this topic from Problems Installing or Upgrading pfSense Software on
                    • First post
                      Last post
                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.