Firewalls Automatically upgrading from 2.4.5 to 2.5.0

Broncoman

Last night one of my firewalls upgraded on it's own from 2.4.5-RELEASE-p1 to 2.5.0-RELEASE. I did not initiate this upgrade and cannot figure out why it did it. This is the second firewall I have had this happen on. I am not seeing any options for auto upgrades in the firewalls, so I am scratching my head as to why this is happening.

Has anyone else seen this happen? Does anyone have any ideas or pointers for what to look for on this?

These firewalls are running in a vmware virtual environment. I have 30+ others that are setup the same way and I don't want to keep having this happen before I can do the upgrades on my time with a scheduled maintenance.

SteveITS

No that's not normal, I've never seen one spontaneously upgrade. Anything in the log about someone logging in just before that?

Broncoman

@steveits That was the first thing I looked for. I looked on the last one for that as well. There was no indication that it was sparked by a user. What I did find interesting is both times, this happened a little after midnight(local time).

I was able to find this in the logs at 00:21:17 CST:

php-fpm: PHPSESSION 1 open sessions left at shutdown script!Array (     [0] => #### phpsession_begin #### simplestacktrace(..) - /etc/inc/phpsessionmanager.inc:38 phpsession_begin(..) - /etc/inc/auth.inc:1964 session_auth(..) - /etc/inc/authgui.inc:33 require_once(..) - /usr/local/www/guiconfig.inc:57 require_once(..) - /usr/local/www/index.php:44     [1] => #### phpsession_end #### simplestacktrace(..) - /etc/inc/phpsessionmanager.inc:54 phpsession_end(..

The firewall was going through the upgrade process around 00:41:00

Broncoman

textdump.tar.0 - This is the dump file that was on the firewall this morning. I had one on my other firewall as well when it auto upgraded:

textdump.tar.0 - This one happened on 2/22/2021 on a different firewall in a different state(which strangely enough was also on a Monday morning.)

SteveITS

Do you have anything that monitors it? I'm thinking of an RMM plugin or something of the sort.

Can always change the password and see who complains they can't access it anymore... :)

serbus

Hello!

<118>        __
<118> _ __  / _|___  ___ _ __  ___  ___
<118>| '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \
<118>| |_) |  _\__ \  __/ | | \__ \  __/
<118>| .__/|_| |___/\___|_| |_|___/\___|
<118>|_|
<118>
<118>
<118>Welcome to pfSense 2.4.4-RELEASE (Patch 2)...
<118>

Where does this come from?

<118>>>> Upgrading necessary packages...

It looks like pfSense-upgrade.sh is the only place this string comes from. Are you running shellcmd, cron @reboot, or something else that would be calling pfSense-upgrade.sh?

John

Broncoman

@steveits I am running zabbix agent 4.0 on these firewalls. Other than that I have a backup script that runs every hour that grabs the config with a user that can't access or do anything else in the firewall. I have the user login session maxed a 3600 seconds and there are no logs in the past couple of weeks of a user logging in out of the normal. I have all logs going to a syslog server as well as on the firewall.

@serbus I don't know of any startup scripts or cron jobs that would run at reboot. I built these firewalls several years ago and have not done any weird custom settings on them.
I ran this:

grep pfSense-upgrade.sh /usr/local/etc/rc.d

I didn't get any results from that. Is there somewhere else I should look?

viragomann

@broncoman
That dump shows an upgrade from 2.4.4-p2 to 2.4.5-p1, nothing to see from 2.5.0.
Nevertheless, the upgrade should not be triggered automatically.

To avoid upgrade to 2.5 you may set the repository branch accordingly in System > Update > Update Settings:

Don't run any package update before your pfSense is up-to-date according to this setting!

serbus

@broncoman said in Firewalls Automatically upgrading from 2.4.5 to 2.5.0:

grep pfSense-upgrade.sh /usr/local/etc/rc.d

Hello!

That is the place to look.

Could the vmware tools be configured to run that script on startup?

John

Broncoman

@serbus I haven't modified anything with any of the packages. Is that something you have seen happen before?

Broncoman

@viragomann I didn't catch that. That dump showed up overnight on that firewall and the firewall was on 2.4.5_1 on the 28th.

I'll set the Branch back to 2.4.5 on my critical firewalls.

serbus

Hello!

It looks like a fresh, stock 2.4.5p1 will run pfSense-upgrade four times at reboot with the following parameters :

pfSense-upgrade -y -U -b 2
pfSense-upgrade -y -U -b 3
pfSense-upgrade -uf
pfSense-upgrade -Uc

where :

Usage: ${me} [-46bdfhnRUy] [-l logfile] [-p socket] [-c|-u|[-i|-d] pkg_name]
	-4          - Force IPv4
	-6          - Force IPv6
	-b          - Platform is booting
	-d          - Turn on debug
	-f          - Force package installation
	-h          - Show this usage help
	-l logfile  - Logfile path (defaults to /cf/conf/upgrade_log.txt)
	-n          - Dry run
	-p socket   - Write pkg progress to socket
	-R          - Do not reboot (this can be dangerous)
	-U          - Do not update repository information
	-y          - Assume yes as the answer to any possible interaction
The following parameters are mutually exclusive:
	-c          - Check if upgrade is necessary
	-i pkg_name - Install package PKG_NAME
	-r pkg_name - Remove package PKG_NAME
	-u          - Update repository information

Your system auto-upgraded at boot from 2.4.4p2 to 2.4.5p1 and then again from 2.4.5p1 to 2.5. Without timestamps it is hard to know if these boot time upgrades were minutes or years apart. Something in your config, or a bug, may have caused one of the normal pfSense-upgrade runs to upgrade or there is an extra call to the script somewhere.

It doesnt seem like a widespread problem, but I would check your other instances to see if they auto-upgraded from 2.4.4 -> 2.4.5 at some point in time, especially if they havent rebooted since 2.5 was released.

John

Broncoman

@serbus The firewall that upgraded yesterday morning was upgraded from 2.4.4p2 to 2.4.5p1 on November 16th, 2020. I triggered that upgrade from the gui. Would that be considered an auto-upgrade?

I work for a broadcast company and it looks like there was some streaming traffic going through the VPN tunnel. It also appears that some backups were triggered a little before that time. I believe that with all that going on, the firewall ran out of resources in the virtual environment which triggered the reboot. I'm concerned that it runs pfSense-upgrade at reboot though. Not sure how to keep that from happening.

skogs

Serbus might have the final solution to the random reports of auto upgrades.

Those 4 cycles are normal assuming a fresh install or upgrade. Sanely checks to see what status is and re-installs packages as necessary and such. Perhaps it has a logic that isn't always writing completion status somewhere so then randomly year(s) later might accidentally trigger a second time when it is not intended.

What file do those particular lines of pfSense-upgrade come from?

Broncoman

@skogs I'm not sure if you are asking me this or @serbus. I don't know the answer to that though.

iampowerslave

Now I know I'm not crazy.

I have my FW on a Virtualbox VM with an Immutable disk, last time I was on site I tried updating to 2.5.0 and something went wrong, it froze, so I manually restarted it, the thing is, it seems I forgot to set the drive to Immutable time ago, but it still stayed at 2.4.5_p1 after the update failed.

Time went by and last week I started having VPN issues. I've logged in to check, and it was updated to 2.5.1!

I had to get an image when it was 2.4.4 set to update to the DEPRECATED channel, updated to 2.4.5_p1 and set it to immutable. Let's see what happens.