Upgrade PFsense 2.7.2 to 2.8: pre-Upgrade Tasks
-
@sokeada Thanks for your suggestion ! Removing only PFBlocker woundn't be a great problem; I'd need only some time to reset rules and configs. The main ussue would be removing HAProxy because it works really fine and I set two different load balancing clusters.
As I have two PFSense 2.7.2 on my local net I'll use these as my upgrade lab before doing anything else on main PFSense firewalls on my clouds.
-
@ddepaolis when you uninstalling pfBlockerNG you may Keep Settings then after upgraded, you just install pfBlockerNG back, every configuration will be restore from you last config.
-
The safest way to upgrade is to remove packages. But it will should work without doing so. I have upgraded numerous devices without removing any packages.
The most important thing is to have some recovery method in place before hand so you can get back if the upgrade doesn't go smoothly.
Yes, all packages should retain their settings when removed. When you reinstall them they should be as before.
-
@stephenw10 Thanks for your experiences ! yep, in past so do I upgraded my PFSense firewalls many times, always without removing any additional package. Being in cloud I'll do a complete snapshot of all machine, before doing anything. In this way rollback will be quite immediate.
For now I'll test the upgrade on my local firewalls, removing only PFBlocker before beginning. Being on physical Fujitsu mini PCs, I'll save backup files of all PFSense areas, one per one.
-
Upgrading without uninstalling all packages has worked for me in the past however I have once run into an issue (2.6 to 2.7 I think) where a specific package failed to automatically reinstall on the final phase and I was not notified that it failed in the notification centre.
Only when looking through the list of services in the menu did I notice one was missing. (I don't recall which one it was but I think it was Avahi)
Checking the main Freebsd system log did show an error when it tried to install the package. I believe it was triggered by a mistake in the metadata for the package. (the name of the package it was trying to automatically install was incorrect)
All I did to fix it was manually install the missing package.
So my advise if you go the route of not uninstalling the packages is to at least make a list of which packages you have installed and check they're all there afterwards!
Uninstalling all packages and reinstalling them afterwards doesn't lose any configuration as others have noted, however it does significantly increase downtime and can lead to undesirable situations such as web content filtering / dns blocking etc not being in place for an extended period of time while users still have access.
-
@DBMandrake Thanks for your experience as well !
I completed first update of my local PFsense firewall from 2.7.2 to 2.8 (standby node)
I only uninstalled PFBlocker before beginning the upgrade. As my Fujitsu mini PC is quite "old" (Version: V4.6.4.1 R1.14.0 for D3003-A1x - Release Date: Fri Jan 27 2012) it took quite 13 minutes to complete reboot after upgrade.
After I completed PFSense upgrade to 2.8 I upgraded as well all additional packages, one per one (ntopng, openvpn-client-export, snort and HAProxy). Everyone completed the upgrade fine. At the end I re-installed PFBlocker as well, to complete my original setup. Recovering all my PFBlocker configs as well.
So everything went fine, more or less... But I discovered probably a bug in Squid, when I wanted to add it as well... I'm going to describe this ussue on a new thread in "Cache/Proxy" area
-
Well,
After upgrading 3 of 4 PFSense systems I have from 2.7.2 to 2.8.0 my opinion has shifted to - backup, fresh install and restore backup - as two of the systems had significant issues. Most were package related, but there were some other issues as well.
Unfortunately I didn't document all the specifics so the summary below is from memory before I forget all the details.
System 1 - a small system at home with limited features enabled including Squid/Squid Guard - upgraded from 2.7.2 to 2.8.0 without issues, this made me comfortable that upgrade in place would probably be OK as it has been in the past.
System 2 - Main firewall at work, lots of packages installed including Squid/Squidguard, upgraded after midnight via a remote connection via another firewall with KVM access on the hardware including the ability to remote mount media, which came in handy...
Initially tried upgrade in place, this seemed to go smoothly at first, however Squid and SquidGuard were not working afterwards and would not start, no problem, uninstall the packages and reinstall them - still not starting.
I had a quick look at /var/log/system.log and could see the Squid process crashing. I remembered reading a thread here a few days ago where Squid was crashing due to left over shared libraries from an older version of PFSense - this would make sense as this box has been upgrading in place since 2.6.0.
I didn't dig any deeper and decided rather than trying to fix this and hope there weren't any other problems lurking it was time for a fresh install directly to 2.8.0 as I had also read that the ZFS boot configurations have changed a bit (for the better) but only if you do a fresh install.
So I mounted the iso for the netgate installer via remote KVM and was successful installing. However when it came time to restore the backup some packages failed to auto-reinstall correctly.
Squid and Squidguard were listed in the Services menu but didn't seem to be installed - udpbroadcastrelay was listed as installed but /udpbroadcastrelay/udpbroadcastrelay.php was 404 not found when trying to click the link on the services menu. There were a couple of other minor package related problems as well.
My conclusion is that package auto-reinstall on a fresh install after a backup is restored is pretty flaky. In the end I had to manually go through every installed package and check the UI was accessible, settings looked OK and that it was actually running.
Squid, SquidGuard, udpbroadcastrelay and one other package I've forgotten (maybe PFBlocker ?) had to be uninstalled and reinstalled to get them working again.
I would liked to have done more troubleshooting of the initial problems but when it's your main firewall and you're working on it after midnight you just want it working properly so fresh install it is.
System 3 is a backup/secondary system with a modest number of packages installed including Squid/Squidguard, and the initial upgrade process failed after the first reboot, seemingly.
It did the big package download then the reboot then came back up, but it was not properly on the new version, notice it says FreeBSD 15, but still version 2.7.2:
So it seems the initial update of the underlying FreeBSD system and reboot succeeded but the upgrade of PFSense packages must have failed. No errors displayed in the GUI, so no indication of problems outside of manually looking through /var/log/system.log.
The GUI also seemed confused about whether the system was up to date and what version it should be on, with it stubbornly insisting it was up to date and not letting me retry the upgrade:
I think I eventually changed the wanted version in Update Settings back to 2.7.2, saved, then changed it back again and saved and then it let me upgrade again.
It seemed to go through the same upgrade process again and after the reboot now reported 2.8.0 and was mostly working, but once again I was having an issue with Squid crashing which uninstalling and reinstalling the squid package did not fix - so probably the shared library issue again.
This system is only a standby system so I thought fine, lets do a clean install here as well, as the downtime won't affect anyone. However I had a lot of trouble the netgate installer on this system.
Despite it being in the same equipment rack as System 2 connected to the same switches, the installer kept failing claiming it couldn't contact the Netgate servers... I tried three times with a reboot and reconfigure in between and was about to go get a 2.7.2 thumb drive when it decided to work...
A bit concerning that there might have been an issue with the servers hosting the image that the netgate installer needed to download as that would prevent me from reinstalling version 2.8.0 as both the upgrade process and the netinstaller rely on netgate servers to be reachable.
After restoring the backup I had the same problem with Squid/Squidguard as system 2 - it was listed in the menu but it did not get automatically reinstalled.
This time I used the "reinstall packages" button in the backup/restore section, (I don't remember seeing that before - is that new in 2.8.0 ?) and this installed the missing/zombie packages and after this the system was working again. (In hindsight this may have fixed the packages I manually uninstalled and reinstalled on System 2 as well)
So all in all a pretty low success rate for upgrade in place with multiple weird issues that ultimately made me to just do a fresh install and restore, and even then the restore process had problems with automatically reinstalling all the packages that were listed as installed and configured in the backup.
By far the worst upgrade in place experience I've had all the way back since 2.6.0 as I've upgraded in place from 2.6.0 to 2.7.2 via all the individual versions as they came out without any real problems on multiple systems.
The final system I have to do is at a remote site and there is no remote KVM access so I'm not even going to attempt an upgrade in place, I'm going drive over there with two thumb drives - one with the net installer to attempt an install directly to 2.8.0 and another with the 2.7.2 image as a fallback.
On the plus side (I don't want to sound like I'm just grumbling and complaining) the fresh installs of 2.8.0 once the package restore issues were sorted out have been working flawlessly so far. I'm not sure if its the version bump from Squid 6.3 to 6.12 but web browsing via proxy seems MUCH snappier now with almost instantaneous page loads of lightweight websites...
Also I'm liking the updated version 6.2 of ntopng as well - it seems to be a lot more stable and usable than the version that shipped with 2.7.2, which I think was version 5.2. Although I updated mainly to stay on a current supported version, the version bumps for Squid and ntopng are appreciated.
-
Just stumbled across this notification on System 3 that I had somehow overlooked:
This occurred after a format and fresh install directly to 2.8.0 followed by restoring the configuration backup made in 2.7.2. (which had squid/squidguard installed and enabled)
I guess that explains why Squid was not installed - this seems similar to the Avahi issue I had last year where it said something similar that the package did not exist when it tried to reinstall it after upgrading from 2.6 to 2.7.0.
Is there some kind of naming discrepancy of these packages between different major versions of PFSense that can cause this ?
I'm assuming that if I restored a backup taken in 2.8.0 into a fresh install of 2.8.0 that it wouldn't happen, although I haven't tried this.
-
Hmm, those packages have never existed in the pfSense repos. But I see that they do exist in the 3rd party pf2ad repo.
It looks like your pfSense install was using that at some point and pulled in those pkgs. But since they don't exist in the official repos it was unable to reinstall them.
-
@stephenw10 Aha! Well spotted.
Yes I did try a third party build of Squid/SquidGuard about a year ago, only on this device which is a testing / spare hardware unit.
I thought I had uninstalled that and reinstalled the standard versions though - would there still be some reference to these other versions in the config xml ?
Unfortunately the device was formatted so no way to go back and check if those 3rd party packages were indeed still installed, also I encrypted the config backup - is there any way to decrypt it to look at the xml in a text editor without actually restoring it to a device ?
-
@stephenw10 Ok I recovered a decrypted backup from the auto config backup service and you're right - it was the 3rd party Squid and SquidGuard installed at least according to the package section of the config xml. (Different "internal name") My bad.
Do you think that could have caused the partial failure to upgrade where it reported FreeBSD 15 but remained on PFSense 2.7.2 ?
-
Yes. If there was a custom repo present on the system it likely would have overridden the system repo even after it was updated. So you may have ended up with the Core repo correctly seeing 2.8 and the custom repo for everything else.
In general using custom pkg repos is a bad idea. But if you choose to do it (for anyone else reading this) be aware of the risks:
https://docs.netgate.com/pfsense/en/latest/recipes/freebsd-pkg-repo.html -
I found the safest way to upgrade was to clone my drive using Clonezilla to a drive connected by USB. Then proceed with the upgrade using the test drive. If it works, then repeat with production pfSense.
Another option, is to backup your full config:
Then use fresh 2.8 USB stick to wipe and install 2.8.0. After booting to a new fresh install, use the backup config to restore. (i have not tried this method, but I was told in the forums here it would work)
-
Yup a clean install and restore will always work. Having that prepared and available is always a good idea.
