Upgrading fails, is it just ME?
-
Every time I've attempted a pfsense upgrade I crash and burn. So much so that I always create xml settings backup files after any changes and I turn off automatic background updates, which have caused hangs and broken my firewall in the past.
Here's the general scenario: The Webgui is setup with my password and network IP. Pfsense is configured to only allow data over my VPN service, nothing going direct to my ISP with firewall settings to ensure this happens. By setting my password in the webgui I can tell if I'm logging into a vanilla version of pfsense when the default PWs are accepted, or into my configuration.
In this latest example of failure I start with v 2.5 which has been pretty stable and I decided holding my breath to update to v2.6.0. The update download starts, then gets to the infinite loop cycle of rebooting in X seconds but doesn't and the console shows nothing. I think it's trying to connect to the internet but hasn't got my ISP and VPN credentials to do so?
At this point after an hour it's time to give up, crash out and reboot. But then the webgui settings are sometimes lost and I lose remote access, having to drag the box to a pc and go into the Console to get access. Fortunately, I've done this many times and have a USB, keyboard, mouse, USB hub and a second DVI cable connection to my monitor input.
What seems to be my problem is features of an update and re-install require a working internet connection. In my case that's not there until I've setup OpenVpn and established the tunnel. In particular, updates by default refreshes package information which I can't stop and without internet access it just hangs.
So, my nuclear option is to create yet another bootable USB flash with v 2.6.0 and do a clean install. That works but I'm not going to start reconfiguring the the vanilla install with my ISP and VPN credentials all over again. Fortunately, the only good news is v2.6.0 still accepts configuration data from the V2.5 xml backup file - at least that's good!
But then pfsense wants to update packages itself with no opt out choice- I only have 2 installed! I switch to console mode thinking it will tell me everything about background processes and it doesn't, justs sits there with 'login successful'. But if you can get in via the webgui and look at traffic flow, you can see data activity on WAN and LAN.
At one stage I hit update packages because I saw no progress activity message on the console. Disaster! What seems to happen is WAN and LAN data rates drop down to a few Kb/seconds. Even with only 2 packages, that's going to take hours to finish?
So you try to stop the package update, and after each reboot pfsense defaults to auto update packages and goes into the same seemingly never ending loop which you can't escape from or see Console progress info for.
As long as my previously saved xml configuration backup still works, the only way I've found to get around these troublesome updates via the webgui is a clean re-install from flash, import the previous saved configuration (if they work!) . Never click re-install packages, monitor data activity on interfaces and let package update re-install the packages it knows about from the last configuration file.
I would expect a flawless Webgui update to save the configuration is saved at the start of the upgrade process and the upgrade procedure considers the webgui and internet access credentials will be lost until pfsense reboots, but can then return with the original firewall configuration, web gui and internet (including VPN) credentials present. This isn't a new idea because flashing routers works by holding a background copy of the original firmware image only erasing it after the new image CRC has been checked.
Perhaps it's too much to ask for, but if an update first saved the configuration to a protected partition, that would not get formatted or overwritten by a usb flash re-install, there would be something to recover from?
-
@voxmagna1 A good backup plan is always be prepared to reinstall completely. In the case from <2.6 to ≥2.6 you will have ZFS as the default installation format. Additionally cleaning up the installation from scratch does help clear out any orphaned files that either were missed or could not be removed in the previous updates. There are downsides, of course, like side-loaded packages, customized files, etc., that are not part of the XML backup.
I've rarely had an update fail to initiate for anything other than DNS reasons. Sometimes files write wrong, write to a bad sector, etc. and those get resolved with a reinstallation from scratch -- not ideal if you're doing the upgrade remotely! I have one firewall that rarely takes an upgrade without complaining so I do its updates on site 100% of the time -- a royal pain when it's a box that has no console port and only video out. I suspect this particular system has bad components (it was a unit shipped directly from overseas and is not a Netgate device).
Basically... YMMV depending on the quality of the componentry you are using.
-
@rcoleman-netgate Thanks. Mine is a simple home network application with LAN interfaces for the local network and a second interface for a Smart TV on a different subnet. The WAN connects via openVPN and the firewall rules are locked down so nothing bypasses the VPN except a Smart TV on the subnet and nothing leaks from LAN to TV and vice vera. I'm using DNS over TLS to Cloudflare.
Whichever way you choose to update you appear need a logged in internet service? That won't happen for me until pfsense gets my VPN login credentials to establish the openvpn tunnel? When it does get this from a previously saved config presumably it sees what packages I have then has to go and get them? But there's nothing to tell you it's downloading. However, I did find going to the dashboard if the webgui lets me in that processor activity, memory (increasing) and traffic on the WAN is a helpful indicator, but there's nothing on the console screen, yet it rolls through everything else.
I now have to solve another problem which seems Firefox browser related. That and my AV software wants to see a trusted certificate when the pfsense webgui logs in. I've ignored errors in the past and accepted the risks they tell me, but discovered a nasty happening:
When I save an XML backup configuration, the filename created with pfsense appears as a file on my HDD but now has zero bytes! That is really bad news the next time I may need to import the config backup. Chrome isn't doing this. Moral is, check the XML file size after a config backup save! If pfsense read the file it saved during backup to HDD and verified it, you would know. It can't be hard to do.
-
@voxmagna1 said in Upgrading fails, is it just ME?:
Whichever way you choose to update you appear need a logged in internet service?
if you want to do an online upgrade, yes. There is no offline upgrade option except to reinstall completely from the image.
@voxmagna1 said in Upgrading fails, is it just ME?:
When I save an XML backup configuration, the filename created with pfsense appears as a file on my HDD but now has zero bytes!
Either try a different build, disable plugins or don't use Firefox. It's almost certainly specifically an issue to your system as I, and many others, user Firefox regularly with pfSense. Typically it's an add-on that cuases these types of issues so I would start there.
As for needing a VPN connection to get online to do the update... that's out of our hands and solely in yours, I'm afraid.
-
@rcoleman-netgate said in Upgrading fails, is it just ME?:
As for needing a VPN connection to get online to do the update... that's out of our hands and solely in yours, I'm afraid.
It would be the same problem for an internet connection being established to an ISP.
The ISP login credentials could come from the last good backup if that's used but when pfsense reboots, it sits there waiting 20 seconds and 20 seconds ad infinitum because I think it can't re-establish the internet connection and if it does it's downloading packages for which the screens say you might have to wait a couple of hours, then cancel the package update?
-
@voxmagna1 "disable plugins or don't use Firefox. It's almost certainly specifically an issue to your system as I, and many others, user Firefox regularly with pfSense."
Firefox is very good but they keep enhancing its security which we should't complain about. The problem I have and solved myself is pfsense installs with HTTPS default Webconfigurator. That's fine unless you have a self signed certificate which FF objects to. I want to avoid creating a trusted certificate which has to be installed on each host PC or browser. Additionally, Firefox default installs as using HTTPS-Only mode which is desirable for secure web browsing.
My solution is to change pfsense webconfigurator to HTTP and add the webconfigurator IP address as an exception to HTTPS mode in Firefox. But with so many using FF I bet many have already discovered this?
Now I can access pfsense config without FF or AV site security warnings and XML backup files are saved correctly.
-
@voxmagna1 The other option is to tell Firefox to accept/ignore the certificate error by clicking Advanced on that error/warning page, or adding an exception manually in Settings/Privacy and Security/Certificates.
Or add the self signed cert to your browser:
https://forum.netgate.com/topic/154187/how-to-accept-self-signed-certificates-from-pfsense-in-firefox
That may get around your a/v warning.HTTPS with a self signed cert is more secure than HTTP.
I've actually had it the other way where a/v blocks logins on HTTP sites.