Unable to install pfSense Plus 24.03
-
I need to perform a fresh install of pfSense Plus 24.03-RELEASE on an XG-1537, but the oldest version the netgate installer currently shows is 24.11. Is there a work-around to installing this version using the netgate installer or a link to offline 24.03 media?
We run two XG-1537s in an HA setup. We had major stability problems a year or two ago and just took down the secondary firewall. We've been running with our single primary firewall without any issues since. We're finally starting to get over that trauma and are looking to bring back the secondary. When we took down the secondary firewall, both devices were on 24.03, so we've been unable to update our primary since then. We're ultimately looking to update both firewalls to 26.03.
Unfortunately, the SATA m.2 ssd in the secondary appears to have completely failed and refuses to even be recognized as a drive in bios/the installer. We're replacing it with a standard 2.5" SATA ssd, since sourcing SATA m.2 ssds is extremely expensive/troublesome now. We are able to successfully install 26.03 while restoring our last backup, but the installer doesn't list 24.03 as an option.
As I understand it, CARP between 24.03 and 26.03 should continue to work between the firewalls, but the config won't sync from an older pfSense version to a newer version via xmlrpc. We've made a lot of config changes since our last backup (just before we took down the secondary firewall) and it would cause major issues if it became primary with that old config, like it would during a normal HA update.
I may be wrong, but the least troublesome path forward seems to be somehow installing the same version as on our primary (24.03) while restoring our secondary's last backup, then allowing xmlrpc to sync config. Then we can go through our normal HA update procedure.
I've also considered creating a backup from the primary firewall and restoring certain areas on the secondary, but that seems way more prone to issues.
-
@tmacalp open a TAC support request at Netgate: https://www.netgate.com/tac-support-request ("Problem Category" "Firmware Access" I guess) and ask for the 24.03 image for you XG-1537.
-
@patient0 Thanks, that sounds like a great next step. I'll do that.
-
@tmacalp why not upgrade the primary to 24.11 then install 24.11+restore on the backup?
-
Because it will take down the primary to do it I imagine. And the secondary won't have the correct config at that point.
-
@stephenw10 You're correct! We're a 24/7 facility, so we're doing our best to minimize downtime.
@patient0 Thanks again for the suggestion to reach out to TAC Support. Unfortunately, we submitted the request, but were denied. We were told 24.03 is deprecated and they will not provide us with the image.
Netgate Support's suggestion was to keep our newly revived secondary on the latest version, but to restore a current backup from our primary. Then we swap over the cabling to use the secondary the new primary. Then we update/reconfigure the previous primary permanently as the new secondary.
If we permanently swap primary and secondary, it'll be a lot of work updating our labels, documentation, physically moving devices on the rack, hostnames, oobm, etc...
I'm leaning more towards temporarily swapping primary and secondary, but just until the primary can be updated. Then I'll restore the secondary's backup again. After that, a config sync should take care of the rest.
But each of those swaps is going to cause some noticeable downtime. I'm expecting a minute or two of downtime, as long as CARP behaves. I may even be able to backup/restore the extra data to bring over the dhcp lease db to reduce some chaos. But it's certainly not ideal.
Because of all of the hassle, we're even considering incurring downtime by just taking everyone down to update the primary like @SteveITS suggested. But that will also leave us with no way to revert to our current working system if something goes wrong.
We'll hopefully figure out something, but it certainly seems more straight forward if we could just restore to 24.03.
-
Hmm, let me see if I can do anything here...
-
@stephenw10 Thanks! I'd appreciate any help. The ticket number is 46032404770.
-
@stephenw10 said in Unable to install pfSense Plus 24.03:
Hmm, let me see if I can do anything here...
@stephenw10 , sorry to prod, but it has been a few days. Is there any hope for us?
-
Yup, sorry I was trying to organize something specific. Let me see where we're at...
-
If anyone is wondering, @stephenw10 came through for us!
We were able to revive our secondary XG-1537 by installing 24.03 on it and restoring the old config from before we took it down.
We were then connected the sync port to the primary and xmlrpc synced config to it. We were able to plug in our primary lagg(vlan trunk) and wan/wan2 and CARP behaved. After reenabling the failover peer ips on each dhcp server and installing a few packages, we are back in a semi-stable state for 4 days now.
We've been pretty swamped this week, but we're building up the courage to test failover by temporarily disabling CARP on the master. If that works, we'll enter persistent maintenance mode on the primary and update the secondary, then our primary.
Since we're so out of date, the biggest jump we can make is to 25.07.1. I guess we'll get both firewalls on 25.07.1 before starting the move to 26.03.1. Wish us luck!
-
Story time: The upgrade did NOT go smoothly, but I think we're ok now.
First, we ran into trouble updating our secondary. After upgrading from 24.03 to 25.07.1, we were met with ZFS errors when rebooting. ZFS threw a zio_read error: 5, ZFS: i/o error - all block copies unavailable, Error: memory allocation error: block too big. We rebooted and the sata drive completely disappeared and dropped us to an EFI shell. Rebooting into bios verified that there was no drive detected. Another reboot and we were dropped to a mountroot prompt after failing to mount zfs:pfSense/ROOT/default. We unplugged everything for a minute, then booted uefi again and it brought us back to 24.03.
We're not very confident that this secondary will survive being rebooted. But if we shut it down and disconnect power for a minute, we seem to be able to get it to boot. We were ultimately able to choose the new 25.07.1 boot environment and complete the update.
Note that when we reinstalled, we installed in UEFI mode, but we probably should have used legacy bios boot. We may have lost bios settings while the firewall was unplugged for over a year, so it wasn't clear. Now that we've had a chance to check our primary XG-1537's settings, it boots in legacy bios mode. We may need to reinstall again using legacy bios and disable UEFI completely. Reinstallation should be much easier, now that our master is up-to-date.
After successfully updating the secondary to 25.07.1, we immediately updated to 26.03.1. After things seemed stable, we entered persistent maintenance mode on our primary and attempted to upgrade the primary to 25.07.1. The initial update seemed to go well, but the firewall got stuck when rebooting at "Updating configuration....", then timed out after 15 mins and booted back into 24.03.
It turns out we had 18GB worth of config history(17.5k 1MB xml files) in /cf/conf/backup, as mentioned here, that caused the timeout. Thanks, @stephenw10 and @SteveITS , it looks like you both also contributed to that helpful thread!
I removed old backup configs from both the 24.03 boot environment and the new "default" 25.07.1 boot environment and successfully completed the upgrade. Upgrading the primary from 25.07.1 to 26.03.1 went smoothly.
We're now just waiting to leave persistent CARP maintenance mode on the primary and resume as CARP master.
Thanks again for all of the help!
-
Epic!
-
@tmacalp said in Unable to install pfSense Plus 24.03:
18GB worth of config history
Yeah that's a sneaky one. I have it in our upgrade notes (coming from an older version, now) to check the config history page until it doesn't time out, before upgrading, but it's been long enough it's not top of mind anymore.
Privacy Policy · Cookie Policy