My Tale of an Upgrade Gone Terribly Wrong (Pfsense 2.4.x to 2.5.1 on Zotac C232)

tcarlisle

TL;DR -- don't get lax on your change & configuration management practices like I did.

I had seen the "a new version is available" in the admin UI for a few weeks, and had meant to schedule an upgrade window but hadn't gotten around to it. So last night, right before bed, I made it a point to upgrade via the UI.

I've been running Pfsense on this hardware for probably about 7 years, and the upgrades via the UI had never failed. It was so reliable that I really didn't make it a point to backup the config, etc. or have a pfsense installation thumb drive.

I noticed the upgrade was going to cross the 2.4.x to 2.5.x barrier, and my common sense as an IT professional of about 35 years did make me think this upgrade carried a little more risk.

But, I had crossed major versions before, including upgrades that uplifted the underlying FreeBSD to a new major version.

The upgrade went fine as far as the UI process was concerned. But when it came time to reboot, well, that is where things went south.

Oh, did I mention the system is headless? LOL. So, I have dig out a monitor and keyboard just to see what is going on. There were no panics or anything of that nature, but the boot process was just stuck at "/". You know, right after it says "booting".

So, I gave it some time, and decided to just cycle the power. Now the system is really mad at me. Now the boot loader is complaining it can't find some lua thing, and all have is some pre-boot loader console with little to no functionality.

I thought maybe the bios settings dumped and been set back to default, and I remembered doing a lot of stuff to the BIOS when I originally set it up. Yeah, in a perfect world all that "stuff" would have been documented.

Since it wasn't booting and was complaining about files/directories not found, I went to bios and the bootup settings, and it is set to boot UEFI. That didn't seem right, so I put it back to legacy.

Still wouldn't boot, and same problem.

So now I have to hook a PC directly up to the internet so I can download the pfsense install and prepare memstick. I figured a clean install is probably not a bad thing, though I was not looking forward to having to redo 7 years of configurations.

Now here is the part where it gets a little scary.... the pfsense install wouldn't boot from the memstick. Same thing -- complaining about things missing and that same basic console.

So I took the memstick to another PC to see if it would boot, and it did.

And yes, the pfsense box was botting from the memstick.... just not able to boot completely.

So I went to freebsd and downloaded 11.2 stable onto that memstick, and it wouldn't boot either.

After searching, I stumbled across some posts that indicate that these zotac c323 boxes need to be set to boot mode "EUFI" even if installing a *nix.

So I go into bios and put it back to "Win7 EUFI". I guess I had that set in the bios for a reason. Again.... documentation. Do as I say, not as I do, right? LOL.

But, even with the bios set to eufi, and the pfsense install memstick would boot into the install, I tried booting from the hard drive and sadly it still didn't boot.

So at this point, I decide its going to have to be the clean install path. I noticed the PFsense install has an option to recover the config.xml from another installation. Boy, that would really be nice if I don't have to re-do 7 years of configurations.

But even though I invoked that option, and pointed it to the right hard drive and partition, a console window opened and seemed to do something, but it goes away so quickly I can't read it.

I would really have felt good if I could see it say "we found a config.xml on that partition and will use it for this install".

But after playing with it and trying to read the console message and not finding success, I decided it was time to just continue and hope that config.xml would be used.

It was! Upon reboot after the installation, I was happy to be able to login to the UI and see all the firewall rules, dhcp static entries, dns resolver mappings, and all the stuff I had to do to make multiple xboxes on the internal network work right.

This took about 3 hours. It should have taken about 1 hour. If I had paused to take a copy of config.xml, and had a memstick with pfsense already prepared -- you know, all the things I used to do so that if worst case happened I could deal with it.

I can't complain about the product. I am sure that whatever went awry after the upgrade reboot, is something specific to my hardware or just a fluke. I wish the part of the installation process that lets you recover a config.xml, I wish that gave better feedback.

But beyond that, this is all on me, for not adhering to the change management practices that I've gotten paid at jobs to ensure happens. And you know, when a project team applies bad judgment and makes a simple upgrade become a 4 hr outage, when you dissect how that happened it almost always is rooted in "we didn't think it would go that badly because it has gone so well for so many times before".

For my own educational purposes, it does appear that freebsd/pfsense has been supporting uefi boot for quite some time, hence why bios was set to uefi boot. That is good to know.

This morning, I installed the updated realtek NIC driver, and at this point the system is right back where it should have been.

Anyone running PFsense on realtek NIC chipset -- if you don't use a newer driver you will probably have problems where the nic just stops working. Apparently, the freebsd 11.2 still has an older, problematic driver. I didn't want to wait and see if there were problems, because I sure there would have been.

vjizzle

I feel your pain :(. Glad it worked out for you. Be aware that the 2.5 line of releases come with their own set of problems which can result in “surprise-quality-time-with-your-firewall”. I do hope you managed to salvage a config.xml for version 2.4.5 p1 just in case...

KOM

@vjizzle Any time I upgrade, I have at hand:

backup image to reinstall if necessary
current copy of config.xml
telnet session open to the console
phone with Internet in case thing go totally sideways