Issues with Supermicro ISOs and reinstalling after upgrades went sideways

TheGushi

Hey all,

We have a pair of netgate 1541's that we bought as a failover pair about five years ago for our corporate HQ, and then moved to just using a single one in a different use case after going full-virtual.

We decided recently to get the other 1541 online and racked it up and had the datacenter staff just get the ILOM online, figuring we can take it from there. I configured my WAN interface via the KVM, used the command line to disable PF and used the web UI to add rules such that I could access its WAN via our corporate VPN, and started doing the long sequence of upgrades. I don't remember which release I started with, but the problem started after two or three upgrades.

Some upgrades went smoothly, others did not. When I got to 2.4.4/2.4.5ish and tried to go further, things got bad. It seems that at one point we wound up with the GUI just reporting that it was "initializing updates, please wait". Doing option 13 from the console reported that a verson of pkg was already running (ps said it wasn't).

When I tried to run pkg, I got an error about missing shared libraries like libarchive, similar to this user.

After trying to re-init with pkg-static bootstrap and pkg-static install -f pkg", we found that pkg complained of a version mismatch, with uname reporting it was a 12.3 system, and freebsd-version reporting a 11.x userland, although weirdly, if we tried to do a pkg bootstrap -f, it complained about the opposite -- about the running kernel being 11.x, mismatching userland being on 12. I'm no stranger to "pkg" being weird, but no matter what I tried, things would not play ball.

At the end of the day, the answer was to reinstall, and that's when the problems really started.

Getting the iso from the PFsense store was easy enough, but getting the supermicro ILOM to consume that iso, was a giant pain. I'm no stranger to goofy iLOMS, so I keep an old windows XP VM around to run the Java Nonsense that's sometimes required.

Note: this is in our datacenter. We don't run DHCP services on our routers, especially for the small DMZ between our router and pfsense boxes. Netbooting via DHCP and PXE is a non-starter.

Nothing worked. HTML5 KVM in Supermicro land cannot do virtual ISO's. Supermicro's IPMIView tools (which used to be on ftp.supermicro.com, and moved to and ftp-like section of www.supermicro.com) wouldn't install under my XP vm, complaining of a corrupted zip archive.

While IPMIView DID work under windows 10, I then found out that there's an additional license required to use virtual media with ipmiview, which isn't available for this board.

While you can upload a floppy .img to emulate (I guess, if you're flashing the bios via floppy?) you cannot upload an ISO file.

I was one step away from paying our datacenter to burn media and attach the USB-CDROM we keep in our cabinet, but I'm too stubborn.

No, the only answer for mounting media, for a system made in 2019, is not only to create a windows share, but an exceptionally old and insecure (circa NT4) Windows Share.

There's some documentation about what one has to do here, but this blogger comes far from saying "Okay, install samba and here's a complete smb.conf to use", so that was left up to the reader (me).

I'll be writing up a blog entry about what it took to make things work under FreeBSD 14.1, including the strategic holes I had to punch in our firewall, what my minimal smb.conf looked like, what i had to do to get a user working (the ilom says a username is optional, I found that it wasn't, and tried to use "Administrator" if you didn't fill one in), but it made me think about other possible recovery options -- what could have made this easier?

I know Netgate is a small integrator, and SuperMicro is unlikely to take much feedback from you, but I would love to know if you've tried. (For what it's worth, I have managed to get them to at least fix their SNMP MIB's, so there is a live person with a voice somewhere).

I don't know if Supermicro has made things any better recently, but I doubt it. However, since they do support the upload of a floppy image, and since the hardware is of a known type, perhaps NG could make a bootable floppy image that just phoned home to netgate to do the rest of the boot/install via gPXE or something like that (perhaps allowing the user to substitute a few variables like interface name, ip address, and the like).
In the past, with problems like this under Normal FreeBSD, I've managed to copy a base "stub" system into /boot, and boot into MFSBSD, and use that to recover the rest of my system. (At the dayjob, we dealt with far-away servers where the / partition was too small to take a modern kernel.) A pfsense "mfsbsd" installer would play super well with the above idea of just a floppy boot.
Okay, fine, if you're forced to use the mount-an-ISO-from-a-share-that-still-uses-NTLMv1 method, perhaps pfsense could put the documentation somewhere on their site as to how to configure samba to serve that file. It would have saved me some headaches.
I downloaded a whole ISO (nearly a gig, so of course you need a DVD rom, not just a cdrom). Does the installer need to be that big? Why does it need to do a net install, versus just having the packages as part of the ISO?
It seems several people have hit this problem specifically around this version -- for the uninitiated, was there a sequence of commands that would have brought me back to a running system without doing a reinstall, or was this the right answer?

-Dan

SteveITS

@TheGushi Sounds like you got it solved, congrats. That seems like quite a feat.

There is a feedback link on the doc pages to submit a ticket for doc changes/suggestions.

I have not actually used the installer yet but I was under the impression it is a generic one for any hardware including ARM models? Also one of the goals was to have one installer, and it can pull down one of a choice of versions, and Plus or CE. So one doesn't need to redownload it for every version of pfSense.

In general I've seen Netgate advise that if a router is more than a few versions old, to just reinstall. If nothing else that gets you ZFS now, but it also skips all the intermediate upgrade steps. FWIW the very few times I've run into a problem with an upgrade over the last 10ish years, it's been when it was more than a couple years old.

TheGushi

@SteveITS Not gonna lie, it felt like I was trying to get the Voyager probe to start talking again. :)

SteveITS

@TheGushi LOL that is a wonderful analogy.