MBT-4220 update to 2.6.0 failed - TWICE

rcfa

I have two pfSense units, one an ancient Dell server: update to 2.6.0 worked (at least as far as I can tell) flawlessly.

Updating the Netgate MBT-4220 looked the same, except it never came back after the reboot.
As a matter of fact, it was stuck at a kernel panic after the reboot.

It was possible for a little while to get it to boot by stopping the boot at the boot menu, and selecting option 6, at which it would boot.
Then things would ALMOST work as they should: except no traffic would be routed to/from the DMZ.

I was silly enough to follow the instructions under “Upgrade troubleshooting” to reinstall everything with

pkg-static clean -ay; pkg-static install -fy pkg pfSense-repo pfSense-upgrade
pkg-static upgrade -f

which worked seemingly without major errors, except, after trying to reboot, the same issue: system panic; except now, the old kernel was also overwritten, so now not even option 6 allowed for a reboot anymore.

So, I figured, time for a re-install. I grabbed my hand-dandy USB-key with the ancient pfSense version on it, that shipped with the MBT-4220. Except, I don’t seem to be able to convince the device to boot from the USB stick. (Any hints there?)

So, why won’t this 2.6.0 release boot a MBT-4220 without a panic? Are there hardware requirements in 2.6.0 that the MBT-4220 doesn’t fulfill?

jimp

Did you install with ZFS or UFS? There is an issue with those MBT systems where they can't boot with both the ZFS modules and the video driver kernel module loaded, which might be what you're hitting. Without seeing the panic backtrace text it's hard to say that with any certainty, though.

There are other posts around which cover how to work around that:

rcfa

@jimp While I can't say that's it, it very much sounds like it, because I always had ZFS and video console active, which wasn't an issue until I guess 2.6.0 :(

Now what? I can attempt to install an older version of the OS, but I need a video console, I have no other somewhat user-friendly way of configuring/maintaining the system. And ZFS was what I was happy to use for years now, on these very systems...

Again, I had a working VGA-console with ZFS setup up and including 2.5.x, so what is the issue with 2.6.0 not allowing both to be active? This decidedly sounds like a bug, not a feature, because if up till 2.5.x can do it, why not 2.6.0?

Here’s where things panic:

rcfa

@rcfa Assuming, and it's highly likely, that I ran into the ZFS/VGA-console issue; I think I'm now caught in a chicken and egg problem: I can't edit the /boot/loader.conf.local file without being booted, but I can't successfully boot without...

...so, is there a way that one can manually disable the VGA driver, e.g. through the BIOS settings, thus preventing a successful loading of the driver, and thus getting the system to boot?

Trying to figure out a way to solve this, without a road-trip...

As for the future: Any plans to fix this, e.g. by using CoreBoot or something like that?
Attempts at fixing this are made double difficult, as the minnowboard.org web site is largely defunct, almost all the links to the the tutorial and documentation pages just link back to the main page, and the internet archive doesn't seem to have crawled many/most of these relevant pages, either.

rcfa

@rcfa One more thing that's not totally standard: I have the GPS-lure installed, and use that as a ntp time source...
...again, wasn't an issue until with 2.6.0

rcfa

@jimp So, after much ado, I managed to boot off a USB stick and mount the zfs partition, in full anticipation to edit /boot/loader.conf.local such as to not load the graphics driver.

Needless to say, this has me baffled: (from the working non MBT-4220 system, equivalent result on the the MBT-4220, but I can't copy/paste from there...)

[2.6.0-RELEASE][root@pfSense]/root: find / -name "loader.conf*" -print
/boot/defaults/loader.conf
/boot/loader.conf
/boot/loader.conf.d
[2.6.0-RELEASE][root@pfSense]

There's no /boot/loader.conf.local !!

So the instructions as per https://forum.netgate.com/topic/170008/which-netgate-devices-are-zfs-capable/8
don't really seem to apply.

stephenw10

Hmm, have you updated the BIOS on that 4220? There are two device types we have seen that differ by BIOS version. The installer uses these to recognise the board but perhaps yours is reporting something else.

The panic you saw above is not what I expect to see after trying to install and boot ZFS.

By far the easiest way to do this, if you want to run ZFS, is to choose the 'make changes' oprion at the end of the installer and edit the loader file then.

Steve

rcfa

@stephenw10 Thing is, I didn't install from scratch, but updated a stable running 2.5.x installation through the web interface.
Up until now, I always had access to the VGA console and was running zfs, only with 2.6.0 was there an issue, that was attributed to the zfs/VGA-console combination.

This is the reported UEFI BIOS version: MNW2MAX1.X64.0100.R01.1811141644

This is the reported platform:
Minnowboard Turbot D0/D1 Platform

So its the/an updated BIOS, but things were running fine under 2.5.x, just the 2.6.0 update broke things.

stephenw10

If you installed a much earlier version that code may not have been in the installer and if you didn't create loader.conf.local at that time it would not have been added.
It doesn't get created at upgrade only during install.

What you're hitting does not look like that issue.

I would install 2.6 clean and see if you're still hitting a problem.

Steve

rcfa

@stephenw10 Doing a clean install is going to be difficult...
I'm literally walking a computer novice step by step through what to do, by sending the command line commands through iMessage, and getting screen shots made with an iPhone back as a reply.
So burning a new USB stick is pretty much out of the question, particularly since the network access at that location, aside from the mobile internet on my friend's iPhone, is depending on the pfSense unit being operational.

So I have two options: reinstalling the original 2.x version that's on the USB-stick that came with the MBT-4220, getting it minimally configured for remote access (which may be a challenge in itself: how does one configure remote admin access through the CLI?) and then upgrading again to 2.6.0..
...OR...
...figuring out what is the problem, fixing the configuration through accessing the file system by means of a rescue session boot.

stephenw10

@rcfa said in MBT-4220 update to 2.6.0 failed - TWICE:

how does one configure remote admin access through the CLI?

You can use the easyrule command to add a WAN firewall rule form your source IP.
https://docs.netgate.com/pfsense/en/latest/firewall/easyrule.html

Steve

rcfa

@stephenw10 Awesome, thanks!
Is the web interface automatically available on the WAN, once the firewall lets traffic pass, or does remove admin still be enabled somehow? (Can't remember, it's been such a long time since I initially set up these systems; since then it's just been updates and minor tweaks...)

stephenw10

Yes, the webgui listens on all interfaces. You only need open a firewall rule to access it.

rcfa

In case anyone finds this having similar issues:

it was NOT the ZFS/i915 driver incompatibility, since the i915 driver was never loaded and the /boot/loader.conf.local file was never generated (continuous updates from an old install)
the actual cause for the update failure could not be determined, however configuration issues can be ruled out, as the very configuration in place before the upgrade was backed up, and restored in the end.
the solution was to install the OS from the original USB key shipped with the MBT-4220 and then update in two stages to 2.6.0, and then to restore the backed up config,
the easiest way to do this remotely was to walk someone through the install from the USB stick (remote location), and at the end guide them to enter the shell and punch in

easyrule pass wan any any any any

after which one can take over remotely with the web UI, update the OS, and then restore the backed up configuration.
Unless one's a high value target under constant attack, the few minutes during which the system is fully open to the wan, I consider rather a low risk, and the simplicity of the approach is worth it, unless one has trained staff at hand on the remote end.

Things are back up and working, same configuration, so the upgrade failure remains a mystery.

Hope this may help someone if anyone reals with the same or similar mess.

stephenw10

Hmm, you were never able to boot from the 2.6 installer then?

Weird.

rcfa

@stephenw10 I don’t know; the original USB stick I had, didn’t feel like walking a non-IT person through downloading and burning a USB stick over a iPhone tethering network connection…

It was simpler using the existing stick and updating…