Minnowboard MBT-4220 & 2.5.0

trickster

Hello,

I‘ve not found anything via searching, so…

When 2.5 come out, I attempted to upgrade from 2.4.5 on my MBT-4220. After the upgrade, the system would not boot.

At the time, no amount of config changes & troubleshooting could fix the issue, so I rolled back to 2.4.5 successfully. On top of that, fresh-installing 2.5.0 did not work either because the same issue appeared.

Can anyone confirm if the issue is fixed in 2.5.2? Anyone successfully upgrade tom 2.5.2 with an MBT-4220? Thanks!

trickster

My issue is best described here: https://forum.netgate.com/topic/161695/2-5-0-hangs-during-boot-on-mbt-4220?_=1625738885645

@TrickyT I saw the latest posts in your thread, that you had the same issue with 2.5.1. Did you happen to test 2.5.2?

TrickyT

@trickster Sorry for the slow reply -- I've been on a long trip with no internet access.

I did try upgrading to CE 2.5.2 and the result is the same (or nearly the same) as I described before. The serial console shows that the upgrade proceeds normally, with lots of updated and new modules getting installed with a few reboots along the way, and then it tries to run the new 2.5.2 kernel. The new kernel and associated kernel modules are loaded into memory and the last message to appear on the serial console is "start @ some-memory-address..." and then nothing. To me this indicates the new kernel runs into some serious problem very early on and doesn't (or can't) even emit an error message on the console.

I was forced to reinstall 2.4.5 to get a working system again. Not sure where to go from here.

stephenw10

Hmm, I tested 2.5.2 on the MBT-4220 quite extensively. Both clean installs and upgrades form 2.5.1 and 2.4.5 without a problem.

Are you running the original BIOS version? Later versions changed the board description throwing the hardware detection.

Are you running ZFS? That could be the issue. The additional space required for the i915 module on the MBT can cause the ZFS module to exhaust the available EFI staging area.
Removing either should allow it to boot.

Edit: Yeah the console log on the linked thread looks excatly like what I saw when I hit that.

Steve

TrickyT

@stephenw10 This is very useful information! I'm running the BIOS that shipped with my board, purchased from Netgate on Dec 17, 2018. It is UEFI v2.60 (EDK II, 0x0010000). But I don't know if this is considered the early version or the later version. And I'm also running ZFS. I'll try a clean install w/o using ZFS and see what happens.

Thanks!

--tom

stephenw10

2.5.2 should recognise both BIOS versions. There was a time when if it had been upgraded the installer would not recognise it and add the loader lines for the i915 graphics driver. That could have come into play here but I doubt it is. Upgrading the BIOS is not something you could do accidentally.

Steve

trickster

Thanks @stephenw10 and @TrickyT for the very helpful replies!

I'm also running ZFS. I could always reinstall as UFS and restore from a backup to get my working system, right?

I bought my box a few years ago and never upgraded the BIOS.

By running dmidecode remotely, I got:
BIOS Information
Vendor: Intel Corp.
Version: MNW2MAX1.X64.0097.R01.1709211052
Release Date: 09/21/2017

Did you get the BIOS version locally at the box? It would be nice to know if it works with X BIOS version. That'd be easier than re-installing.

Thanks!

stephenw10

@trickster said in Minnowboard MBT-4220 & 2.5.0:

I could always reinstall as UFS and restore from a backup to get my working system, right?

Yes.

If you run at the command line kenv this is the important line:

smbios.system.product="Minnowboard Turbot D0 PLATFORM"

That's from:

smbios.bios.reldate="03/28/2017"
smbios.bios.vendor="Intel Corp."
smbios.bios.version="MNW2MAX1.X64.0095.R01.1703281124"

Later BIOS versions showed as:

smbios.bios.reldate="08/07/2019"
smbios.bios.vendor="Intel Corp."
smbios.bios.version="MNW2MAX1.X64.0101.R01.1908071815"
...
smbios.system.product="Minnowboard Turbot D0/D1 PLATFORM"

2.5.2 recognises both smbios.system.product strings.

Steve

trickster

@stephenw10

From kenv I'm getting:

smbios.system.product="Minnowboard Turbot D0 PLATFORM"

I'm not completely clear on your first post. In my case, the BIOS version should not be a factor. Is my best bet to switch to UFS because there is a lack of space for the ZFS module in the EFI staging area?

stephenw10

In 2.5.2 the BIOS should not be a factor at all. It will recognise both strings and load the i915 drivers. That will cause ZFS to fail.

I only mentioned it because if you had updated the BIOS and were running ZFS you might have seen behaviour inconsistent with the description before this was fixed:
https://redmine.pfsense.org/issues/9242

Steve

trickster

@stephenw10 Thanks. That clears it up. Not having much experience with Minnowboard, I looked up upgrading the firmware. Do you have any idea why their tutorials appear to be non-existent? They have a tutorials page, but each tutorial links directly to their homepage. Same deal with the link in pfSense's Minnowboard page. On that page, there is an absolute link to https://minnowboard.org/tutorials/updating-the-firmware but it redirects to the homepage, too.

stephenw10

Been a while since I did it. I think I used the Intel support page. You can do it from the EFI shell directly.

There's no need to update it though. I only did it to test. .

Steve

TrickyT

@stephenw10 I can confirm that using UFS instead of ZFS resolves my hang-on-boot issue. Thanks for your posts about this problem and how to avoid it! In the end the solution was easy (i.e., reinstall using UFS), but ZFS had worked well for my customer since January 2019 and it just never occurred to me to suspect that it was the root cause of the problem. So I'm very happy to have finally resolved the issue.

Nevertheless, I've decided to replace the MBT-4220 with an SG-1100, at my own expense, for this customer. I felt that my reputation was beginning to take a hit (this was my second attempt to upgrade to 2.5.x for him, resulting overall in several hours of internet outage), and because I'm a small business owner and referrals and reputation mean everything to me, I'm justifying the relatively small expense of replacing his firewall with an officially branded Netgate device. I bought the MBT-4220 during the (6-month?) window after the SG-1000 was discontinued but before the SG-1100 was available and hence it was the only low cost pfSense option available from Netgate. I'll move the MBT-4220 into my test lab and make good use of it there. I haven't seen any issues as severe as hang-on-boot during s/w upgrades with officially branded Netgate hardware, so feel confident I'm making the right move.

--tom

stephenw10

Good to hear.

The MBT-4220 is significantly more powerful than the SG-1100. That may be no problem in your application but something to be aware of.

It's possible to run ZFS on the MBT if you disable the i915 driver. But that then prevents hot-plug on the HDMI port which is inconvenient for troubleshooting. So, compromises.

I added a com port on my personal device which gets around that. Serial console is just easier for me anyway.

At some point we will probably move to ZFS as the default so the MBT is something we have been discussing internally.

Steve

TrickyT

@stephenw10 I've used the internal serial port on the MBT (using an FTDI serial to USB cable) and I assume that's similar to what you did when you say "I added a com port." I agree, it is easier to use that interface and a terminal emulator rather than the HDMI/USB keyboard approach.

How would I go about disabling the i915 driver? I'm certainly willing to try try ZFS without i915/HDMI.

stephenw10

Edit /boot/loader.conf.local and comment out the lines. Like:

#i915kms_load=YES
#drm.i915.enable_unsupported=1
#kern.vt.fb.modes.VGA-1=d
#kern.vt.fb.default_mode="1024x768"

Only the first line really needs to be commented out but I'm not sure if the others will throws if left in. Unlikely to be a problem though.

Without that loaded you will only have a console if the HDMI is connected and active at boot.

Yup, I used the internal header and a TTL USB-serial adapter. I used a 3.5mm stereo jack since it fits in there great and the adapters are easily available.

Steve

TrickyT

@stephenw10 Nice use of that existing 1/4" hole in the MBT chassis!

stephenw10

Yeah I was pleased with that.

trickster

@stephenw10 I was just wondering...
When you consider the features of ZFS, is there an advantage to it over UFS when using it for pfSense on a little box like a Minnowboard? The reason I ask is because it isn't being used for storage and you can backup & restore the config at any time.

stephenw10

The biggest advantage, in my personal opinion, is the increased resilience against damage in the even of power loss. Certainly in current pfSense. It also has snapshots etc which we hope to take advantage of on the long term.