Minnowboard MBT-4220 & 2.5.0
-
Hello,
I‘ve not found anything via searching, so…
When 2.5 come out, I attempted to upgrade from 2.4.5 on my MBT-4220. After the upgrade, the system would not boot.
At the time, no amount of config changes & troubleshooting could fix the issue, so I rolled back to 2.4.5 successfully. On top of that, fresh-installing 2.5.0 did not work either because the same issue appeared.
Can anyone confirm if the issue is fixed in 2.5.2? Anyone successfully upgrade tom 2.5.2 with an MBT-4220? Thanks!
-
My issue is best described here: https://forum.netgate.com/topic/161695/2-5-0-hangs-during-boot-on-mbt-4220?_=1625738885645
@TrickyT I saw the latest posts in your thread, that you had the same issue with 2.5.1. Did you happen to test 2.5.2?
-
@trickster Sorry for the slow reply -- I've been on a long trip with no internet access.
I did try upgrading to CE 2.5.2 and the result is the same (or nearly the same) as I described before. The serial console shows that the upgrade proceeds normally, with lots of updated and new modules getting installed with a few reboots along the way, and then it tries to run the new 2.5.2 kernel. The new kernel and associated kernel modules are loaded into memory and the last message to appear on the serial console is "start @ some-memory-address..." and then nothing. To me this indicates the new kernel runs into some serious problem very early on and doesn't (or can't) even emit an error message on the console.
I was forced to reinstall 2.4.5 to get a working system again. Not sure where to go from here.
-
Hmm, I tested 2.5.2 on the MBT-4220 quite extensively. Both clean installs and upgrades form 2.5.1 and 2.4.5 without a problem.
Are you running the original BIOS version? Later versions changed the board description throwing the hardware detection.
Are you running ZFS? That could be the issue. The additional space required for the i915 module on the MBT can cause the ZFS module to exhaust the available EFI staging area.
Removing either should allow it to boot.Edit: Yeah the console log on the linked thread looks excatly like what I saw when I hit that.
Steve
-
@stephenw10 This is very useful information! I'm running the BIOS that shipped with my board, purchased from Netgate on Dec 17, 2018. It is UEFI v2.60 (EDK II, 0x0010000). But I don't know if this is considered the early version or the later version. And I'm also running ZFS. I'll try a clean install w/o using ZFS and see what happens.
Thanks!
--tom
-
2.5.2 should recognise both BIOS versions. There was a time when if it had been upgraded the installer would not recognise it and add the loader lines for the i915 graphics driver. That could have come into play here but I doubt it is. Upgrading the BIOS is not something you could do accidentally.
Steve
-
Thanks @stephenw10 and @TrickyT for the very helpful replies!
I'm also running ZFS. I could always reinstall as UFS and restore from a backup to get my working system, right?
I bought my box a few years ago and never upgraded the BIOS.
By running dmidecode remotely, I got:
BIOS Information
Vendor: Intel Corp.
Version: MNW2MAX1.X64.0097.R01.1709211052
Release Date: 09/21/2017Did you get the BIOS version locally at the box? It would be nice to know if it works with X BIOS version. That'd be easier than re-installing.
Thanks!
-
@trickster said in Minnowboard MBT-4220 & 2.5.0:
I could always reinstall as UFS and restore from a backup to get my working system, right?
Yes.
If you run at the command line
kenv
this is the important line:smbios.system.product="Minnowboard Turbot D0 PLATFORM"
That's from:
smbios.bios.reldate="03/28/2017" smbios.bios.vendor="Intel Corp." smbios.bios.version="MNW2MAX1.X64.0095.R01.1703281124"
Later BIOS versions showed as:
smbios.bios.reldate="08/07/2019" smbios.bios.vendor="Intel Corp." smbios.bios.version="MNW2MAX1.X64.0101.R01.1908071815" ... smbios.system.product="Minnowboard Turbot D0/D1 PLATFORM"
2.5.2 recognises both smbios.system.product strings.
Steve
-
From kenv I'm getting:
smbios.system.product="Minnowboard Turbot D0 PLATFORM"
I'm not completely clear on your first post. In my case, the BIOS version should not be a factor. Is my best bet to switch to UFS because there is a lack of space for the ZFS module in the EFI staging area?
-
In 2.5.2 the BIOS should not be a factor at all. It will recognise both strings and load the i915 drivers. That will cause ZFS to fail.
I only mentioned it because if you had updated the BIOS and were running ZFS you might have seen behaviour inconsistent with the description before this was fixed:
https://redmine.pfsense.org/issues/9242Steve
-
@stephenw10 Thanks. That clears it up. Not having much experience with Minnowboard, I looked up upgrading the firmware. Do you have any idea why their tutorials appear to be non-existent? They have a tutorials page, but each tutorial links directly to their homepage. Same deal with the link in pfSense's Minnowboard page. On that page, there is an absolute link to https://minnowboard.org/tutorials/updating-the-firmware but it redirects to the homepage, too.
-
Been a while since I did it. I think I used the Intel support page. You can do it from the EFI shell directly.
There's no need to update it though. I only did it to test. .
Steve
-
@stephenw10 I can confirm that using UFS instead of ZFS resolves my hang-on-boot issue. Thanks for your posts about this problem and how to avoid it! In the end the solution was easy (i.e., reinstall using UFS), but ZFS had worked well for my customer since January 2019 and it just never occurred to me to suspect that it was the root cause of the problem. So I'm very happy to have finally resolved the issue.
Nevertheless, I've decided to replace the MBT-4220 with an SG-1100, at my own expense, for this customer. I felt that my reputation was beginning to take a hit (this was my second attempt to upgrade to 2.5.x for him, resulting overall in several hours of internet outage), and because I'm a small business owner and referrals and reputation mean everything to me, I'm justifying the relatively small expense of replacing his firewall with an officially branded Netgate device. I bought the MBT-4220 during the (6-month?) window after the SG-1000 was discontinued but before the SG-1100 was available and hence it was the only low cost pfSense option available from Netgate. I'll move the MBT-4220 into my test lab and make good use of it there. I haven't seen any issues as severe as hang-on-boot during s/w upgrades with officially branded Netgate hardware, so feel confident I'm making the right move.
--tom
-
Good to hear.
The MBT-4220 is significantly more powerful than the SG-1100. That may be no problem in your application but something to be aware of.
It's possible to run ZFS on the MBT if you disable the i915 driver. But that then prevents hot-plug on the HDMI port which is inconvenient for troubleshooting. So, compromises.
I added a com port on my personal device which gets around that. Serial console is just easier for me anyway.
At some point we will probably move to ZFS as the default so the MBT is something we have been discussing internally.
Steve
-
@stephenw10 I've used the internal serial port on the MBT (using an FTDI serial to USB cable) and I assume that's similar to what you did when you say "I added a com port." I agree, it is easier to use that interface and a terminal emulator rather than the HDMI/USB keyboard approach.
How would I go about disabling the i915 driver? I'm certainly willing to try try ZFS without i915/HDMI.
-
Edit /boot/loader.conf.local and comment out the lines. Like:
#i915kms_load=YES #drm.i915.enable_unsupported=1 #kern.vt.fb.modes.VGA-1=d #kern.vt.fb.default_mode="1024x768"
Only the first line really needs to be commented out but I'm not sure if the others will throws if left in. Unlikely to be a problem though.
Without that loaded you will only have a console if the HDMI is connected and active at boot.
Yup, I used the internal header and a TTL USB-serial adapter. I used a 3.5mm stereo jack since it fits in there great and the adapters are easily available.
Steve
-
@stephenw10 Nice use of that existing 1/4" hole in the MBT chassis!
-
Yeah I was pleased with that.
-
@stephenw10 I was just wondering...
When you consider the features of ZFS, is there an advantage to it over UFS when using it for pfSense on a little box like a Minnowboard? The reason I ask is because it isn't being used for storage and you can backup & restore the config at any time. -
The biggest advantage, in my personal opinion, is the increased resilience against damage in the even of power loss. Certainly in current pfSense. It also has snapshots etc which we hope to take advantage of on the long term.
-
@stephenw10 Could it be that the CE 2.5.2 installation image (available here: https://www.pfsense.org/download/) does not include some of the modules required to create a ZFS-based installation? When I try to follow the instructions given on https://docs.netgate.com/pfsense/en/latest/install/install-zfs.html I get to the step where I'm asked to select a disk and the list is supposed to include a Virtual Device Type. But no VDTs are included in the list and if I try to select the real on-board SSD disk I get an error saying no disks have been selected (0 < 1 disks).
-
Hmm, it gives you the choice between mirror and stripe though?
The ZFS installer is quite a bit different to UFS. It's sometimes unclear you have to actually select a disk to use in the VDT.
-
Yes, the stripe/mirror/raidXXX menu is shown, as described in the Netgate documentation, and I chose "stripe" . It's the following menu, Select Disks, that differs. No vtbdX (VirtIO Block Device) is listed and of the two choices that are listed, the SSD and the USB memory stick, selecting the SSD gives the error I described.
I'll try going back to my old 2.4.4 installation USB and see what happens. That's how I originally configured my MBT to use ZFS.
-
@trickyt said in Minnowboard MBT-4220 & 2.5.0:
VirtIO Block Device
You would only see that if you're installing in KVM. The docs show it because it's much easier to get screenshots that way. You should be able to select just the SSD and continue.
-
You can definitely do stripe with one disk in 2.5.2. You have to actually select the disk by pressing space though. That's not obvious. Should look like:
Steve
-
Figured the same thing out about the time you were posting your last message. Yes, must select the device using the space bar! Duh. Have now successfully partitioned with ZFS and installed 2.5.1. At the end of the installation process it gave me a chance to invoke a shell and edit /boot/loader.conf, which I did. But there were no lines there referencing the i915 kernel module. And now I'm in the same situation I started with: insufficient EFI staging area that leads to the kernel hanging just after starting. I'm thinking that the path /boot/loader.conf likely was the file on the USB drive and I need to use a different pathname for the file on the MBT's SSD drive. That make sense to you?
BTW, I added the stereo jack serial console connector today. Works GREAT! 5-stars on this idea.
--tom
-
It adds the i915 values to
loader.conf.local
for the MBT. You need to comment them out there. -
Just to wrap things up: My MBT is working fine now, running pfSense Plus 2.5.2 and configured with ZFS partitioning. So I'm confirming the process described in this thread works. I also tried installing 2.5.1 and then upgrading to 2.5.2 and my modified loader.conf.local was not changed by the upgrade process and thus the upgrade went smoothly.
One last question for @stephenw10 though: when, if ever, should I be concerned that doing an upgrade might change loader.conf.local or that the i915 kernel module might get loaded by some other config file? Then I'd be back to my original problem of the MBT hanging during boot. As far as I've determined, there's no easy way to edit loader.conf.local except during the last step of the installation process.
-
loader.conf.local should never get overwritten at upgrade. It's the installer that creates that and adds the i915 lines.
It is possible to disable loading modules manually at the loader prompt should it ever get added.
It's something I do so infrequently though I forget the exact syntax.Steve
-
Then I gotta say this option provides the best choice for MBT boards. ZFS provides more resiliently than UFS in the event of crashes or power failures and, at least in my book, the serial console is easier to deal with than a separate HDMI display and USB keyboard.
Thanks you Steve for all the insight, answers, and rapid responses. It is most appreciated!
--tom
-
Thanks @stephenw10 :)
I reinstalled 2.4.5 against the auto UFS partition scheme, then upgraded to 2.5.2, and now the MBT-4220 boots successfully. As you mentioned months ago, the issue was loading ZFS at boot caused the error. It required too much space than what was available. -
Why commenting out the line like this:
#i915kms_load=YES
and not do this:
i915kms_load=NO
Just curious...
Also, I don't seem to have ANY /boot/loader.conf.local file after 2.6.0 update...
...but am likely affected, because the system won't boot anymore since the attempt of updating to 2.6.0 -
@rcfa Either commenting out the i915kms_load variable or setting it to NO will accomplish the same thing.
The key issue issue here is that you cannot use both the ZFS file system and the i915 driver because of lack of adequate buffer space during boot.
-
@trickyt Thanks!
So is 2.6.0 auto-loading drivers based on some plug&play type thing? Because as I mentioned I don't even have the /boot/loader.conf.local file, and the /boot/loader.conf file has no mention at all of the i915kms at all.
So it seems to get loaded despite no instructions telling it to load the driver. That leaves me two options:
(re)moving the /boot/kernel/i915kms.ko so it's not found, or to create the /boot/loader.conf.local file, and explicitly set i915kms_load=NOI'm still a bit puzzled why that file is totally missing...
...particularly, since the driver is loaded despite no instructions, why would commenting out the line work, because that should be equivalent to not having the file, and yet I end up with a VGA console. -
It should not get loaded by default. The installer has to specifically detect the MBT boards and creates the loader.conf.local file in order to load it.
What board do you have? Are you are actually seeing the module loaded or just zfs failing to boot?
Steve
-
@stephenw10 I can't say what is or isn't being loaded, but since the 2.6.0 is an upgrade from 2.5.x, and I always had the VGA console AND zfs file system active before, I must assume that's the issue.
Or is there a way to get a VGA console without the i915 driver being loaded?I described the situation and posted a screenshot of where things end here:
https://forum.netgate.com/topic/170869/mbt-4220-update-to-2-6-0-failed-twice/7
Also, just to make sure: that /boot/loader.conf.local file is something that ends up in the final install, right? So it's not something the installer creates, to generate something that gets dumped into the bsd-boot partition, and then is deleted?
Because neither of my two pfSense installations (one an old DELL server, the other the MBT-4220 has a /boot/loader.conf.local file, they both just have a /boot/loader.conf file. -
Yes, you can still use the console without the i915 driver (and with ZFS) as long as it's connected at boot. The driver enables HDMI hot plug, without it you cannot just connect a screen to do some troubleshooting without rebooting.
Yes, /boot/loader.conf.local is created by the installer on the target drive when it detects a Minnowboard. So you would not expect it on a Dell box. If it's not there for you check the reported product ID:
[2.6.0-RELEASE][admin@4220.stevew.lan]/root: kenv smbios.system.product Minnowboard Turbot D0/D1 PLATFORM
That's what the installer uses to determine it's running on an MBT. That unit has an updated BIOS with a new ID.
Steve
-
@stephenw10 OK, so then all is fine in that regards, the HDMI-VGA adapter is permanently attached, so hot-plugging HDMI is really the least of my concerns.
Seeing that none of the updates ever seems to have added the /boot/loader.conf.local file, not even the 2.6.0 update, that would indicate, that something other than the conflict between zfs and i915 drivers is the problem.
But what? Now I'm really frazzled...
...because from all I could tell, the update proceeded without seemingly significant errors, just failure on boot.
I was able to boot once from the old kernel, and attempted to reinstall everything according to the troubleshooting instructions, and although there seem to have been no errors that would indicated a non-bootable system, I get these panics.
A complete log of that update session is attached.
Of course, this second update, overwrote the old kernel, so now, I can't boot that system anymore, other than booting into a rescue session from the USB stick that came originally with the MBT-4220. -
Yeah, loader.conf.local is only added at install by the installer. So if you never had it in the first place it would never have been added.
When you install 2.6 clean does it install and boot as expected?
I assume it does as UFS?Does it as ZFS if you comment out the i915 lines before rebooting after the install?
What you're hitting appears to be something unrelated to that issue.
Steve