SG-5100 takes over 20 minutes to boot after eMMC failure

haraldinho · Aug 6, 2022, 4:34 PM

@stephenw10 Hi Steve, it is really good to hear that there is a BIOS update in the works for the 5100. I would really appreciate if this specific issue could be taken along in the update. It does not seem to be a big thing to do and would really uncripple the devices of people with a dead eMMC. @stephenw10 is this something you can take up with the firmware folks and report back here?

stephenw10 · Aug 6, 2022, 4:44 PM

I can certainly look into it.

haraldinho · Aug 6, 2022, 5:18 PM

@stephenw10 Hi Stephen, I thought to give this little hack a try and hey presto! My boot issues are gone!! I can do a warm reboot now without any issues! Speed is also good!

So a happy camper here! The only drawback to this solution is that you have to SSH into your box to do it and it will be wiped with a clean install. So it would still be best to solve it in BIOS or with some setting in pfSense where you can disable eMMC on Netgate boxes suffering from the issue.

Thanks Stephen!

gabacho4 · Aug 6, 2022, 5:43 PM

@haraldinho what is the hack to which you refer?

EDIT: do you mean creating the file that Stephen mentioned earlier? If so, that is a great interim fix while they hopefully get something more permanent and formal put in place.

haraldinho · Aug 6, 2022, 5:52 PM

@gabacho4 the forum indentation is a bit confusing indeed. I replied to the post from @stephenw10 with the 'hack' and it then shows that post as a reply to Stephens post, but also as a new post at the bottom of the whole topic, with just a small reference to the fact that it is a reply to an earlier message. Anyway, the life saving configuration change is this:

Create the file /boot/loader.conf.local and add to it:
hint.mmcsd.0.disabled="1"

It is as simple as that :-)

The only thing I am not sure about is what happens to this setting when you do a regular upgrade to a new version of pfsense. @stephenw10 does it get preserved?

hayescompatible · Aug 6, 2022, 5:54 PM

@haraldinho said in SG-5100 takes over 20 minutes to boot after eMMC failure:

@gabacho4 the forum indentation is a bit confusing indeed. I replied to the post from @stephenw10 with the 'hack' and it then shows that post as a reply to Stephens post, but also as a new post at the bottom of the whole topic, with just a small reference to the fact that it is a reply to an earlier message. Anyway, the life saving configuration change is this:

Create the file /boot/loader.conf.local and add to it:
hint.mmcsd.0.disabled="1"

It is as simple as that :-)

The only thing I am not sure about is what happens to this setting when you do a regular upgrade to a new version of pfsense. @stephenw10 does it get preserved?

I'll have to try that later when I'm able to reboot my 5100. Sounds promising.

gabacho4 · Aug 6, 2022, 6:01 PM

Just to make sure I understand the hack right - it’s merely telling pfSense to ignore the first of the storage options being provided by the BIOS therefore causing pfSense to turn to the M.2 right away rather than trying the eMMC. Correct? If so, this seems like something Netgate could code in as part of the pfSense config and make a toggle for in the GUI (“ignore eMMC at boot”) and we’d all happily be on our way. If coded, it would be persistent across any installation/upgrade or config restoration. Could it be that simple?

haraldinho · Aug 6, 2022, 6:10 PM

@gabacho4 That's how I understand what this setting does indeed. And I guess this setting could be done in pfsense, however, I feel that it is a design flaw of these Lanner boxes and should be fixed in BIOS. But I would be happy with either ;-)

gabacho4 · Aug 6, 2022, 6:14 PM

@haraldinho fair enough point. I wonder if ALL of their devices with eMMC suffer from this. If so, the coding with GUI would make sense unless they can get BIOS updates for all the devices. If the 5100 is alone in suffering from this, the you’re right; a final fix from Lanner should be secured. Ultimately I just need a device that I can rely on 100%. I’m pretty spooked right now considering my very remote router could become (if not already) very dangerous to reboot or update. Yikes!

hayescompatible · Aug 7, 2022, 6:14 AM

@haraldinho said in SG-5100 takes over 20 minutes to boot after eMMC failure:

@gabacho4 That's how I understand what this setting does indeed. And I guess this setting could be done in pfsense, however, I feel that it is a design flaw of these Lanner boxes and should be fixed in BIOS. But I would be happy with either ;-)

The boot loader hint didn't work for me but I didn't think it would. My 5100 still appears to hang after I issue a reboot from the webConfigurator. The power light stays lit (good) but so do the lights on the igb0 and igb1 ports. They do this on a cold boot too but after a few seconds, they clear and then the boot process resumes. Still seems to be a BIOS issue.

FWIW, even after adding the boot loader hint that @stephenw10 suggested, pfSense/FreeBSD still sees the eMMC (via dmesg):

sdhci_pci0: <Intel Denverton eMMC 5.0 Controller> mem 0xdff9a000-0xdff9afff,0xdff99000-0xdff99fff irq 16 at device 28.0 on pci0
mmc0: <MMC/SD bus> on sdhci_pci0

Ideally there'd be a way to disable that completely in the BIOS.

haraldinho · Aug 7, 2022, 7:47 AM

@hayescompatible Hmm, that's a bummer. I must say I also have that same line still in my logs after the hack. So there is no difference there between our setups.

NOTE
after you made the modification, you need to power cycle your box once more to get the line loaded and effectuated. After that power cycle, warm reboots from the web configurator work again for me.

Some things you can check:

If you disable "Quiet mode" in BIOS and do a boot, do you see a difference in your logs with and without the hack?

For me all the references to timeouts like the one below went away in the logs (next to the fact I can now issue a Warm Reboot from the WebConfigurator.

mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
sdhci_pci0-slot0: Controller timeout
sdhci_pci0-slot0: ============== REGISTER DUMP ==============
sdhci_pci0-slot0: Sys addr: 0x06040000 | Version:  0x00001002
sdhci_pci0-slot0: Blk size: 0x00005200 | Blk cnt:  0x00000010
sdhci_pci0-slot0: Argument: 0x00000010 | Trn mode: 0x00000033
sdhci_pci0-slot0: Present:  0x1fff0206 | Host ctl: 0x00000025
sdhci_pci0-slot0: Power:    0x0000000b | Blk gap:  0x00000080
sdhci_pci0-slot0: Wake-up:  0x00000000 | Clock:    0x00000207
sdhci_pci0-slot0: Timeout:  0x0000000d | Int stat: 0x00000001
sdhci_pci0-slot0: Int enab: 0x01ff003b | Sig enab: 0x01ff003a
sdhci_pci0-slot0: AC12 err: 0x00000000 | Host ctl2:0x0000000c
sdhci_pci0-slot0: Caps:     0x546ec8b2 | Caps2:    0x80000007
sdhci_pci0-slot0: Max curr: 0x00000000 | ADMA err: 0x00000000
sdhci_pci0-slot0: ADMA addr:0x00000000 | Slot int: 0x00000000
sdhci_pci0-slot0: ===========================================

Do you see the last line from below snippet in your logs (not sure why the characters are all repeated, but that's not the issues here)? It indicates the file is loaded during boot.

Loading /boot/defaults/loader.conf
Loading /boot/defaults/loader.conf
Loading /boot/device.hints
Loading /boot/loader.conf
HLLooaaddiinngg  //bboooott//llooaaddeerr..ccoonnff..llooccaall

if you ssh into your box, and enter the below command, does it then show the exact text of the hack?

cat /boot/loader.conf.local

It should show:

hint.mmcsd.0.disabled="1"

check character set issues
Double check the quote symbols. I spent many hours debugging 'wrong quote symbols' issues in the past (" and ' and `). Also carefully check the other spellings. If you copy-pasted the line, try deleting the file and recreating it by typing it manually.

On my box, the file did not exist. I created it using vi with this commands:

cd /boot
sudo vi loader.conf.local

I typed the hack line manually, no copy-paste. Had to google how vi works, but that is not that complicated for this simple modification. Just get vi into insert mode by pressing i, type the line, hit escape to get vi into command mode and then enter :w to save the file and then again escape followed by :q to quit. Then do a reboot. But whom am I telling, perhaps you are a Linux guru :-D

stephenw10 · Aug 7, 2022, 11:22 AM

@haraldinho said in SG-5100 takes over 20 minutes to boot after eMMC failure:

The only thing I am not sure about is what happens to this setting when you do a regular upgrade to a new version of pfsense. @stephenw10 does it get preserved?

Yes, loader.conf.local is preserved across a firmware upgrade.

@hayescompatible said in SG-5100 takes over 20 minutes to boot after eMMC failure:

FWIW, even after adding the boot loader hint that @stephenw10 suggested, pfSense/FreeBSD still sees the eMMC (via dmesg):
sdhci_pci0: <Intel Denverton eMMC 5.0 Controller> mem 0xdff9a000-0xdff9afff,0xdff99000-0xdff99fff irq 16 at device 28.0 on pci0
mmc0: <MMC/SD bus> on sdhci_pci0

That's the mmc bus. mmcsd0 is the first mmc device on that bus which is what has been disabled.

If there was an issue with the bus or the controller you could disable those instead. Disabling only the emmc storage device itself is the minimum you need to do to stop is trying to read it.

Steve

cswroe · Aug 7, 2022, 12:13 PM

I just added a drive in a 5100 and it works fine. Actually, seems a bit snappier.
This was what I used: https://www.amazon.com/gp/aw/d/B07KG2G152/

hayescompatible · Aug 7, 2022, 12:30 PM

@haraldinho said in SG-5100 takes over 20 minutes to boot after eMMC failure:

@hayescompatible Hmm, that's a bummer. I must say I also have that same line still in my logs after the hack. So there is no difference there between our setups.

NOTE
after you made the modification, you need to power cycle your box once more to get the line loaded and effectuated. After that power cycle, warm reboots from the web configurator work again for me.

Yup, I power-cycled a couple of times: pulled the power cord, waited a minute then reconnected to start up again.

Some things you can check:

If you disable "Quiet mode" in BIOS and do a boot, do you see a difference in your logs with and without the hack?

For me all the references to timeouts like the one below went away in the logs (next to the fact I can now issue a Warm Reboot from the WebConfigurator.

mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
sdhci_pci0-slot0: Controller timeout
sdhci_pci0-slot0: ============== REGISTER DUMP ==============
sdhci_pci0-slot0: Sys addr: 0x06040000 | Version:  0x00001002
sdhci_pci0-slot0: Blk size: 0x00005200 | Blk cnt:  0x00000010
sdhci_pci0-slot0: Argument: 0x00000010 | Trn mode: 0x00000033
sdhci_pci0-slot0: Present:  0x1fff0206 | Host ctl: 0x00000025
sdhci_pci0-slot0: Power:    0x0000000b | Blk gap:  0x00000080
sdhci_pci0-slot0: Wake-up:  0x00000000 | Clock:    0x00000207
sdhci_pci0-slot0: Timeout:  0x0000000d | Int stat: 0x00000001
sdhci_pci0-slot0: Int enab: 0x01ff003b | Sig enab: 0x01ff003a
sdhci_pci0-slot0: AC12 err: 0x00000000 | Host ctl2:0x0000000c
sdhci_pci0-slot0: Caps:     0x546ec8b2 | Caps2:    0x80000007
sdhci_pci0-slot0: Max curr: 0x00000000 | ADMA err: 0x00000000
sdhci_pci0-slot0: ADMA addr:0x00000000 | Slot int: 0x00000000
sdhci_pci0-slot0: ===========================================

I will have to set aside some time to check this at my next opportunity to take the Internet down here…

Do you see the last line from below snippet in your logs (not sure why the characters are all repeated, but that's not the issues here)? It indicates the file is loaded during boot.
Loading /boot/defaults/loader.conf
Loading /boot/defaults/loader.conf
Loading /boot/device.hints
Loading /boot/loader.conf
HLLooaaddiinngg  //bboooott//llooaaddeerr..ccoonnff..llooccaall

…I will check this next time, too; however, based on what @stephenw10 seems to have confirmed in his previous post, applying the hint did work since I don't see mmcsd0 in my dmesg output.

if you ssh into your box, and enter the below command, does it then show the exact text of the hack?
cat /boot/loader.conf.local
It should show:
hint.mmcsd.0.disabled="1"
check character set issues
Double check the quote symbols. I spent many hours debugging 'wrong quote symbols' issues in the past (" and ' and `). Also carefully check the other spellings. If you copy-pasted the line, try deleting the file and recreating it by typing it manually.

On my box, the file did not exist. I created it using vi with this commands:
cd /boot
sudo vi loader.conf.local
I typed the hack line manually, no copy-paste. Had to google how vi works, but that is not that complicated for this simple modification. Just get vi into insert mode by pressing i, type the line, hit escape to get vi into command mode and then enter :w to save the file and then again escape followed by :q to quit. Then do a reboot. But whom am I telling, perhaps you are a Linux guru :-D

I do sysadmin mostly on Linux boxes so I know my way around vi and such. The file exists there, I checked the quoting method and from what I understand, dmesg confirms the eMMC is no longer being recognized (if it was at all before).

I wonder if the state of the eMMC is a thing? Mine seems completely dead: it used to give me timeout errors but the last time I tried specifically accessing it (during a clean install of pfSense and accidentally selecting the eMMC as the install root), the installer just froze. I gave up after several minutes of waiting. IIRC (possible I don't), I would get some timeout errors and the installer would eventually raise an error.

In any event, this trick isn't working for me. I'll try to schedule some downtime in the next week to try again.

hayescompatible · Aug 7, 2022, 12:36 PM

@stephenw10 said in SG-5100 takes over 20 minutes to boot after eMMC failure:

@haraldinho You can disable it completely in pfSense and prevent those errors.
Create the file /boot/loader.conf.local and add to it:
hint.mmcsd.0.disabled="1"
But that doesn't help with the slow boot in the BIOS.

Steve

@haraldinho I think the part I've emphasized here is relevant. My 5100 hangs either before or during POST on a warm boot. The boot process hasn't even begun yet. So this makes me even more skeptical that the loader hint would be applicable in my situation since the OS isn't being loaded yet.

stephenw10 · Aug 7, 2022, 3:44 PM

Yeah, if it's hanging at POST before it even tries to boot from SSD then adding a loader variable won't make any difference.

hayescompatible · Aug 7, 2022, 3:44 PM

@stephenw10 said in SG-5100 takes over 20 minutes to boot after eMMC failure:

Yeah, if it's hanging at POST before it even tries to boot from SSD then adding a loader variable won't make any difference.

Thanks for confirming that. I also just confirmed that myself by rebooting the 5100 before I left the house this morning. Here are the last messages I see from pfSense as it is rebooting:

pflog0: promiscuous mode disabled
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 0 0 0 0 0 0 done
Waiting (max 60 seconds) for system thread `bufdaemon' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-0' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-2' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-6' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-1' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-5' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-3' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-4' to stop... done
All buffers synced.
Uptime: 7h50m45s

After an hour and a half, it never came back up so it is definitely hanging on warm boot, it's not just a delayed startup.

The POST codes that come up in the console on a cold boot are A2 99 B2. On a warm boot, there are no POST codes. I'm hopeful a BIOS update can address this soon.

FYI @haraldinho just to follow up on my last message, I can confirm the loader.conf.local file is loaded:

FreeBSD/x86 bootstrap loader, Revision 1.1
Loading /boot/defaults/loader.conf
Loading /boot/defaults/loader.conf
Loading /boot/device.hints
Loading /boot/loader.conf
Loading /boot/loader.conf.local

gabacho4 · Aug 7, 2022, 5:44 PM

quick question for the group - when you installed the m.2 sata, did you have to do anything special in the BIOS to get it recognized as the primary boot device or is that automatic? I have someone who is going to help get an m.2 and pfsense reinstalled on my remote router and I just want to understand the process as best as I can in advance. Unfortunately, I won't get my m.2 here until later this month due to the time it takes to get USPS overseas. The Netgate documentation on this process suggests it should be basically automatic but I'd rather make sure rather than run around in circles. Thanks!

stephenw10 · Aug 7, 2022, 6:15 PM

You do have to change the boot device order. Or at least I have had to when I have done it myself.

hayescompatible · Aug 7, 2022, 6:32 PM

@gabacho4 said in SG-5100 takes over 20 minutes to boot after eMMC failure:

quick question for the group - when you installed the m.2 sata, did you have to do anything special in the BIOS to get it recognized as the primary boot device or is that automatic? I have someone who is going to help get an m.2 and pfsense reinstalled on my remote router and I just want to understand the process as best as I can in advance. Unfortunately, I won't get my m.2 here until later this month due to the time it takes to get USPS overseas. The Netgate documentation on this process suggests it should be basically automatic but I'd rather make sure rather than run around in circles. Thanks!

You will need to go into the BIOS to change the boot order but it's a pretty easy thing to do. What wasn't clear when I went through this is that the SSD has to be formatted UEFI or you won't be able to select it as a boot device.