SG-5100 takes over 20 minutes to boot after eMMC failure

hayescompatible

@stephenw10 said in SG-5100 takes over 20 minutes to boot after eMMC failure:

@haraldinho You can disable it completely in pfSense and prevent those errors.
Create the file /boot/loader.conf.local and add to it:
hint.mmcsd.0.disabled="1"
But that doesn't help with the slow boot in the BIOS.

Steve

@haraldinho I think the part I've emphasized here is relevant. My 5100 hangs either before or during POST on a warm boot. The boot process hasn't even begun yet. So this makes me even more skeptical that the loader hint would be applicable in my situation since the OS isn't being loaded yet.

stephenw10

Yeah, if it's hanging at POST before it even tries to boot from SSD then adding a loader variable won't make any difference.

hayescompatible

@stephenw10 said in SG-5100 takes over 20 minutes to boot after eMMC failure:

Yeah, if it's hanging at POST before it even tries to boot from SSD then adding a loader variable won't make any difference.

Thanks for confirming that. I also just confirmed that myself by rebooting the 5100 before I left the house this morning. Here are the last messages I see from pfSense as it is rebooting:

pflog0: promiscuous mode disabled
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 0 0 0 0 0 0 done
Waiting (max 60 seconds) for system thread `bufdaemon' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-0' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-2' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-6' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-1' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-5' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-3' to stop... done
Waiting (max 60 seconds) for system thread `bufspacedaemon-4' to stop... done
All buffers synced.
Uptime: 7h50m45s

After an hour and a half, it never came back up so it is definitely hanging on warm boot, it's not just a delayed startup.

The POST codes that come up in the console on a cold boot are A2 99 B2. On a warm boot, there are no POST codes. I'm hopeful a BIOS update can address this soon.

FYI @haraldinho just to follow up on my last message, I can confirm the loader.conf.local file is loaded:

FreeBSD/x86 bootstrap loader, Revision 1.1
Loading /boot/defaults/loader.conf
Loading /boot/defaults/loader.conf
Loading /boot/device.hints
Loading /boot/loader.conf
Loading /boot/loader.conf.local

gabacho4

quick question for the group - when you installed the m.2 sata, did you have to do anything special in the BIOS to get it recognized as the primary boot device or is that automatic? I have someone who is going to help get an m.2 and pfsense reinstalled on my remote router and I just want to understand the process as best as I can in advance. Unfortunately, I won't get my m.2 here until later this month due to the time it takes to get USPS overseas. The Netgate documentation on this process suggests it should be basically automatic but I'd rather make sure rather than run around in circles. Thanks!

stephenw10

You do have to change the boot device order. Or at least I have had to when I have done it myself.

hayescompatible

@gabacho4 said in SG-5100 takes over 20 minutes to boot after eMMC failure:

quick question for the group - when you installed the m.2 sata, did you have to do anything special in the BIOS to get it recognized as the primary boot device or is that automatic? I have someone who is going to help get an m.2 and pfsense reinstalled on my remote router and I just want to understand the process as best as I can in advance. Unfortunately, I won't get my m.2 here until later this month due to the time it takes to get USPS overseas. The Netgate documentation on this process suggests it should be basically automatic but I'd rather make sure rather than run around in circles. Thanks!

You will need to go into the BIOS to change the boot order but it's a pretty easy thing to do. What wasn't clear when I went through this is that the SSD has to be formatted UEFI or you won't be able to select it as a boot device.

gabacho4

@hayescompatible is that something you do when installing pfsense or when/how? There seem to be a lot of little details missing from the guide.

hayescompatible

@gabacho4 said in SG-5100 takes over 20 minutes to boot after eMMC failure:

@hayescompatible is that something you do when installing pfsense or when/how? There seem to be a lot of little details missing from the guide.

See https://docs.netgate.com/pfsense/en/latest/install/install-walkthrough.html - you would choose "Auto (ZFS)" then set the partition scheme to one of the UEFI options.

gabacho4

@hayescompatible ah ok. That part is easy. I was afraid you were going to tell me it was something more obscure. Phew!

stephenw10

You can boot from eMMC or SSD as a legacy boot device as long as CSM is enabled.

haraldinho

@stephenw10 Are you making any progress with getting this issue as a feature in the upcoming BIOS update?

stephenw10

We are working on it, yes. This is unlikely to be a fast turn around though.

haraldinho

@stephenw10 I understand that BIOS changes are lengthy processes I have two questions for you:

can you confirm that the change that fixes the issue described in this thread is now confirmed part of this upcoming BIOS update?
can you give us any indication when we could expect it? Just so we can avoid bugging you before that moment

Thanks for all your help Steve!

stephenw10

Right now I can't give you any updates on either of those things unfortunately.

I'll add info here as I know more.

Steve

gabacho4

@stephenw10 just wanted to check in to see if there is any status update on the BIOS update addressing the eMMC issue. Know you said it could be a while but figured a check in after a few months was reasonable.

stephenw10

Unfortunately it's unlikely to happen, at least not as a BIOS update like that. There doesn't appear to be a hidden setting of some sort for this so it would require re-coding in the BIOS with all that entails.
If I find anything further I'll update here.

Steve

pdnx

Any developments on this?

I recently had the same issue happen on 2 units, and just finally put together what was going on. I have at least a dozen of these in service, and as I started checking them all for their MMC status, they are all indicating above 100%. Which means I have to get all of them swapped with SSDs pronto, before any more fail.

But what I am looking at on my desk right now are effectively bricks if they are going to take 20 minutes to reboot.

stephenw10

I'm not aware of any update on this issue. No workaround I've seen.
I'll update here if we find anything.

Steve

cswroe

I ended up setting up my 5100 as a backup but forgot about the long boot-up time with the bad eMMC. I ended up taking the eMMC off the board. Since it was just on the shelf, if it did not work, there was really no loss. Regardless, all works well, and I was surprised at how easy it was. I was apprehensive to try to do the hot air rework due to the potential of the small nearby components being blown away, so I just used a small razor blade under a corner and very gently pulled it off. A little clean-up to remove any lingering solder balls and success. Boot up times, warm and cold, are perfect.

rcoleman-netgate

@cswroe yep, I did this a few weeks ago with one I purchased from eBay.

I am pretty sure we can break some of those resistors to get it back up without a reworking station but I need a donor system to do it.