Reboot gets stuck at "Installing Nvme Lens"

tariqali

If I use the reboot command either through the GUI or the console, I keep getting stuck at the "Installing Nvme Lens" part of the reboot.

The only way to "reboot" is to pull the power and plug it back in, then it goes through the process just fine and boots like normal.

I recently replaced the SSD, in fact I was able to create a "mirror" ZFS setup (Dual Nvme SSDs), I did a restore from ACB to get the device working after a SSD failure. This is a 6100 Max device.

Running on the latest stable 24.11 release.

Loading Usb Lens...
 Provider:  Insyde Software
 Version:   05.12.02.0040.0009.16
 Product:   BlinkBoot(R) Harrisonville USB Lens
 Licensee:  Silicom Ltd.
 Start:     03/20/2018
 End:       12/31/2099

Validating Signature... [Pass]

Installing Usb Lens...

Loading Nvme Lens...
 Provider:  Insyde Software
 Version:   05.12.02.0040.0009.16
 Product:   BlinkBoot(R) Harrisonville NVMe Lens
 Licensee:  Silicom Ltd.
 Start:     11/16/2020
 End:       12/31/2099

Validating Signature... [Pass]

Installing Nvme Lens...

stephenw10

Does it just stop there or does it repeat the NVMe lens load sequence?

Did you replace the SSD with some different model device(s)? What device if so?

A ZFS mirror there should work fine, I run that here.

Does it do this at every warm boot? Requires a full power cycle to boot?

What if you halt the device and then start it using the PWR button?

tariqali

Hi @stephenw10 thanks for the quick response.

It just stops doesn't repeat. A cold boot gets past that step almost immediately.
I'm currently using two SK hynix BC501 NVMe 256GB, I found them used by a local ebay seller (needed to get my system back up and running). I will replace these with 2 new KingSpec 256GB M.2 NVMe SSDs which will arrive this week.
Thanks for confirming the mirror settings, it's really cool that we can do this.
Every warm boot it gets stuck, always need to do a full power cycle.
I've used Halt, but pulled to the power to restart, I'll need to try the power buttons, I'll follow up once I get a chance to try.

stephenw10

Hmm, sounds like some low level incompatibility with that particular drive unfortunately. I'd probably test a single drive to be sure something odd with the mirror isn't happening.

Also check the Blinkboot version shown in the dashboard is: CORDOBA-03.00.00.03t

tariqali

@stephenw10

Not sure where to see the Blinkboot version, but here's my current firmware which looks like what you're referring to:

Netgate Firmware details:
Current Firmware Version: 03.00.00.03t-uc-18
Latest Firmware Version: 03.00.00.03t-uc-18

I tried Halt and restarting with the power button (3 seconds), successfully.

stephenw10

Yup that's the current version, so all good there.

So it only fails at reboot?

Interesting. I'm not sure there's much we can do though. Blinkboot doesn't really have any settings you might try.

tariqali

@stephenw10 Thanks for following up. I'll report back at the end of the week once I swap the SSDs to see if it keeps happening.

tariqali

@stephenw10 I was trying to do a fresh install via the USB drive on the new drives and kept getting an error "Failed to fetch the pfSense repository data" during the setup. I'm guessing the repo was down?

So I decided to clone the old drives, and that seemed to work, but now I have degraded mirror:

  pool: pfSense
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
config:

        NAME        STATE     READ WRITE CKSUM
        pfSense     DEGRADED     0     0     0
          mirror-0  DEGRADED     0     0     0
            nda0p4  FAULTED      0     0     0  corrupted data
            nda1p4  ONLINE       0     0     0

errors: No known data errors

Any idea how I can fix this, the "faulted" drive is brand new, I think I just cloned it wrong, so if I could somehow format that faulted drive and recreate the mirror I should be good. I'm hoping to do this without opening up the 6100 again?

Thanks for any help.

tariqali

I ended up trying a fresh install again today, and it seems like the repo is back online, everything seems to be working as expected, including reboots.

Thanks @stephenw10 for all your help.

stephenw10

Great, Yes there was a backend issue last night. Should be fixed now though.

JonathanLee

@tariqali I have some Kingspecs they have taken a beating on my system and keep working.

tariqali

Thanks @JonathanLee, that's good to know, hoping to get at least another 4 years out of the 6100