Certificate verification failed

rcoleman-netgate

@stephenw10 here's one for you.

SteveITS

@rcoleman-netgate So fun story, and/or in case anyone else runs into this. My tech tried reimaging again, this time with me watching. This flew by:

Loading /boot/device.hintsx0000
Loading /boot/loader.conf0x0000
Loading /boot/loader.conf.local
/ad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
Scanning disk usb_mass_storage.lun0...
Found 5 disks
   _ __  / _|___  ___ _ __  ___  ___      _
  | '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \   _| |_
  | |_) |  _\__ \  __/ | | \__ \  __/  |_   _|
  | .__/|_| |___/\___|_| |_|___/\___|    |_|
  |_|

So we recreated a USB stick with 23.05, reimaged, and that seems to be just fine so far. IOW it seems to have been a bad USB stick. I just can't figure out why it would seem to work, let us restore and boot, and then the second boot had a problem. "Back away slowly" as I often say in this business.

Edit: apologies for hijacking the thread. There were very few posts about the loader.lua error.

stephenw10

Yeah that error is usually either a bad USB stick or the stick 'doesn't like' the USB port. It will often boot and install fine from the other USB port on the 1100 if you see that.

I don't suppose you have an upgrade or console log from the upgrade that failed to the missing loader.lua file?

Steve

SteveITS

@stephenw10 Our situation was on a 2100, though that's similar hardware of course. I started in the middle of this thread finding the loader error. We could not copy off an entire log, since in the "bad" condition there was no shell, and we could not type. Some of the console output was overwriting itself at times so it was a bit mixed together anyway.

I do have screen shots I was texted at one point in the discussion. I don't recall if this is after booting, after the loader error? My coworker is in transit now and I might not get him until tomorrow. At this point we had been thinking the emmc failed but writing the image was fine each time.

SteveITS

I'm being told the "cannot open /boot/lua/loader.lua" message shows after the "run usbrecovery" process, and the three screenfuls are what shows after the boot attempt without the USB stick attached.

stephenw10

Hmm, is that a 2100-MAX?

After running usbrecovery there should be nothing on the eMMC. Unless it boots the USB and appears to reinstall successfully?

SteveITS

@stephenw10 No, a base.

That's the thing, it does successfully wipe the eMMC, does reinstall, does reboot, does let us log in to restore the backup, does restart successfully, does email that it booted up successfully per notification settings in the config, then it loses networking and the next boot fails. Which makes no sense to me. It's like it starts off fine then ZFS runs off the rails or something and chews itself up.

After reinstalling using 23.05 from a different USB stick he restarted a few times without issue so it's good.

We think this may have happened before, a unit that didn't boot after a power loss a few months ago. I did a reinstall with this same USB stick (since it was the small EFI), and did a restore, and after that it didn't boot up, but I don't recall the error message specifically, could have been the loader message. At the time I had 15 minutes left on site and a spare 2100 so I put that in. We are going to try to resurrect it in our spare time...we pulled it out of our recycling pile.

Our tech wanted to use a USB stick that had been used before rather than create one.

Edit:
The router today had an older version, not sure now, maybe 22.01? Possibly earlier. We hadn't upgraded to 22.05 as they are an unmanaged client, and didn't go to 23.01 because of the EFI partition. I doubt something in the config restore could break things though, never had a problem before.

stephenw10

Hmm, that feels like a ZFS BE issue. Let me see if we can see anything.....

SteveITS

@SteveITS said in 23.05 firmware upgrade crashed a 3100 and an 1100:

FWIW a coworker reinstalled the "dead" 2100 with the same 23.05 USB he used a couple weeks ago and it seems to be fine in very limited usage. He's restarted it several times.

Per https://forum.netgate.com/topic/180755/23-05-firmware-upgrade-crashed-a-3100-and-an-1100/5 it sounds like there is/was a path for the EFI loader to not be updated and/or written properly.

stephenw10

Indeed, that should now be fixed. Will be in 23.05.1