Certificate verification failed

[zitstif]

Is anyone else running into this with an SG1100 or in general? I have tried rebooting and doing a cold boot but I'm still running into the same problem:

Messages:
Your Netgate device has pfSense+ as part of your device purchase.
ERROR: It was not possible to determine pfSense-u-boot-1100 remote version
ERROR: It was not possible to determine pkg remote version

Updating repositories metadata...
Updating pfSense-core repository catalogue...
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/OU=pfSense Plus/CN=pfsense-plus-pkg00.atx.netgate.com
2191183872:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/ssl/statem/statem_clnt.c:1921:
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/OU=pfSense Plus/CN=pfsense-plus-pkg00.atx.netgate.com
2191183872:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/ssl/statem/statem_clnt.c:1921:
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/OU=pfSense Plus/CN=pfsense-plus-pkg00.atx.netgate.com
2191183872:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/ssl/statem/statem_clnt.c:1921:
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/OU=pfSense Plus/CN=pfsense-plus-pkg00.atx.netgate.com
2191183872:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/ssl/statem/statem_clnt.c:1921:
pkg-static: https://pfsense-plus-pkg00.atx.netgate.com/pfSense_plus-v23_05_aarch64-core/meta.txz: Authentication error
repository pfSense-core has no meta file, using default settings
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/OU=pfSense Plus/CN=pfsense-plus-pkg00.atx.netgate.com
2191183872:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/ssl/statem/statem_clnt.c:1921:
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/OU=pfSense Plus/CN=pfsense-plus-pkg00.atx.netgate.com
2191183872:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/ssl/statem/statem_clnt.c:1921:
pkg-static: https://pfsense-plus-pkg00.atx.netgate.com/pfSense_plus-v23_05_aarch64-core/packagesite.pkg: Authentication error
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/OU=pfSense Plus/CN=pfsense-plus-pkg00.atx.netgate.com
[..truncated..]
Error updating repositories!
ERROR: It was not possible to determine pfSense-upgrade remote version

Thanks in advance for any insight or info.

[zitstif]

Anyone have any ideas?

[rcoleman-netgate]

@zitstif Please open a ticket at https://go.netgate.com/

[zitstif]

@rcoleman-netgate Ticket made. Thanks.

[zitstif]

The way to fix this issue is to go to System update, update settings, then click the save button on that page. However, I DO NOT RECOMMEND upgrading to 23.10, after the update took place the appliance would no longer boot and was getting stuck on:

Filename '6379773F.img'.
Load address: 0x7000000
Loading: T T TIM-1.0
WTMI-devel-18.12.1-1a13f2f
WTMI: system early-init
SVC REV: 5, CPU VDD voltage: 1.213V
NOTICE: Booting Trusted Firmware
NOTICE: BL1: v1.5(release):1f8ca7e-dirty (Marvell-devel-18.12.2)
NOTICE: BL1: Built : 18:22:47, Oct 7 2021
NOTICE: BL1: Booting BL2
NOTICE: BL2: v1.5(release):1f8ca7e-dirty (Marvell-devel-18.12.2)
NOTICE: BL2: Built : 18:22:52, Oct 7 2021
NOTICE: BL1: Booting BL31
NOTICE: BL31: v1.5(release):1f8ca7e-dirty (Marvell-devel-18.12.2)
NOTICE: BL31: Built : 18

U-Boot 2018.03-devel-18.12.3-gc9aa92c-dirty (Oct 07 2021 - 18:20:55 -0300)

Model: Netgate 1100
CPU 1200 [MHz]
L2 800 [MHz]
TClock 200 [MHz]
DDR 750 [MHz]
DRAM: 1 GiB
Comphy chip #0:
Comphy-0: USB3 5 Gbps
Comphy-1: PEX0 2.5 Gbps
Comphy-2: SATA0 6 Gbps
SATA link 0 timeout.
AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
flags: ncq led only pmp fbss pio slum part sxs
PCIE-0: Link down
MMC: sdhci@d0000: 0, sdhci@d8000: 1
Loading Environment from SPI Flash... SF: Detected mx25u3235f with page size 256 Bytes, erase size 64 KiB, total 4 MiB
console comconsole failed to initialize
Consoles: EFI console
Reading loader env vars from /efi/freebsd/loader.env
Setting currdev to disk0p2:ge: 0x0, reg: 0x0, val: 0xFFFF
FreeBSD/arm64 EFI loader, Revision 1.1g: 0x0, val: 0xFFFF
(Fri Feb 10 20:26:39 UTC 2023 root@freebsd)
Hit any key to stop autoboot: 0
Command line arguments: loader.efit!
Image base: 0x7000000netgate-1100.dtb
EFI version: 2.70720-sg1100.dtb
EFI Firmware: Das U-Boot (rev 0.00)tb
Console: efi,comconsole (0).dtb
Load Path: /\armada-3720-sg1100.dtb
Load Device: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/eMMC(1)/eMMC(0)/HD( 2,0x01,0,0x64001,0x1117c)
Trying ESP: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/eMMC(1)/eMMC(0)/HD(2,0x 01,0,0x64001,0x1117c)0 ms (1.7 MiB/s)
Setting currdev to disk0p2: at 07000000 ...
Trying: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/eMMC(1)/eMMC(0)/HD(1,0x01,0 ,0x1,0x64000) sdhci@d0000.blk...
Setting currdev to disk0p1:ady
Trying: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/eMMC(1)/eMMC(0)/HD(3,0x01,0 ,0x7517d,0xe1ae83)
Setting currdev to zfs:pfSense/ROOT/default:
ERROR: cannot open /boot/lua/loader.lua: no such file or directory.

[rcoleman-netgate]

@zitstif Power off the unit completely.

Let it sit for 2-3 minutes.

Power back on. It should reboot without issue.

Also there is no release "23.10"

[SteveITS]

@rcoleman-netgate Hi Ryan, we've hit this on a 2100 that was fine. We install 23.01 via image (had the USB stick already), it boots, we restore the backup config, it boots, emails us that it boot up, installs packages, and then we lose Internet connectivity from LAN and can't connect to pfSense via the WAN IP. The next boot attempt (Diagnostics/Halt, or Restart), we get what looks like the same errors as above...it overwrites itself but ends with:

Load Device: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/eMMC(1)/eMMC(0)/HD( 2,0x01,0,0x64001,0x1117c)
Trying ESP: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/eMMC(1)/eMMC(0)/HD(2,0x 01,0,0x64001,0x1117c)0 ms (1.7 MiB/s)
Setting currdev to disk0p2: at 07000000 ...
Trying: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/eMMC(1)/eMMC(0)/HD(1,0x01,0 ,0x1,0x64000) sdhci@d0000.blk...
Setting currdev to disk0p1:ady
Trying: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/eMMC(1)/eMMC(0)/HD(3,0x01,0 ,0x7517d,0xe1ae83)
Setting currdev to zfs:pfSense/ROOT/default:
ERROR: cannot open /boot/lua/loader.lua: no such file or directory.

Powering off for 3 minutes does not help. We re-imaged again, same thing.

[rcoleman-netgate]

@stephenw10 here's one for you.

[SteveITS]

@rcoleman-netgate So fun story, and/or in case anyone else runs into this. My tech tried reimaging again, this time with me watching. This flew by:

Loading /boot/device.hintsx0000
Loading /boot/loader.conf0x0000
Loading /boot/loader.conf.local
/ad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
bad MBR sector signature 0x0000
Scanning disk usb_mass_storage.lun0...
Found 5 disks
   _ __  / _|___  ___ _ __  ___  ___      _
  | '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \   _| |_
  | |_) |  _\__ \  __/ | | \__ \  __/  |_   _|
  | .__/|_| |___/\___|_| |_|___/\___|    |_|
  |_|

So we recreated a USB stick with 23.05, reimaged, and that seems to be just fine so far. IOW it seems to have been a bad USB stick. I just can't figure out why it would seem to work, let us restore and boot, and then the second boot had a problem. "Back away slowly" as I often say in this business.

Edit: apologies for hijacking the thread. There were very few posts about the loader.lua error.

[stephenw10]

Yeah that error is usually either a bad USB stick or the stick 'doesn't like' the USB port. It will often boot and install fine from the other USB port on the 1100 if you see that.

I don't suppose you have an upgrade or console log from the upgrade that failed to the missing loader.lua file?

Steve

[SteveITS]

@stephenw10 Our situation was on a 2100, though that's similar hardware of course. I started in the middle of this thread finding the loader error. We could not copy off an entire log, since in the "bad" condition there was no shell, and we could not type. Some of the console output was overwriting itself at times so it was a bit mixed together anyway.

I do have screen shots I was texted at one point in the discussion. I don't recall if this is after booting, after the loader error? My coworker is in transit now and I might not get him until tomorrow. At this point we had been thinking the emmc failed but writing the image was fine each time.

[SteveITS]

I'm being told the "cannot open /boot/lua/loader.lua" message shows after the "run usbrecovery" process, and the three screenfuls are what shows after the boot attempt without the USB stick attached.

[stephenw10]

Hmm, is that a 2100-MAX?

After running usbrecovery there should be nothing on the eMMC. Unless it boots the USB and appears to reinstall successfully?

[SteveITS]

@stephenw10 No, a base.

That's the thing, it does successfully wipe the eMMC, does reinstall, does reboot, does let us log in to restore the backup, does restart successfully, does email that it booted up successfully per notification settings in the config, then it loses networking and the next boot fails. Which makes no sense to me. It's like it starts off fine then ZFS runs off the rails or something and chews itself up.

After reinstalling using 23.05 from a different USB stick he restarted a few times without issue so it's good.

We think this may have happened before, a unit that didn't boot after a power loss a few months ago. I did a reinstall with this same USB stick (since it was the small EFI), and did a restore, and after that it didn't boot up, but I don't recall the error message specifically, could have been the loader message. At the time I had 15 minutes left on site and a spare 2100 so I put that in. We are going to try to resurrect it in our spare time...we pulled it out of our recycling pile.

Our tech wanted to use a USB stick that had been used before rather than create one.

Edit:
The router today had an older version, not sure now, maybe 22.01? Possibly earlier. We hadn't upgraded to 22.05 as they are an unmanaged client, and didn't go to 23.01 because of the EFI partition. I doubt something in the config restore could break things though, never had a problem before.

[stephenw10]

Hmm, that feels like a ZFS BE issue. Let me see if we can see anything.....

[SteveITS]

@SteveITS said in 23.05 firmware upgrade crashed a 3100 and an 1100:

FWIW a coworker reinstalled the "dead" 2100 with the same 23.05 USB he used a couple weeks ago and it seems to be fine in very limited usage. He's restarted it several times.

Per https://forum.netgate.com/topic/180755/23-05-firmware-upgrade-crashed-a-3100-and-an-1100/5 it sounds like there is/was a path for the EFI loader to not be updated and/or written properly.

[stephenw10]

Indeed, that should now be fixed. Will be in 23.05.1