upgrade failed: tls_process_server_certificate:certificate verify failed

[mig39]

Getting this error when attempting to upgrade from 21.05.1 to 22.01 on an SG1100, using the web interface:

[96/187] Fetching pam_ldap-186_1.pkg: ..... done
[97/187] Fetching p7zip-16.02_3.pkg: .......... done
[98/187] Fetching openvpn-client-export-2.5.2.pkg: .......... done
[99/187] Fetching openvpn-auth-script-1.0.0.3.pkg: . done
[100/187] Fetching openvpn-2.5.4_1.pkg: .......... done
[101/187] Fetching opensc-0.22.0.pkg: .......... done
[102/187] Fetching oniguruma-6.9.7.1.pkg: .......... done
[103/187] Fetching ntp-4.2.8p15_3.pkg: .......... done
1082953728:error:141F0006:SSL routines:tls_construct_cert_verify:EVP lib:/var/jenkins/workspace/pfSense-build-release-tarballs/BUILD_NODE/pkg-aarch64/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/crypto/openssl/ssl/statem/statem_lib.c:283:
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/CN=repo00.netgate.com
1082953728:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-build-release-tarballs/BUILD_NODE/pkg-aarch64/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/crypto/openssl/ssl/statem/statem_clnt.c:1916:
Child process pid=49132 terminated abnormally: Segmentation fault
Failed

[bmeeks]

@mig39 said in upgrade failed: tls_process_server_certificate:certificate verify failed:

Getting this error when attempting to upgrade from 21.05.1 to 22.01 on an SG1100, using the web interface:

[96/187] Fetching pam_ldap-186_1.pkg: ..... done
[97/187] Fetching p7zip-16.02_3.pkg: .......... done
[98/187] Fetching openvpn-client-export-2.5.2.pkg: .......... done
[99/187] Fetching openvpn-auth-script-1.0.0.3.pkg: . done
[100/187] Fetching openvpn-2.5.4_1.pkg: .......... done
[101/187] Fetching opensc-0.22.0.pkg: .......... done
[102/187] Fetching oniguruma-6.9.7.1.pkg: .......... done
[103/187] Fetching ntp-4.2.8p15_3.pkg: .......... done
1082953728:error:141F0006:SSL routines:tls_construct_cert_verify:EVP lib:/var/jenkins/workspace/pfSense-build-release-tarballs/BUILD_NODE/pkg-aarch64/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/crypto/openssl/ssl/statem/statem_lib.c:283:
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/CN=repo00.netgate.com
1082953728:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-build-release-tarballs/BUILD_NODE/pkg-aarch64/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/crypto/openssl/ssl/statem/statem_clnt.c:1916:
Child process pid=49132 terminated abnormally: Segmentation fault
Failed

I believe in the past, when users have hit this error, the recommendation was a power-off reset of the box. The cryto chip inside the SG-1100 can get confused, and the only way to reset it is a power-off and power-on sequence. A simple reboot does not do it.

So gracefully shutdown the box, and after insuring it is fully halted, remove the power for several seconds and then reapply. It should boot up and then you will be able to upgrade without issue.

[mig39]

@bmeeks said in upgrade failed: tls_process_server_certificate:certificate verify failed:

So gracefully shutdown the box, and after insuring it is fully halted, remove the power for a several seconds and then reapply

Thanks! Will try to do so this evening.

[mfld]

@mig39 on my SG-1100 the issue was there is not hardware RTC. Had it shelved for a while and when it came up it couldn't syc the time from NTP server via hostname because DNS over TLS was broken due to the time being way off. Chicken/egg thing.

Check your system clock, if it is off you can set the time manually or hardcode an IP address, not a hostname for NTP. If you aren't using DNS over TLS this won't effect you and won't help. But do check your system clock and NTP status anyway to be sure.

[jimp]

@mfld said in upgrade failed: tls_process_server_certificate:certificate verify failed:

@mig39 on my SG-1100 the issue was there is not hardware RTC. Had it shelved for a while and when it came up it couldn't syc the time from NTP server via hostname because DNS over TLS was broken due to the time being way off. Chicken/egg thing.

FYI- That is handled better on 22.01/2.6.0:
https://docs.netgate.com/pfsense/en/latest/services/ntpd/bootstrap.html#ntp-bootstrap

[mfld]

FYI- That is handled better on 22.01/2.6.0:
https://docs.netgate.com/pfsense/en/latest/services/ntpd/bootstrap.html#ntp-bootstrap

That's awesome! And as always, great docu.

[mig39]

For anyone finding this thread in the future...

Following the advice and halting the system, physically disconnecting power for 30 seconds and then plugging in again did the trick.

I've successfully upgraded.

Thanks!

[SteveITS]

@mig39 Just to link the doc page on it:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/upgrades.html#segmentation-fault-in-pkg
...which is similar but mentions "SSL routines:ssl3_send_client_verify:internal error" (which is what I recall seeing, last spring) instead of "SSL routines:tls_construct_cert_verify."