24.03 Stuck at Not Ready Yet.

jrey

@stephenw10 said in 24.03 Stuck at Not Ready Yet.:

What did you do to allow it to complete?

allow what to complete ? the pfblocker pulls?
I just stopped it to prevent pfblocker from pulling anything at this point while I figure out the cert issue.

there was nothing really wrong with the upgrade as such that I can see in the logs. just the cert issue. which I wouldn't have expected to be a problem.

jrey

@stephenw10

well this is interesting -- in the pfblocker error.log

timestamped right around the time the upgrade had completed.
But also seems this has nothing to do with my certificate

[1713961838] unbound[84846:0] error: Error for server-cert-file: /var/unbound/unbound_server.pem
[1713961838] unbound[84846:0] error: Error in SSL_CTX use_certificate_chain_file crypto error:80000002:system library::No such file or directory
[1713961838] unbound[84846:0] error: and additionally crypto error:10080002:BIO routines::system lib
[1713961838] unbound[84846:0] error: and additionally crypto error:0A080002:SSL routines::system lib
[1713961838] unbound[84846:0] fatal error: could not set up remote-control

edit:
I don't see any errors like those displayed in the redmine you referenced, just the ones above and they were referenced in the pfblocker error.log

Strange also that now after this latest reboot (removing/re-adding the local trust and certctl rehash, I see the "Verify question" at the top of the dashboard.

Still seems to be running 24.03

pfsjap

@jrey Nothing new in those unbound errors, had and reported them already in BETA.

NOCling

I run into the same error with my 6100 yesterday:

Unbound stopped in 1 sec.
Additional mounts (DNSBL python):
Starting Unbound Resolver Not completed.
[1713888880] unbound[71102:0] error: Error for server-cert-file: /var/unbound/unbound_server.pem
[1713888880] unbound[71102:0] error: Error in SSL_CTX use_certificate_chain_file crypto error:80000002:system library::No such file or directory
[1713888880] unbound[71102:0] error: and additionally crypto error:10080002:BIO routines::system lib
[1713888880] unbound[71102:0] error: and additionally crypto error:0A080002:SSL routines::system lib
[1713888880] unbound[71102:0] fatal error: could not set up remote-control

Unbound is broken and couldn't be started.

Apr 23 18:19:13	php-fpm	29637	/services_unbound.php: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1713889153] unbound[49797:0] error: Error for server-cert-file: /var/unbound/unbound_server.pem [1713889153] unbound[49797:0] error: Error in SSL_CTX use_certificate_chain_file crypto error:80000002:system library::No such file or directory [1713889153] unbound[49797:0] error: and additionally crypto error:10080002:BIO routines::system lib [1713889153] unbound[49797:0] error: and additionally crypto error:0A080002:SSL routines::system lib [1713889153] unbound[49797:0] fatal error: could not set up remote-control'
Apr 23 18:19:05	php-fpm	645	/status_services.php: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1713889145] unbound[27355:0] error: Error for server-cert-file: /var/unbound/unbound_server.pem [1713889145] unbound[27355:0] error: Error in SSL_CTX use_certificate_chain_file crypto error:80000002:system library::No such file or directory [1713889145] unbound[27355:0] error: and additionally crypto error:10080002:BIO routines::system lib [1713889145] unbound[27355:0] error: and additionally crypto error:0A080002:SSL routines::system lib [1713889145] unbound[27355:0] fatal error: could not set up remote-control'
Apr 23 18:18:56	php-fpm	48050	/services_unbound.php: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1713889136] unbound[47160:0] error: Error for server-cert-file: /var/unbound/unbound_server.pem [1713889136] unbound[47160:0] error: Error in SSL_CTX use_certificate_chain_file crypto error:80000002:system library::No such file or directory [1713889136] unbound[47160:0] error: and additionally crypto error:10080002:BIO routines::system lib [1713889136] unbound[47160:0] error: and additionally crypto error:0A080002:SSL routines::system lib [1713889136] unbound[47160:0] fatal error: could not set up remote-control'

I use the boot environment to go back to 23.09.1 and install the patches.

pfsjap

@NOCling Manual reboot at that point would have resolved unbound issue, I have done it many times since BETA.

jrey

@NOCling

Thanks, but this is not at the end of the day the issue causing the problem for me.

Everything is working except the CA for the let's encrypt certificate, when adding it to the local store so pfblocker can use it when talking to another local server.

there are two certificates

the one with CN=ISRG Root X1
and the second with CN=R3

both created by the let's encrypt process and both working previously

if I uncheck the "add this certificate authority to the operating system trust store"
and look at the /etc/ssl "stuff" neither are there (expected)
then if I check the "add this.. " again, only the ISRG Root X1 cert shows up in the "stuff" the R3 one no longer seems to want to add to the local store.

I've recreated that cert per let's encrypt info (the data is exactly the same as what was there) but it still won't go to the local store.

as a result now when pfblocker attempts to pull from the local server using that cert it now returns

curl: (60) SSL certificate problem: unable to get local issuer certificate
makes senses it is not there anymore !

debugging the curl connection does indeed say the cert needed in the CA is not there

* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection

the cert is valid works from other machines, and of course is not expired. Just doesn't seem to want to go to the local cert -- and therefore curl can't find it. If the R3 cert where there it would then chain up to the ISRG which is there.

NOCling

@pfsjap
Thx, its resolved.
So simple, but I didn't think of it.

jrey

@stephenw10

So I can confirm that the CA GUI is not adding the certificate to the OS Trust Store. but that the Certificate is working

curl -Iv https://123.sample.com

fails

* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection

export the certificate from the GUI, put it in a pem file in the tmp directory

curl -Iv --cacert /tmp/lets-encrypt-r3.pem https://123.sample.com

works like a charm

* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):

So I guess the question is why isn't the GUI getting this to the local trust store when saved? I can see the certs directory rebuilding on save, but clearly when finished this cert is not there. Curl can't find it and neither can I

I suspect I can just drop a copy of the file in /usr/share/certs/trusted and then link the file to /etc/ssl/certs and it would fine, but that seems like something that should be happening on save?

there is nothing "fun" in logs that I can see - general log php-frm (process) just says there was a configuration change when I hit save to toggle the option on/off

Updated Certificate Authority Acmecert: O=Lets Encrypt, CN=R3, C=US

Guess I need to look at system_camanager.php to see exactly what it is (not) doing

stephenw10

Have you opened a bug for this?

jrey

@stephenw10

yup is broken for sure. I fired up my 2.7.2 test virtual. (had no certs on it at all)
CA added (by cutting and pasting the values in the cert on the 24.03 (24.03_1) box into a new certificate - check the box and hit save..

Immediately the hashed file name appeared in /etc/ssl/certs (the file name in green is the cert "Acmecert: O=Lets Encrypt, CN=R3, C=US" (data in the file looks fine)

Screen Shot 2024-04-25 at 8.59.10 AM.png

and curl on the test 2.7.2 box immediately recognized it..

* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):

Works as expected, no issues

jrey

@stephenw10

Nope - have not -- too busy trying to validate the issue and make it go ! (this is a production box that is borked, so getting it back in production is my number 1 priority at this time)

I'll likely either go back to 23.09.1 through boot environments and wait for the system to be ready for prime time.

two updates in one day (24.03 and 24.03_1) and hours of down time is actually enough for me.

Seems in order to create a bug report, I'd have to have yet another account on the bug tracker. That's not happening today.

jrey

@stephenw10 said in 24.03 Stuck at Not Ready Yet.:

Have you opened a bug for this?

have now,
https://redmine.pfsense.org/issues/15440

jimp

Looks like something changed in certctl and now it wipes the directory when writing the CAs which also wipes out the CA files from the cert manager.

Try this diff and see if it helps, and keep in mind that certctl rehash can take up to a few minutes to run, so check ps uxaww | grep certctl and make sure it's done before you try running any commands.

diff --git a/src/etc/inc/certs.inc b/src/etc/inc/certs.inc
index be5a0de777..5b7735d191 100644
--- a/src/etc/inc/certs.inc
+++ b/src/etc/inc/certs.inc
@@ -2371,6 +2371,8 @@ function ca_setup_trust_store() {
        safe_mkdir($trust_store_directory);
        unlink_if_exists("{$trust_store_directory}/*.0");
 
+       mwexec_bg('/usr/sbin/certctl rehash');
+
        foreach (config_get_path('ca', []) as $ca) {
                /* If the entry is invalid or is not trusted, skip it. */
                if (!is_array($ca) ||
@@ -2382,7 +2384,6 @@ function ca_setup_trust_store() {
 
                ca_setup_capath($ca, $trust_store_directory);
        }
-       mwexec_bg('/usr/sbin/certctl rehash');
 }
 
 /****f* certs/ca_setup_capath

If that doesn't help there are more options just needs some experimentation to figure out the best path forward.

EDIT: It's also worth noting that Let's Encrypt CA chains are in the OS default trust store already so any custom entries that are duplicating Let's Encrypt CAs in the chain may be conflicting with the built-in copies.

jrey

@jimp said in 24.03 Stuck at Not Ready Yet.:

and now it wipes the directory when writing the CAs

this patch didn't help.

I watched the /etc/ssl/certs directory repopulate and then tried - still
unable to get local issuer certificate error

What does work, in the short term, is to add the .pem named file into /usr/share/certs/trusted (this is the certificate that is regenerated every 60 days and the one causing the issue.)
I just exported it from the GUI CA screen, renamed the file as "Lets_Encrypt_R3.pem" and copied it to /usr/share/certs/trusted.

now when the /etc/ssl/certs gets cleared and rehashed the system thinks it is part of all root certificates and it gets processed.

it will work for about 50 more days until the next renewal.

I'll leave the "patch" in place for now - let me know if there is something else you want me to try.

jimp

Revert the previous diff and use the System Patches package and then create an entry for 27fc5a3020fe981b7a5bc98fc9b1660e8773fc7d to apply the fix I committed.

See also: https://redmine.pfsense.org/issues/15440#note-1

jrey

@jimp said in 24.03 Stuck at Not Ready Yet.:

Revert the previous diff

Of Course revert, then I was already reading your note about the /usr/local/etc/ssl/certs directory, and was going to comment that the directory does not exist. -- but I see you have changed the reference for the path in the new commit id --
testing that as I type.. ..

so I now see the file in /usr/local/etc/ssl/certs

total 5
-rw-r--r--  1 root wheel 1856 Apr 25 14:04 8d33f237.crt

i see the /etc/ssl/certs rebuilding
and after the rehash the file is now in the mix

-r--r--r--  2 root wheel 2629 Apr 18 20:47 8cb5ee0f.0
-rw-r--r--  2 root wheel 1856 Apr 25 14:04 8d33f237.0  < < -- 
-r--r--r--  2 root wheel 4616 Apr 18 20:47 8d86cdd1.0
-r--r--r--  1 root wheel  875 Apr 25 14:04 8d89cda1.0

Works..

i take it since the change is in certs.inc that an auto renew will be handled by this change as well ?

Thanks so much.

jimp

It should be handled properly any time the CA trust store gets refreshed now.

I'll get that into the system patches package in the near future as well.