24.03 Stuck at Not Ready Yet.

NOCling

@pfsjap
Thx, its resolved.
So simple, but I didn't think of it.

jrey

So I can confirm that the CA GUI is not adding the certificate to the OS Trust Store. but that the Certificate is working

curl -Iv https://123.sample.com

fails

* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection

export the certificate from the GUI, put it in a pem file in the tmp directory

curl -Iv --cacert /tmp/lets-encrypt-r3.pem https://123.sample.com

works like a charm

* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):

So I guess the question is why isn't the GUI getting this to the local trust store when saved? I can see the certs directory rebuilding on save, but clearly when finished this cert is not there. Curl can't find it and neither can I

I suspect I can just drop a copy of the file in /usr/share/certs/trusted and then link the file to /etc/ssl/certs and it would fine, but that seems like something that should be happening on save?

there is nothing "fun" in logs that I can see - general log php-frm (process) just says there was a configuration change when I hit save to toggle the option on/off

Updated Certificate Authority Acmecert: O=Lets Encrypt, CN=R3, C=US

Guess I need to look at system_camanager.php to see exactly what it is (not) doing

stephenw10

Have you opened a bug for this?

jrey

@stephenw10

yup is broken for sure. I fired up my 2.7.2 test virtual. (had no certs on it at all)
CA added (by cutting and pasting the values in the cert on the 24.03 (24.03_1) box into a new certificate - check the box and hit save..

Immediately the hashed file name appeared in /etc/ssl/certs (the file name in green is the cert "Acmecert: O=Lets Encrypt, CN=R3, C=US" (data in the file looks fine)

Screen Shot 2024-04-25 at 8.59.10 AM.png

and curl on the test 2.7.2 box immediately recognized it..

* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):

Works as expected, no issues

jrey

@stephenw10

Nope - have not -- too busy trying to validate the issue and make it go ! (this is a production box that is borked, so getting it back in production is my number 1 priority at this time)

I'll likely either go back to 23.09.1 through boot environments and wait for the system to be ready for prime time.

two updates in one day (24.03 and 24.03_1) and hours of down time is actually enough for me.

Seems in order to create a bug report, I'd have to have yet another account on the bug tracker. That's not happening today.

jrey

@stephenw10 said in 24.03 Stuck at Not Ready Yet.:

Have you opened a bug for this?

have now,
https://redmine.pfsense.org/issues/15440

jimp

Looks like something changed in certctl and now it wipes the directory when writing the CAs which also wipes out the CA files from the cert manager.

Try this diff and see if it helps, and keep in mind that certctl rehash can take up to a few minutes to run, so check ps uxaww | grep certctl and make sure it's done before you try running any commands.

diff --git a/src/etc/inc/certs.inc b/src/etc/inc/certs.inc
index be5a0de777..5b7735d191 100644
--- a/src/etc/inc/certs.inc
+++ b/src/etc/inc/certs.inc
@@ -2371,6 +2371,8 @@ function ca_setup_trust_store() {
        safe_mkdir($trust_store_directory);
        unlink_if_exists("{$trust_store_directory}/*.0");
 
+       mwexec_bg('/usr/sbin/certctl rehash');
+
        foreach (config_get_path('ca', []) as $ca) {
                /* If the entry is invalid or is not trusted, skip it. */
                if (!is_array($ca) ||
@@ -2382,7 +2384,6 @@ function ca_setup_trust_store() {
 
                ca_setup_capath($ca, $trust_store_directory);
        }
-       mwexec_bg('/usr/sbin/certctl rehash');
 }
 
 /****f* certs/ca_setup_capath

If that doesn't help there are more options just needs some experimentation to figure out the best path forward.

EDIT: It's also worth noting that Let's Encrypt CA chains are in the OS default trust store already so any custom entries that are duplicating Let's Encrypt CAs in the chain may be conflicting with the built-in copies.

jrey

@jimp said in 24.03 Stuck at Not Ready Yet.:

and now it wipes the directory when writing the CAs

this patch didn't help.

I watched the /etc/ssl/certs directory repopulate and then tried - still
unable to get local issuer certificate error

What does work, in the short term, is to add the .pem named file into /usr/share/certs/trusted (this is the certificate that is regenerated every 60 days and the one causing the issue.)
I just exported it from the GUI CA screen, renamed the file as "Lets_Encrypt_R3.pem" and copied it to /usr/share/certs/trusted.

now when the /etc/ssl/certs gets cleared and rehashed the system thinks it is part of all root certificates and it gets processed.

it will work for about 50 more days until the next renewal.

I'll leave the "patch" in place for now - let me know if there is something else you want me to try.

jimp

Revert the previous diff and use the System Patches package and then create an entry for 27fc5a3020fe981b7a5bc98fc9b1660e8773fc7d to apply the fix I committed.

See also: https://redmine.pfsense.org/issues/15440#note-1

jrey

@jimp said in 24.03 Stuck at Not Ready Yet.:

Revert the previous diff

Of Course revert, then I was already reading your note about the /usr/local/etc/ssl/certs directory, and was going to comment that the directory does not exist. -- but I see you have changed the reference for the path in the new commit id --
testing that as I type.. ..

so I now see the file in /usr/local/etc/ssl/certs

total 5
-rw-r--r--  1 root wheel 1856 Apr 25 14:04 8d33f237.crt

i see the /etc/ssl/certs rebuilding
and after the rehash the file is now in the mix

-r--r--r--  2 root wheel 2629 Apr 18 20:47 8cb5ee0f.0
-rw-r--r--  2 root wheel 1856 Apr 25 14:04 8d33f237.0  < < -- 
-r--r--r--  2 root wheel 4616 Apr 18 20:47 8d86cdd1.0
-r--r--r--  1 root wheel  875 Apr 25 14:04 8d89cda1.0

Works..

i take it since the change is in certs.inc that an auto renew will be handled by this change as well ?

Thanks so much.

jimp

It should be handled properly any time the CA trust store gets refreshed now.

I'll get that into the system patches package in the near future as well.