Phantom ISRG Root X1 CA cert

LarryFahnoe

Netgate 4200, pfSense+ 24.03

I believe I have a phantom ISRG Root X1 CA cert which is preventing my IPsec site to site VPN from coming up. I base this partially on jimp's comment in https://forum.netgate.com/topic/187658/24-03-stuck-at-not-ready-yet/17:

EDIT: It's also worth noting that Let's Encrypt CA chains are in the OS default trust store already so any custom entries that are duplicating Let's Encrypt CAs in the chain may be conflicting with the built-in copies.

My IPsec site to site tunnel would not come up and it appeared that the problem was due to the LE certs though I did not save the logs. The certs were recently renewed and are now using the LE R10 and R11 intermediate CA certs whereas the old certs were using the R3 intermediate CA (if I recall correctly).

The IPsec tunnel is configured using mutual cert authentication using LE certs and has been working properly for some time. Initially I had trouble getting the tunnel to establish and the solution was to manually add the ISRG Root X1 CA cert which was necessary to get IPsec to trust intermediate CA certificate "C=US, O=Let's Encrypt, CN=R3" That seemed like a hack at the time but I was not able to get it to work otherwise; PKI is not one of my core competencies!

Most recently I had enabled Mobile IPsec which also used LE certs but I do not think this is relevant to the problem. The ACME / LE certs have been renewing normally & I could see that the newest ones were referencing intermediate R10 and R11 certs, so my first effort was to delete the manually imported ISRG Root X1 CA cert because that cert is now in the OS trust store (verified by certctl -v list | grep ISRG). OS trust store has both X1 and X2 CA certs. I deleted the X1 CA cert on both firewalls and rebooted. Tunnel would not come up. The R3 intermediate CA cert was still showing as in use by IPsec Tunnel (which I later found was due to P1 Peer Certificate Authority set to use R3).

Eventually I renewed all of my LE certs but the tunnel refused to come up. The certs were originally RSA but after looking at Let's Encrypt Hierarchy as of June 2024 (https://letsencrypt.org/certificates/) I decided to switch all the certs to ECDSA and renew them to further move away from the problematic ISRG Root X1. This resulted in the E5 and E6 intermediate CAs being added. I then reset the P1 Peer Certificate Authority to use E5 and deleted R3, R10 and R11 intermediate CA certs. After rebooting both firewalls the tunnel refuses to come up and the reason is that it is still unhappy with ISRG Root X1...

Easiest to see from the responder's logs:

<con1|15> selected peer config 'con1'
<con1|15>   using certificate "CN=vpn.mumble.com"
<con1|15>   certificate "CN=vpn.mumble.com" key: 384 bit ECDSA
<con1|15>   using trusted intermediate ca certificate "C=US, O=Let's Encrypt, CN=E5"
<con1|15>   certificate "C=US, O=Let's Encrypt, CN=E5" key: 384 bit ECDSA
<con1|15> no issuer certificate found for "C=US, O=Let's Encrypt, CN=E5"
<con1|15>   issuer is "C=US, O=Internet Security Research Group, CN=ISRG Root X1"
<con1|15> no trusted ECDSA public key found for 'gateway.mumble.com'
[...]
<con1|15> generating IKE_AUTH response 1 [ N(AUTH_FAILED) ]

And to my eye, an interesting clue is in the initiator's logs:

<con1|8> received cert request for unknown ca with keyid 23:6c:5f:b9:72:44:d6:50:9d:48:8c:2e:9b:83:e4:0e:d9:e4:84:59
<con1|8> received cert request for "C=US, O=Let's Encrypt, CN=E5"
<con1|8> received 1 cert requests for an unknown ca

I did record the following before deleting the manually imported ISRG Root X1 CA cert:

This is the current state:

So my question is, if I am correct, how do I exorcise any remnants of that ISRG Root X1 cert I manually added and then later deleted????

--Larry