IPsec with EAP-TLS client cert auth failing [SOLVED]

hugh_jarse

TL;DR: I'm trying to set up IPSec with EAP-TLS, following the guides at https://forum.pfsense.org/index.php?topic=106433.0 and https://doc.pfsense.org/index.php/IKEv2_with_EAP-TLS. I have the Phase 1 and 2 encryption settings configured, but authentication of the client certificate is failing, and I would like help figuring out why. I have two sections I'm looking for feedback on: questions, and troubleshooting. They are likely related.

Questions:

What information does the server use to verify that the client should be authenticated. Is it simply that the client cert was signed by the CA? Is there no need to verify that a certain part of the certificate (such as all/part of the CN) is valid? I ask because our current SonicWall L2TP/IPsec VPN filters on a portion of the client cert CN, matching the {user} side of "{user}@{server_FQDN}" to a local account username.
Do I need to create a user account within User Manager to correspond with the VPN client user? If so, does the account information need to match part of the certificate? I tried creating an account with a username of {user}@{server_FQDN}, but was denied because it contains an @ and is more than 16 characters.

Troubleshooting:

pfSense is 2.3.4-RELEASE-p1
TLS certificates for the server and the client were all generated on a separate box from pfSense (i.e. I did not use pfSense's GUI to generate the keys/certs)
These same TLS certs are currently being successfully used on a SonicWall L2TP/IPsec with EAP-TLS VPN (but note I'm not trying to use L2TP here with pfSense)
The server cert contains a CN of "{server_FQDN}", has EKUs of "Server Authentication" and "IKE Intermediate", and a SAN DNS Name of {server_FQDN}.
The client cert contains a CN of "{user}@{server_FQDN}" and has the "Client Authentication" EKU, but there is no SAN.
The signing CA cert (but not the key) has been added to pfSense's Certificate Manager > CA
IPSec Phase 1 Proposal (Authentication) settings are the following:
- Authentication Method: EAP-TLS
- My identifier: {server_FQDN}
- Peer identifier: Any (I suspect the problem is this setting)
- My Certificate {server cert}
- Peer Certificate Authority: {cert of the CA that signed the server and client certs}
Client OS is Mac OS X Sierra with the native IKEv2 client. The CA cert and the client PKCS#12 cert/key have been correctly imported into the OS certificate store, trust granted, and all applications given permission to access the key (in the past, only racoon needed access).
Client VPN settings use "{server_FQDN}" as the Remote ID, and "{user}@{server_FQDN}" as the Local ID. Connecting to the VPN server successfully negotiates AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048, and the client successfully verifies the server and sends the client cert, but the server cannot verify it:

[snip]
Sep 21 15:23:06	charon		12[NET] received packet: from {client_IP}[20432] to {server_IP}[4500] (180 bytes)
Sep 21 15:23:06	charon		12[NET] <con1|17> received packet: from {client_IP}[20432] to {server_IP}[4500] (180 bytes)
Sep 21 15:23:06	charon		12[ENC] parsed IKE_AUTH request 9 [ EF(2/2) ]
Sep 21 15:23:06	charon		12[ENC] <con1|17> parsed IKE_AUTH request 9 [ EF(2/2) ]
Sep 21 15:23:06	charon		12[ENC] received fragment #2 of 2, reassembling fragmented IKE message
Sep 21 15:23:06	charon		12[ENC] <con1|17> received fragment #2 of 2, reassembling fragmented IKE message
Sep 21 15:23:06	charon		12[ENC] parsed IKE_AUTH request 9 [ EAP/RES/TLS ]
Sep 21 15:23:06	charon		12[ENC] <con1|17> parsed IKE_AUTH request 9 [ EAP/RES/TLS ]
Sep 21 15:23:06	charon		12[TLS] received TLS peer certificate 'C={our_C}, ST={our_ST}, O={our_O}, OU={our_OU}, CN={user}@{server_FQDN}, E={user}@{our_domain}'
Sep 21 15:23:06	charon		12[TLS] <con1|17> received TLS peer certificate 'C={our_C}, ST={our_ST}, O={our_O}, OU={our_OU}, CN={user}@{server_FQDN}, E={user}@{our_domain}'
Sep 21 15:23:06	charon		12[TLS] received TLS intermediate certificate 'C={our_C}, ST={our_ST}, L={our_L}, O={our_O}, OU={our_OU}, CN={our_CA_name}, E=info@{our_domain}'
Sep 21 15:23:06	charon		12[TLS] <con1|17> received TLS intermediate certificate 'C={our_C}, ST={our_ST}, L={our_L}, O={our_O}, OU={our_OU}, CN={our_CA_name}, E=info@{our_domain}'
Sep 21 15:23:06	charon		12[TLS] no trusted certificate found for '{user}@{server_FQDN}' to verify TLS peer
Sep 21 15:23:06	charon		12[TLS] <con1|17> no trusted certificate found for '{user}@{server_FQDN}' to verify TLS peer
Sep 21 15:23:06	charon		12[TLS] sending fatal TLS alert 'certificate unknown'
Sep 21 15:23:06	charon		12[TLS] <con1|17> sending fatal TLS alert 'certificate unknown'
Sep 21 15:23:06	charon		12[ENC] generating IKE_AUTH response 9 [ EAP/REQ/TLS ]
Sep 21 15:23:06	charon		12[ENC] <con1|17> generating IKE_AUTH response 9 [ EAP/REQ/TLS ]
Sep 21 15:23:06	charon		12[NET] sending packet: from {server_IP}[4500] to {client_IP}[20432] (96 bytes)
Sep 21 15:23:06	charon		12[NET] <con1|17> sending packet: from {server_IP}[4500] to {client_IP}[20432] (96 bytes)
Sep 21 15:23:06	charon		12[NET] received packet: from {client_IP}[20432] to {server_IP}[4500] (128 bytes)
Sep 21 15:23:06	charon		12[NET] <con1|17> received packet: from {client_IP}[20432] to {server_IP}[4500] (128 bytes)
Sep 21 15:23:06	charon		12[ENC] parsed IKE_AUTH request 10 [ EAP/RES/TLS ]
Sep 21 15:23:06	charon		12[ENC] <con1|17> parsed IKE_AUTH request 10 [ EAP/RES/TLS ]
Sep 21 15:23:06	charon		12[IKE] EAP method EAP_TLS failed for peer {user}@{server_FQDN}
Sep 21 15:23:06	charon		12[IKE] <con1|17> EAP method EAP_TLS failed for peer {user}@{server_FQDN}
Sep 21 15:23:06	charon		12[ENC] generating IKE_AUTH response 10 [ EAP/FAIL ]
Sep 21 15:23:06	charon		12[ENC] <con1|17> generating IKE_AUTH response 10 [ EAP/FAIL ]
Sep 21 15:23:06	charon		12[NET] sending packet: from {server_IP}[4500] to {client_IP}[20432] (80 bytes)
Sep 21 15:23:06	charon		12[NET] <con1|17> sending packet: from {server_IP}[4500] to {client_IP}[20432] (80 bytes)</con1|17></con1|17></con1|17></con1|17></con1|17></con1|17></con1|17></con1|17></con1|17></con1|17></con1|17></con1|17></con1|17></con1|17></con1|17>

The "no trusted certificate found" error made me think that perhaps the client cert and key needed to actually be in pfSense's Certificate Manager, so I added them, but the same error occurs.

hugh_jarse

@hugh_jarse:

Questions:

What information does the server use to verify that the client should be authenticated. Is it simply that the client cert was signed by the CA? Is there no need to verify that a certain part of the certificate (such as all/part of the CN) is valid? I ask because our current SonicWall L2TP/IPsec VPN filters on a portion of the client cert CN, matching the {user} side of "{user}@{server_FQDN}" to a local account username.

Do I need to create a user account within User Manager to correspond with the VPN client user? If so, does the account information need to match part of the certificate? I tried creating an account with a username of {user}@{server_FQDN}, but was denied because it contains an @ and is more than 16 characters.

whether the client certificate was signed by the CA configured under "Peer Certificate Authority" (the strongswan setting for this in ipsec.conf is "rightca")
whether the client certificate has a Subject Alternative Name (SAN) DNS entry that is the same value as the certificate's CN (this is all outlined at https://wiki.strongswan.org/projects/strongswan/wiki/Win7CertReq). There is no matching of the SAN or CN value to a local pfSense account, so there is no need to create a user account within User Manager in the UI.

anybody who can sign a certificate with the CA can use that certificate to connect to the VPN without prior authorization.

@hugh_jarse:

Troubleshooting:

TLS certificates for the server and the client were all generated on a separate box from pfSense (i.e. I did not use pfSense's GUI to generate the keys/certs)

The signing CA cert (but not the key) has been added to pfSense's Certificate Manager > CA

should

certificate

@hugh_jarse:

The server cert contains a CN of "{server_FQDN}", has EKUs of "Server Authentication" and "IKE Intermediate", and a SAN DNS Name of {server_FQDN}.

The client cert contains a CN of "{user}@{server_FQDN}" and has the "Client Authentication" EKU, but there is no SAN.

Here is where my mistakes were.

the server certificate should have also had a SAN DNS entry (not an IP entry) of the IP address of the server
all client certificates MUST have a SAN DNS entry that is the same as the certificate's CN. Without this SAN, authentication will fail exactly as I was experiencing.

@hugh_jarse:

IPSec Phase 1 Proposal (Authentication) settings are the following:

Authentication Method: EAP-TLS

My identifier: {server_FQDN}

Peer identifier: Any (I suspect the problem is this setting)

My Certificate {server cert}

Peer Certificate Authority: {cert of the CA that signed the server and client certs}

Client OS is Mac OS X Sierra with the native IKEv2 client. The CA cert and the client PKCS#12 cert/key have been correctly imported into the OS certificate store, trust granted, and all applications given permission to access the key (in the past, only racoon needed access).

Client VPN settings use "{server_FQDN}" as the Remote ID, and "{user}@{server_FQDN}" as the Local ID. Connecting to the VPN server successfully negotiates AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048, and the client successfully verifies the server and sends the client cert, but the server cannot verify it:

"Peer identifier" being set to ANY is fine.
For the Mac OS X IKEv2 client, the only application that you should grant access to the client's private key is /usr/libexec/neagent (the L2TP/IPsec client is /usr/sbin/racoon). Allowing all applications to access the private key is a security risk – don't do it.
For Mac OS X's native IKEv2 client, the Local ID needs to be set to the same value as the client certificate's CN (and therefore SAN)

andrew_241

I was getting the same error with the certificates not being trusted for EAP-TLS. As it turns out, I needed to import my root CA's public key into pfSense. It's that key that signs my pfSense router/firewall's intermediate CA.

hugh_jarse

@andrew_241:

I was getting the same error with the certificates not being trusted for EAP-TLS. As it turns out, I needed to import my root CA's public key into pfSense. It's that key that signs my pfSense router/firewall's intermediate CA.

Yes, that is something that one should certainly keep in mind during setup. If the entire certificate chain is not present and thus cannot be verified, authentication will fail.

PatrickF

Hi, since i'm currently messing around with the same stuff - first a big thanks to youf or outlining this!
May i ask you a question not 100% related to the topic.

How to you protect your VPN with client-certifcates?
At my current point of view, any machines that can be accessed by a different person then the VPN-Certificate owner is a potential security issue. Since you can simply copy over the Client-Certificate and login from any machine you want at a later point of time.

Previously on our SonicWall IPSec solution, we had the PSK hidden/only installed by admins (and encrypted saved by the Sonicwall VPN Client).
So in addition to the credentials, anyone who wanted to setup the VPN tunnel, had a hard time to get access to the PSK / needed to ask a admin to help out.
I'm aware that this is not 100% protected, but at least it was not AS easy like copy a certificate to your usb.

Is there anything i do not see / i can do to fill this security hole?

Thanks,
Patrick

hugh_jarse

@PatrickF:

How to you protect your VPN with client-certifcates?
At my current point of view, any machines that can be accessed by a different person then the VPN-Certificate owner is a potential security issue. Since you can simply copy over the Client-Certificate and login from any machine you want at a later point of time.

Previously on our SonicWall IPSec solution, we had the PSK hidden/only installed by admins (and encrypted saved by the Sonicwall VPN Client).
So in addition to the credentials, anyone who wanted to setup the VPN tunnel, had a hard time to get access to the PSK / needed to ask a admin to help out.
I'm aware that this is not 100% protected, but at least it was not AS easy like copy a certificate to your usb.

Disclaimer: This was an interesting though experiment for me, and actually helped me clarify some issues for myself. I apologize that my answer is so long. This is probably more information than you needed, but I feel like I've answered your question.

With your SonicWALL solution, you used a pseudo-two-factor authentication method: 1. user credentials, 2. PSK. Looking at this, client certificate authentication might at first appear comparatively weak, but I assure you that it's superior.

First, let's make sure we're on the same page: remember that in order for client certificate authentication to work, having only the certificate is not enough. The client must also have the private key that the certificate was generated from in order for the server to verify that the presenter of the certificate is the true owner (with the assumption being that only the true owner would have the private key). For someone to actually use the client certificate to authenticate to the VPN server, she must have access to both the client certificate and the private key. The certificate is designed to be public, but the key is designed to be private. No key, no authentication. With that out of the way, the following assumes that the certificate and the key are always together.

Let me rephrase your question just for clarity (please feel free to correct me if I'm mistaken):
"Since client certificates are a single-factor authentication method that is based on "something you have", what is to prevent an attacker from stealing the client certificate/key and using it on some other machine to authenticate as the certificate's owner?"

There are actually multiple vectors that are possible, so to address them, let's do some simplistic threat modeling regarding unauthorized VPN access:

Threat 1: UserB on the same system copies UserA's certificate/key to another system and connects to the VPN
Threat 2: UserA copies their own certificate/key to another system and connects to the VPN
Threat 3: an rogue process accesses the certificate/key (either to use them to connect to the VPN or export them for later use)

The first line of defense is that the private key should be protected with a strong passphrase. Knowledge of that passphrase is required to import the key into another system. This helps protect against Threat 1, but not necessarily Threat 2 or 3.

Additional controls to prevent these threats to varying degrees could be implemented at either A) the VPN server, or B) on the client. Let's look at each of these:

A) The VPN server could prevent unauthorized client certificate authentication by requiring a second piece of authentication information, such as a password/PSK/etc in addition to the client certificate. This is similar to what your SonicWALL was doing with the PSK. The client would have to present the certificate as well as the password to successfully authenticate. I was doing this on our SonicWALL with an L2TP/IPsec VPN. Clients needed to authenticate with their client certificate (IPsec) as well as L2TP credentials. This would be great, but I don't know if it's possible with an IKEv2 VPN. If anyone knows more than I do, please share.

B) The client's OS could somehow restrict access to the client certificate/key and/or require a second piece of authentication information (such as a password) in order to access the client certificate/key.
This is likely the answer to your question. Remember that in order for an attacker to actually use the client certificate to authenticate to the VPN server, she must have access to both the client certificate and the private key.

Windows
–--------
Firstly, on the client machine, the VPN client certificate/key bundle (PKCS#12) should be imported only for the "Current User" (importing the bundle for the "Local Machine" would make it a machine certificate which could be used by any account on the system). This also helps mitigate Threat 1. Theoretically, a local administrator could export any certificate/key on the system, but that's why you configured the key with a strong passphrase, right?

Secondly, Windows offers two more controls:

when importing the certificate/key, the default setting is to NOT allow the key to later be exported. This prevents the user herself from exporting her own certificate/key and using it on another machine (even if she knows the key's passphrase). This mitigates Threat 2. However, if the user is local admin on the machine the certificate/key was imported on, you are likely vulnerable. Yet another reason to not allow users to have local admin privileges on a PC.
when importing the certificate/key, there is an option to "Enable strong private key protection". Enabling this gives you the option of prompting the user to either a) click OK to allow access to the key, or b) input an "access" password to allow access to the key (this is separate from the key's own passphrase). The password is set when you import the certificate/key bundle. This prevents some rogue process running with the user's privileges from silently accessing the key without the user knowing. This mitigates Threat 3.

macOS

Similar to Windows, the certificate/key bundle should be imported into the User's Keychain, which helps mitigate Threat 1.

When exporting a private key, Keychain Access will prompt the user for a (Administrator's?) password to authorize the export. This can mitigate Threat 2. But again, if the user is a local admin or knows the admin password, they can just enter it to successfully export the certificate/key bundle. To be honest, I don't know if the password prompt is explicitly for an Administrator, or just the currently logged-in user (there is no separate field to enter a different username, so I have a sinking feeling that it's merely prompting for the user's password as consent to export the bundle). I have not tested this, but I'm curious if anyone knows. Perhaps there's a potential weakness here?

When you import the certificate/key bundle, double-clicking the key allows you to configure Access Control for the key. One options is to "Confirm before allowing access". This will prompt the user to enter their account password every time they want to access the key to connect to the VPN. This can mitigate Threat 3. (you could balance security/usability by only allowing /usr/libexec/neagent to access the key without prompting)

iOS/Android/*nix/Commodore 64/whatevs

I have no idea what controls these might offer to mitigate certificate/key misuse, but I'm curious. If anyone could elaborate, I'd love to learn more.

Conclusion

Assuming your IT staff configure import the client certificate/key bundle into the user's machine, and the user does not have privileges to export the key, you should be relatively safe.
What can you do on top of this? I would recommend two things:

Limit the valid period of the certificates. You don't want a situation where an employee has left the organization, and can still connect to the VPN 2 years later. I would issue certificates with a maximum period of 1 year. Make this shorter as long as you can handle the overhead of issuing/configuring certificates more frequently.
Security monitoring. Use your logging infrastructure to alert on things like simultaneous use by the same certificate, and maybe (if possible) deviations in which cryptography suites are negotiated. While not perfect, this could theoretically detect certain instances where someone is using a copied certificate/key on another machine.

hugh_jarse

@PatrickF Going one step further that is very likely to be beyond your use case: one benefit of client certificates is that they don't need to actually be on the machine that's accessing the VPN. You could store the client certificate/key on a smart card (or YubiKey?) that the user must physically use with a card reader when they access the VPN. I have zero experience with this, but it should be possible. Of course, then more threat modeling is required to account for what types of physical attacks could be used and how to mitigate smart card theft/borrowing. :P

PatrickF

@hugh_jarse,
thank you very much for this detailed post. I'll need some time now to work through it :P