IPSec with certificate authentication for clients

[jelockwood]

I started off by building a standard Ubuntu box and installing Racoon and successfully configured it to allow IPSec connections from a Mac using Apple's built-in IPSec client and an iPhone also using Apple's built-in client. This was however using a pre-shared-key and xauth via LDAP.

I then tried to reconfigure it to allow using a server certificate at the Racoon end, and a client certificate at the Mac/iPhone end. Unfortunately I could not get this to work. I then came across pfSense and thought it worth a go as it automates and idiot proofs much of the setup therefore I hoped it would prevent any mistakes on my part.

Unfortunately I have had pretty much identical results. I again was able to setup a PSK with xauth via LDAP setup and it again worked, however when I tried a certificate configuration using RSA + xauth it failed as with Ubuntu. As pfSense is also using Racoon it is not too surprising I suppose.

So in a little more detail what I am trying to do is -

Run a Cisco IPSec compatible VPN server, i.e. compatible with the built-in Apple clients
Use certificates rather than a pre-shared-key
Authenticate users against LDAP
Send all client traffic via the VPN connection

The fact it works with a PSK means IP addressing, firewalls, routing, and NAT should all be correctly setup (I can ping in both directions when connected).

All I need it seems is some help getting the certificate authentication side sorted.

The errors I get in both Ubuntu and pfSense are

Jan 31 18:15:46 racoon: ERROR: Inpropper ID type passed: KEY_ID.
Jan 31 18:15:46 racoon: ERROR: no peer's CERT payload found.

(repeated multiple times)

With the Ubuntu setup I was using OpenSSL and a self-signed-root-CA, with pfSense I am using the built-in Cert Manager and its generated self-signed-root-CA. I then generated a server certificate and a user certificate. If anyone else has succeeded I am happy to follow your recipe.

I was running Ubuntu 12.0.4 64bit, I am now trying the following version of pfSense…

2.1-RELEASE (amd64)
built on Wed Sep 11 18:17:48 EDT 2013
FreeBSD 8.3-RELEASE-p11

You are on the latest version.

[jelockwood]

Ok I tracked down and solved one huge problem I was experiencing :) and have now found a new one.  :(

It turns out there is a nasty bug in the built-in Apple iPhone iOS 7.0.4 IPSec client. I had started off with (standard) Racoon in Ubuntu, and then tried pfSense both configured to PSK, and therefore had of course also started off with the iPhone and its IPSec client also configured to a PSK. When I reconfigured Racoon and pfSense to use Certificates I of course reconfigured the iPhone to match. I did not however delete the existing profile on the iPhone I merely modified it. This should have been fine, however it turns out the iPhone was still sending the Group Name to the IPSec server and this was certainly giving pfSense indegestion.

Note: I spotted this in the logs for StrongSwan, no entry in the Racoon logs suggested this. I had started to move on to testing StrongSwan since I had been unsuccessful with pfSense and Racoon.

Once I made a fresh profile on the iPhone I was then able to successfully make IPSec with certificate connections from the iPhone to pfSense. So that is the good news. Unfortunately I still have a problem.

I am wanting to route all traffic via the VPN connection, this works for IPSec with no certificates and is achieved by not ticking the option in pfSense to 'Provide a list of networks to clients'. If however with certificates I have this option unticked then the connection fails with the following errors in the log.

Feb 3 15:47:31  racoon: [Self]: INFO: respond new phase 2 negotiation: 81.x.x.12[500]<=>86.x.x.247[500]
Feb 3 15:47:31  racoon: ERROR: failed to get sainfo.
Feb 3 15:47:31  racoon: ERROR: failed to get sainfo.
Feb 3 15:47:31  racoon: [86.x.x.247] ERROR: failed to pre-process ph2 packet [Check Phase 2 settings, networks] (side: 1, status: 1).

With that option ticked it works fine but of course means that only traffic for the LAN gets routed via the VPN connection.

I did also notice that there maybe a Phase2 mismatch between the client and the server. The pfSense server is configured to use a virtual IP range of 10.0.1.0/24 for clients. However when the iPhone connects I get the following message in the log.

racoon: INFO: no policy found, try to generate the policy : 10.0.1.1/32[0] 192.168.16.0/24[0] proto=any dir=in

As you can see it is listing the policy as 10.0.1.1/32 and not as expected 10.0.1.1/24. Although from on point of view the fact that there will only be a single device at the client end makes a subnet mask of 32 logical. This does mean however that if I set the IPSec Tunnel proposal checking to anything other than Obey it fails due to a mis-match between the client and server ends. The full set of log entries for this type of failure looks like

Feb 3 15:58:41  racoon: [Self]: INFO: respond new phase 2 negotiation: 81.x.x.12[500]<=>86.x.x.247[500]
Feb 3 15:58:41  racoon: INFO: no policy found, try to generate the policy : 10.0.1.1/32[0] 192.168.16.0/24[0] proto=any dir=in
Feb 3 15:58:41  racoon: ERROR: pfs group mismatched: my:2 peer:0
Feb 3 15:58:41  racoon: ERROR: not matched
Feb 3 15:58:41  racoon: ERROR: no suitable policy found.
Feb 3 15:58:41  racoon: [86.x.x.247] ERROR: no proposal chosen [Check Phase 2 settings, algorithm].
Feb 3 15:58:41  racoon: [86.x.x.247] ERROR: failed to pre-process ph2 packet [Check Phase 2 settings, networks] (side: 1, status: 1).

There is of course no-way to configure this on the iPhone.

So I am now very close. I can do IPSec with PSK+Xauth for LDAP and route all traffic, I can do IPSec with RSA+Xauth for LDAP but cannot route all traffic.

Has anyone been able to do IPSec with RSA+Xauth and route all traffic with an iPhone?