[HowTo] - Windows IPSec VPN without 3rd party IPSec client

Grimeton

Hi guys,

A few days ago I had to configure native IPSec access from some Windows 7 machines to a box running the racoon IPSec daemon. As this daemon is also used on pfSense, I thought, it could be helpful to have the information available here. Maybe this leads to a feature that allows this to be configured without a hassle.

I think I'm not the only one that is tired of a zillion IPSec clients that all just work with that specific implementation and not the default standards. That's why I started to configure VPN access based on the IPSec stuff that is provided with Windows itself since Windows 2000. No external client, no magic, just a few settings are needed to make it work.

First of all a few things upfront:

Use of certificate authentication

This setup uses certificate authentication, but it can easily be changed to password or Kerberos authentication. Authentication via certificates offers some possibilities that you don't get with PSK. As the identifier is derived from the certificate's DN, you won't see unknown IP addresses there (that evil iPhone …). Also, the connection is only allowed if the cert could be validated and wasn't rejected before (CRL check). On the Windows site, you can mark the private key of the certificate as “not exportable”. So whatever happens, no one is able to get access to the private key. And btw: You don't really want to use password authentication, do you?

This is a tunnel setup. Transport is possible too.
You need Windows Vista or Windows 2008 or later.

Windows XP will not work as it does not offer the „Windows Firewall with Advanced Security“.

Yes, I said the toolchain is delivered since Windows 2000, but as AES-CBC-256 is only supported starting with Windows Vista and later, I can't go for it right now. Another point is the mentioned Windows firewall that allows to use extended encryption like AES-CBC-256, which is not supported by the GPOs available for Windows XP and earlier. It also allows to manage those IPSec rules, like enabling or disabling them, by the logged on user. And even this can be controlled by adding ACLs to the rules, so that only specific users are able to enable/disable them.

Usually you don't even need this, as the rules are used by the system “on demand”, so keeping them enabled the whole time doesn't really hurt anyone. As soon as there is a connection to your pfSense box and open ports on UDP (500,4500), there will be a successful connection to your LAN. And you don't need to use the GPOs which can be good in some situations.

But there is also a downside: Things like broadcasts won't work over plain IPSec as they do on L2TP, so the Windows Network Browser won't see the other end as long as it is not part of a Windows Domain (ADDS).

All commands executed in this „howto“ - including all tools, shells and so on are run with administrative permissions if not noted otherwise.
Tunnel does not use phase two authentication
The setup has the following parameters:

ESP
Authentication method is rsasig (can change based on your authentication)
AES-CBC-256 encryption
SHA1 for integrity (hash)
DH-group 2 for PFS
We allow NAT-Traversal
Local Subnet is 192.168.1.0/24
Remote subnet is 10.1.1.0/24
Local Endpoint is 192.168.1.5 (yes it's inside the tunneled network)
Remote Endpoint is 1.2.3.4

Let's get started.

By default, the Windows firewall only uses 3DES or AES-CBC-128 bit encryption, to change that we have to open a Windows command prompt and enter the following commands:

To change the defaults for IKE's main mode:

C:> netsh advfirewall set global mainmode mmsecmethods dhgroup2:aes256-sha1

To change the key lifetime:

C:> netsh advfirewall set global mainmode mmkeylifetime 120min,0sess

To enable and enforce Diffie Hellmann:

C:> netsh advfirewall set global mainmode mmforcedh yes

And to enable IPSec through routers:

C:> netsh advfirewall set global ipsec ipsecthroughnat serverandclientbehindnat

So far we changed the defaults, to get a “working” fallback environment. – Now we have to setup the main connection. As we are using options that can't be configured through the GUI, we have to use the shell again:

This is the command that will be issued:

C:> netsh advfirewall consec add rule name="ipserver" endpoint1=192.168.1.0/24 endpoint2=10.1.1.0/24 action=requireinrequireout mode=tunnel enable=no profile=any type=static localtunnelendpoint=any remotetunnelendpoint=1.2.3.4 protocol=any interfacetype=any auth1=ComputerCert auth1ca="CN=le.ca, DC=le, DC=ca" qmpfs=dhgroup2 qmsecmethods="ESP:SHA1-AES256+60min+1000000000kb"

The parameters explained:

„name“ - is the name of the rule
„endpoint1“ - is the LOCAL subnet/machine of the connection, 192.168.1.0/24 in this case
„endpoint2“ - is the REMOTE subnet/machine of the connection, 10.1.1.0/24 in this case
„action“ - defines what has to be done to make it work. In this case we request encryption for all traffic.
„mode“ - sets the mode – in this case tunnel
„enable“ - defines if the rule is enabled – at this point we're not enabling the rule.
„profile“ - refers to the different profiles Windows uses when identifying a connection. If you are not sure, just choose „any“
„type“ - defines the type of the rule. It should be static to „survive“ a reboot
„localtunnelendpoint“ - defines the local address used to connect to the other end – this has to be any, even if we have an IP address defined.
„remotetunnelendpoint“ - defines the remote address that we connect to
„protocol“ - defines the filter for the protocol we want to tunnel – in this case: any protocol
„interfacetype“ - defines the interface type this rule is applied to. Like LAN, WIFI, and so on – we use „any“
„auth1“ - the authentication method we use – in this case we're authenticating via a certificate stored in the computer's certificate store
„auth1ca“ - this represents the certificate authority's subject/distinguished name.
„qmpfs“ - defines the Diffie Hellman group to be used for „Perfect Forward Security“
„qmsecmethods“ - are the settings that are used to configure the main tunnel setup. We're running a tunnel setup (ESP) with SHA1 hashes and AES256 encryption. The keys will be renewed after 60 minutes (60min) or one terabyte of data (1000000000kb). Which ever comes first.

In setkey and racoon syntax it would looke like this:

One note: this is from the view of the machine we're configuring at the moment (Windows), and not the other end (pfSense)!

Setkey:


spdadd 192.168.1.0/24 10.1.1.0/24 any -P out ipsec esp/tunnel/192.168.1.5-1.2.3.4/require;
spdadd 10.1.1.0/24 192.168.1.0/24 any -P in ipsec esp/tunnel/1.2.3.4-192.168.1.5/require;

So we have:


spdadd $endpoint1 $endpoint2 $protocol -P $action1 $action2 $mode1/$mode2/$localtunnelendpoint-$remotetunnelendpoint/$action3;

spdadd $endpoint2 $endpoint1 $protocol -P $action1 $action2 $mode1/$mode2/$remotetunnelendpoint-$localtunnelendpoint/$action3;

The endpoints are pretty clear, same goes for $protocol which refers to any. The direction of the traffic flow ($action1 = out) and that we want it to be IPSec encrypted ($action2 = ipsec), are generated out of the one action option supplied. Same goes for the fact that the encryption is mandatory ($action3 = require).

The option „requireinrequireout“ that is handed to „action“ also makes two setkey rules mandatory. The one for incoming and the other one for outgoing traffic.

The mode options are used for the tunneling protocol, and the mode that is used. So in this case, $mode1 is „esp“ and $mode2 is „tunnel“. Both derived from the one setting that is handed to the “mode” option.

The tunnel endpoints usually represent the “outer IPs” of the firewalls that are used. In this case, it is better/easier to leave the setting to “any” on the Windows side, if the tunnel is only initiated from the Windows box to pfSense. This is, btw., the moment where the Windows IPSec stack / IKE daemons are the better implementation. Based on the policies, Windows chooses the correct local address that it uses to connect to the remote subnet through the tunnel.

These policies MUST NOT be set on the other end. We do not use them and leave the work to racoon!

Based on the IPSec policies we have defined so far, it becomes necessary to configure racoon and the proposal/sainfo sections.

The main setup should look like this:


# the path to your certstore that should be used by racoon. DO NOT use /etc/ssl/certs/ here
# or you will open your network to any CA that is in that directory.
# You have been warned.
path certificate “/etc/ssl/racoon” 

# as racoon sometimes has problems with inheritance and anonymous sections
# we will use 0.0.0.1 as the remote end to be able to define some global
# settings – using something like this: “foobar” as the name of the remote
# section doesn't work either
remote 0.0.0.1
{
	# you don't use aggressive, do you?
	exchange_mode main;  

	 ### SECTION-X509-START
	 # changes based on the auth methods and other settings
	my_identifier asn1dn;

	# changes based on the auth methods and other settings
	peers_identifier asn1dn; 

	 # always verify peer's identifier
	verify_identifier on;

	 # always deny the authentication of certs with an empty DN
	match_empty_cr off;

	# host's certificate and key file
	certificate_type x509 “host.certificate.pem” “host.privatekey.pem”; 

	# the public certificate of the CA that is used to validate all certs
	ca_type x509 “host.cacert.pem”; 

	### SECTION-X509-END

	# dead peer detection delay and maxfails
	dpd_delay = 10; 
	dpd_maxfail = 5;

	# this is how the proposal has to be matched against the other end's one
	# set this at least to “claim” and never use “obey”...
	proposal_check claim;

	# this is where it gets interesting
	proposal
	{
		# as said before, we're using “rsasig” here because of the certs
		# X509
		authentication_method rsasig;

		# set the encryption to aes256 which matches Windows' AES-CBC-256
		# No, there is no space between “aes” and “256”.
		#qmsecmethods
		encryption_algorithm aes256; 

		# as we discussed before, we use SHA1 as hash algorithm
		# qmsecmethods
		hash_algorithm sha1;

		# the Diffie Hellman group we use is “2” or “1024”
		dh_group 2;

		# the lifetime is set to 28800 secs
		#qmsecmethods
		lifetime time 28800 secs;
	}
 }

# and finally the anonymous remote block with some additional settings
# we inherit the main settings from 0.0.0.1
remote anonymous inherit 0.0.0.1
{
	nat_traversal on;
	generate_policy on;
}

So far we have the basic setup for our IPSec environment. But how do these settings belong to the Windows side of things?

Well, if we look at the commands that changed the global settings for IKE's “main mode”:

C:> netsh advfirewall set global mainmode mmsecmethods dhgroup2:aes256-sha1
C:> netsh advfirewall set global mainmode mmkeylifetime 120min,0sess
C:> netsh advfirewall set global mainmode mmforcedh yes
C:> netsh advfirewall set global ipsec ipsecthroughnat serverandclientbehindnat

we can see the following options:

“mmsecmethods” - MainModeSECurityMethods are defining the main proposal that should be used if nothing else is set. In this case we're defining a DH-group of “2” or “1024”, want to use AES-CBC-256 encryption and have all this hashed by the SHA1 hash algorithm.

Used with racoon's parameters, the first command would looke like this:

C:> netsh advfirewall set global mainmode mmsecmethods $dh_group:$encryption_algorithm-$hash_algorithm

The second one would look like this:

C:> netsh advfirewall set global mainmode mmkeylifetime $lifetime

The third one wouldn't change, but it will enforce the usage of the DH-group set via the first command. And the fourth one, will just enable NAT-Traversal, which is enabled in the “anonymous” section of racoon.

These settings are used for the phase one proposal as well as the sainfo in phase two, if nothing else is defined on the windows side.

To finally complete the main setup, let's have a look at the following options that are passed to the main rule. These options are valid for the “main mode” settings as well as the “SAINFO” settings later. It's a bit mess^Z^Z^Zixed up in Windows.

“auth1” - This option sets the authentication method that the Windows machine uses to authenticate itself to the other end. In this case it is set to “ComputerCert” which means that the system uses a certificate from Windows' computer certificate store (!).
“auth1ca” - Here we define the distinguished name of the CA-cert that the system should use. Based on this name, the CA is selected and the system searches for a computer certificate that has been signed by this CA. So our CA is “le.ca” with the DN of “CN=le.ca, DC=le, DC=ca” and the system's certificate needs to be signed by it.

The options “auth1” and “auth1ca” are representing the racoon settings that are inside the “SECTION-X509-START/END” area. Instead of issuing a DN to racoon, we do this via the files that hold all necessary certificates. The “authentication_method” in the proposal is also affected by those two settings. As it is chosen automagically by Windows' IKE.

But let's move on.

The next option we have to check is “qmsecmethods”. Here, the outer protocol, the hash and the encryption settings are defined. As well as the key lifetime in minutes and transferred bytes. Once again these settings are used for the proposal in the remote section as well as the encryption settings in the “SAINFO” - section.

The protocol is derived from the “mode” option we talked about earlier. Everything else is set in “qmsecmethods” directly. Using racoon's options to create the “qmsecmethods” option, we would see something like this:

qmsecmethods=”$mode:$hash_algorithm-$encrytion_algorithm+$lifetime”

And finally, to set the Diffie Hellman group to use, “qmpfs” is set to “dhgroup2”. Which should be pretty straight forward.

Now one last thing is left todo. We need to create an SAINFO section in racoon's configuration file, to allow the subnets to connect to each other, which could look like this:


sainfo subnet 10.1.1.0/24 any 192.168.1.0/24 any
{
	encryption_algorithm aes256;
	authentication_algorithm hmac_sha1;
	lifetime time 3600 secs;
}

And again we see that those parameters are derived by the “main mode” settings and/or by the “qmsecmethods” option that is handed to netsh.

The “qmsecmethods” option, created with racoon's parameters would look like this:

qmsecmethods=”ESP:$authentication_algorithm-$encrytion_algorithm+$lifetime”

And the “SAINFO” - networks are created by the options “endpoint1” and “endpoint2” as well as the “protocol” option, so with the use of Windows' parameters it would look like this:


sainfo subnet $endpoint2 $protocol $endpoint1 $protocol
{
	...
}

After the last command is executed, a rule is created that will be visible in the “Connection Security Rules” section in the “Windows Firewall with Advanced Security” MMC snap-in.

When you double click on it, you can see the warning “This rule contains properties that are not supported through this interface.”. This means, that we set options in the shell that the GUI is not able to display. So if you change anything in the GUI and save, those settings will be lost.

Enable the rule, and try to access the other end's subnet. And monitor racoon's logfile for errors.

If it doesn't work, check the settings again.

Have fun.

KR,

Grimeton

jimp

And how is that easier than making 10 clicks and installing OpenVPN?

Seems like that's a lot of convoluted hoops to jump through just to avoid installing an extra bit of software. I'm not sure I see the benefit, though others might…

Grimeton

Hi,

this is the basic explanation on how it works together. Setting it up is just four commands, or clicks, and importing the certs - which can all be done automagically via Windows GPOs. So you configure it once and then roll it out with the new WFAS GPOs and that's it. If it is one or one hundred machines doesn't really matter anymore.

Also you can enroll other policies and actions that have to be taken care of that you use more complicated VPN clients for today.

And OpenVPN isn't really integrating into the whole Windows ADDS environment and not allowed on the iPhone. So this looks like the better way to go. The fact that everything is already built in and can be 100% integrated into the Windows ADDS environment is also lowering the TCO.

KR,

Grimeton

Heliborn

Thank you for a very informative post. This may not be the implementation I am looking to do, but it gave me a good idea for another project I will be doing down the line.

Phonebuff

Is this a TYPO or do the Windows 7 versions not support this ??

You need Windows Vista or Windows 2008 or later.

TIA –