PFSense 2.5 problems with Site-to-Site AWS VPN connection

[jutley]

Last week, we stood up a pair of bare metal PFSense 2.5 servers in HA mode, to bridge traffic between a VLAN in our colo and a VPC in AWS using their managed Site-To-Site VPN service. However, since trying to set up the VPN connection, we have had nothing but very strange problems. The hardware are SuperMicro servers with Dual Xeon E5620 CPUs, and Intel 10GBit interfaces on both the WAN and LAN. Basic networking for the servers is working beautifully right now. I configured the VPN on Amazon's side per the instructions, downloaded the PFSense configuration instructions, and set up the IPSec VPN connection per those instructions, however the VPN will never connect. In fact, if I start a packet capture of all traffic to the AWS VPN endpoint, then go to Status-IPSec-Overview and click the "Connect VPN" button, I actually see zero traffic to the AWS VPN endpoint whatsoever - so to the best of my knowledge, it never even attempts to make the connection.

swanctl --list-conns:

bypass: IKEv1/2, no reauthentication, rekeying every 14400s
  local:  %any
  remote: 127.0.0.1
  local unspecified authentication:
  remote unspecified authentication:
  bypasslan: PASS, no rekeying
    local:  172.31.92.0/24|/0
    remote: 172.31.92.0/24|/0
con100000: IKEv2, no reauthentication, rekeying every 25920s, dpd delay 10s
  local:  66.152.77.120
  remote: 52.207.141.26
  local pre-shared key authentication:
    id: 66.152.77.120
  remote pre-shared key authentication:
    id: 52.207.141.26
  con100000: TUNNEL, rekeying every 3240s, dpd action is hold
    local:  172.31.92.0/24|/0
    remote: 10.50.0.0/16|/0

swanctl --load-all --file /var/etc/ipsec/swanctl.conf --debug 1:

loaded ike secret 'ike-0'
no authorities found, 0 unloaded
no pools found, 0 unloaded
loaded connection 'bypass'
loaded connection 'con100000'
successfully loaded 2 connections, 0 unloaded

/var/etc/ipsec/swanctl.conf:

connections {
	bypass {
		remote_addrs = 127.0.0.1
		children {
			bypasslan {
				local_ts = 172.31.92.0/24
				remote_ts = 172.31.92.0/24
				mode = pass
				start_action = trap
			}
		}
	}
	con100000 {
		fragmentation = yes
		unique = replace
		version = 2
		proposals = aes256-sha512-ecp521
		dpd_delay = 10s
		dpd_timeout = 40s
		rekey_time = 25920s
		reauth_time = 0s
		over_time = 2880s
		rand_time = 2880s
		encap = no
		mobike = no
		local_addrs = 66.152.77.120
		remote_addrs = 52.207.141.26
		pools =
		local {
			id = 66.152.77.120
			auth = psk
		}
		remote {
			id = 52.207.141.26
			auth = psk
		}
		children {
			con100000 {
				dpd_action = trap
				mode = tunnel
				policies = yes
				life_time = 3600s
				rekey_time = 3240s
				rand_time = 360s
				start_action = trap
				remote_ts = 10.50.0.0/16
				local_ts = 172.31.92.0/24
				esp_proposals = aes256-sha512-ecp521
			}
		}
	}
}
secrets {
	ike-0 {
		secret = 0sOHQyWGROdFE4ZzBfSXhrQW5pYm1wcEc1b1YyQ05nWHQ=
		id-0 = %any
		id-1 = 52.207.141.26
	}
}

What's interesting to me is the ipsec.log file - I see no entries for IKE - it's mostly CFG entries, with a few KNL, LIB, JOB entries. A dump of my log from earlier testing is available at:

https://pastebin.com/F8XHwvaw

Some other troubleshooting I've done:

Connected the primary of these machines with IPSec to another, older PFSense server (2.4.5) successfully

Connected that older PFSense 2.4.5 to an identically configured AWS Site-To-Site VPN in our dev account without difficulty.

My next test I'm about to do is stand up a new 2.5 pfsense and try to connect it to the same VPN in our dev account to see if that works.

Does anyone have any ideas I have not thought of?

[jimp]

Do you see it try to connect if you initiate via traffic (e.g. ping from 172.31.92.x to 10.50.x.x)?

There are several issues with IPsec on 21.02/2.5 that might affect what you are seeing, such as one bug which prevents the connect button on the status page from working properly in cases like yours.

You can install the System Patches package and then create entries for the following commit IDs to apply the fixes:

• ead6515637a34ce6e170e2d2b0802e4fa1e63a00 #11435
• 57beb9ad8ca11703778fc483c7cba0f6770657ac #11435
• 10eb04259fd139c62e08df8de877b71fdd0eedc8 #11442
• ded7970ba57a99767e08243103e55d8a58edfc35 #11486
• afffe759c4fd19fe6b8311196f4b6d5e288ea4fb #11487
• 2fe5cc52bd881ed26723a81e0eed848fd505fba6 #11488

[jutley]

Thanks Jim!

Those patches seem to have done the trick! Our S2S VPN is now up and passing traffic between our in-house DC Vlan and an EC2 instance inside the AWS VPC. Have some issues with IPSec failover (shift the primary into CARP maintenance mode, and we only drop a single ping during the swithover, but put it back to normal, and traffic stops) but I will investigate that separately.

Quick question - will applying those patches imply any additional steps when we upgrade to the next release?

[jimp]

No, when there is another release you can simply remove those patches (don't revert them, just remove the entries).