Adding IPSec to GRE Tunnel breaks TCP connections

[MLIT]

I've attached a diagram below of my test network. Anyway, I'm attempting to use IPSec to encrypt a GRE tunnel. Anyway, my network is configured as such:

R1:

WAN: 12.12.12.1/24
LAN: 10.10.1.0/24
GRE: 10.10.10.1/30 (Assigned to interface OPT2)

R2:

WAN: 12.12.12.2/24
LAN: 10.11.1.0/24
GRE: 10.10.10.2/30 (Assigned to interface OPT2)

Anyway, before I turn on IPSec, everything works as expected. I can ping and connect with TCP between 10.10.1.0/24 and 10.11.1.0/24 .

Okay now that I turn on IPSec to encrypt the tunnels, I can ping between 10.10.1.200 and 10.11.1.200 (CentOS Box 1 & 2) fine. I've verified the ICMP packets are making it end to end with tcpdump. Anyway, when I try to SSH from 10.10.1.200 (Centos Box 1) to 10.11.1.200 (Centos Box 2), I see the SYN packet from Box 1 trying to set up a TCP session then I see an ICMP destination unreachable packet from R2 indicating that CentOS Box 1 cannot be reached. I will then see several more SYN packets from Box 1 (It is retrying) followed by ICMP destination unreachables from R2 until Box 1 finally gives up.

So why can't R2 route TCP traffic back to the other subnet but it can route ICMP traffic back? This is not a firewall issue as I've been extremely generous with my allows on all interfaces. Any ideas or help here is much appreciated. Thank you!

[doktornotor]

@MLIT:

So why can't R2 route TCP traffic back to the other subnet but it can route ICMP traffic back

MTU?

[MLIT]

@doktornotor:

MTU?

I might believe that if it was dying further into the session. This is not relaying the SYN, ACK packet back to the other end however.

Just to double check I set the MTU of both CentOS boxes to 1000 (I have ethernet between everything) and the same thing occurs.

[pfsuser246]

Did this ever get resolved?

I am seeing the same thing with GRE over IPSec, with PFSense 2.3.1-RELEASE-p5.

If I disable pf (pfctl -d) traffic flows as expected
If I remove IPsec transport mode for the GRE tunnel, traffic flows as expected (with pf enabled or disabled)

There seem to be a few posts relating to issues with GRE over IPsec, so maybe there is a deeper issue:
Bug 207598 - pf adds icmp unreach on gre/ipsec somehow
(https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=207598)
Although this one seems to be related to MTU size

Packets from gre interface bypassing PF?
https://forums.freebsd.org/threads/55181/
This might be related if the state tables are incorrect if packets flow via GRE into a pfsense firewall.

Is anyone successfully using GRE over an IPSec tunnel with firewalling enabled?

Kind regards,
Mike

[jimp]

I've seen a couple customers running it with success and I don't recall them needing any specific workarounds.

If there is an issue with your configuration and states not being added correctly, which may mimic the symptoms of asymmetric routing, you can solve that with the same rules used to allow asymmetric routing: https://doc.pfsense.org/index.php/Asymmetric_Routing_and_Firewall_Rules#Manual_Fix

[jammcla]

2.3.1-RELEASE-p5(amd64)

On the link jimp posted:

I tried the manual fix for my GRE Tunnel over IPSEC and it allowed the traffic through.  Tried the Automatic Fix and it didn't work, so will have to do the manual fix for all the traffic.

I see ticket 4479 talks about the issue:

https://redmine.pfsense.org/issues/4479

So trying to dig into this a bit further:

While creating rules to allow the traffic I ended up creating both rules on the Floating tab.

Rule 1:
GRE Interface, direction out, Source was the local network, destination was the remote network, any TCP flags, and Sloppy State

Rule 2:
Local Network interface,  direction in, source was the Remote network, destination was the local network, any TCP Flags, and Sloppy State