IPSEC Transport Mode brings down GRE Tunnel

xtropx

Hello.
I have a GRE tunnel between two sites set up and working fine. Using OSPFd to transmit routes between them. I set up IPSEC in transport mode using the public IP addresses between sites and instantly I can see the GRE tunnel go down. I check status > gateways and they are offline. Nothing in IPSEC log that would indicate a problem with the IPSEC tunnel itself. status > ipsec does have the yellow X "error" but appears to set up properly.

I am just wondering if there is a better way from a design perspective to do this, whether it be with pfsense or in some other fashion. I was looking into OpenVPN but I am unsure whether I can get OSPFd working over that tunnel either. If all else fails I will just do some redesign of IP addressing and use a summary route over IPSEC in tunnel mode, or with OpenVPN, but I would like to continue to use OSPFd if possible. Thoughts?

cmb

That means your IPsec isn't setup correctly. That scenario works where transport mode IPsec is properly configured. It also works with OpenVPN.

xtropx

Hi I am back again. I've been using IPSEC in tunnel mode for a while but I am giving transport another go.
I have tried again and I cannot get IPSEC transport mode to come up.
I have disabled IPSEC ESP and am just using AH for the time being.
I have allowed both protocols on the WAN interface (ESP & HA) from the public IP address of each side to "any" (as well as ICMP from either sides)
Prefer old IPSEC SAs is OFF

I have:
IP: IPv4
INTERFACE: WAN
REMOTE GATEWAY: PUBLIC IP OF OTHER SIDE
AUTHENTICATION PROTOCOL: Mutual PSK
NEGOTIATION: Agressive
MY IDENTIFIER: My IP Address
PEER IDENTIFIER: Peer IP Address
PRESHARED KEY: <psk>(COPY & PASTED, THEY ARE THE SAME)
POLICY GENERATION: Default
PROPOSAL CHECKING: Default
ENCRYPTION ALGORITHM: 3DES
HASH ALGORITHM: SHA384
DH KEY GROUP: 2(1024 bit)
LIFETIME: 28800
NAT TRAVERSAL: DISABLE
DEAD PEER: UNCHECKED

And for Phase 2:

MODE: TRANSPORT
PROTOCOL: AH
HASH ALGORITHMS: MD5
PFS KEY GROUP: OFF
LIFETIME: 86400
AUTOMATICALLY PING HOST: BLANK

I know in IPSEC it is CRITICAL to make sure sides match, so I have ensured. I've deleted the SPD on both sides and restart racoon and still comes up with "error" under Status > IPSEC. No obvious errors in the logs (Ive googled just about everything in there)
GRE is up and running, with OSPF over it. I can ping/access my remote subnets, but it breaks when I turn on IPSEC. I'd be really grateful for any ideas!</psk>