Outbound NAT return packet gets dropped on IPSec VTI tunnel

[Chluz]

Hi all,

I've switched over from a OpenVPN to a IPSec with VTI configuration, to get improved VPN performance on apu2 devices. This is setup on 2.4.5 PFSense routers on both sides.

The tunnel is working fine (except for errors of the type <con4000|7> querying policy 0.0.0.0/0|/0 === 0.0.0.0/0|/0 in failed, not found popping up every few minutes, but reading around this seems cosmetic).

I would like to port forward requests on public IPs on one side of the tunnel to a server on the other side of the tunnel, effectively doing this.
[internet client -> PFSense1 port forward to server 1 -> IPsec tunnel -> PFSense 2 -> server1]. I am however having issues with the outbound nat I have to use. tcpdump show packets coming back to the src IP used for outbound nat, and then disappearing.

Detail
Reading the PfSense documentation for VTI, and a few threads, namely here https://forum.netgate.com/topic/148771/solved-port-forwarding-across-ipsec-tunnel/9 and here https://forum.netgate.com/topic/139593/traffic-from-internet-through-ipsec-vti-not-returning-the-same-way/18 I understand that reply_to will not work through VTI.
Because of this, the verified return path for the packets returning from the port forward are [server1 -> PFSense2 (default gateway)->internet client].

Not great, but I enjoy the performance improvements of VTI, and so attempted to setup a outbound NAT on the IPSec tunnel so that packets coming through the IPSec tunnel for the forwarded port to the target Server 1 get as a source IP the PFSense1 IPSec tunnel IP. This is what is suggested on the threads listed above.

Using TCP dump, I now see the packets coming through the PFSense1 WAN interface, going through the IPSec tunnel, reaching the server1, and returning. I see the packet reaching the PFSense1 IPSec tunnel IP, as it should, and then nothing, it stops there.
In other words, going in I get:
[internet client -> PFSense1 port forward to server 1 -> IPsec tunnel -> PFSense 2 -> server1].
And coming back I get
[server 1 -> PFSense2 -> IPsec tunnel -> PFSense 1 IPsec tunnel IP ]. and then nothing.

I feel like there is something obvious that I am missing. That said I did find this thread https://forum.netgate.com/topic/143649/nat-over-routed-vti , which although starting from a weird IP subnet arrangement, seems to have the same issue as me.

Workaround
At the moment, I created an additional OpenVPN Peer to Peer tunnel between Pfsense1 and Pfsense2 as a workaround. I am doing a port forward to from PFSense 1 WAN the PFsense 2 OpenVPN IP (to force it to go through the OpenVPN tunnel) and have an outbound nat on the PFsense 1 so that outgoing packets through the openvpn tunnel get a PFsense 1 openvpn ip source address. On Pfsense 2 I then have a port forward for incoming packets to the Pfsense 2 openvpn IP to server 1.
In other words, this works :
[internet client -> PFSense1 port forward to Pfsense 2 openvpn IP (with outbound nat PFSense 1 openvpn ip source IP)-> OpenVPN tunnel tunnel -> PFSense 2 port forward to server 1 -> server1].
This is a pretty complex and clumsy setup though, and I would love to have your opinion about the issue with outbound nat on the IPsec tunnel.
Thanks in advance,

[jimp]

That type of config isn't possible on VTI yet. It doesn't respect per-interface rules nor does it properly use reply-to and some NAT cases.

[Chluz]

Hi Jimp, thanks for the answer.
What would be the best alternative then, would using IPSec + Gre allow this type of NAT (or better yet, working reply_to) and show an interface for the tunnel that I can select as a gateway for outgoing LAN rules (to force certain connections to the internet from server1 to go through the tunnel and out through PFsense 1)

[jimp]

IPsec+GRE has its own problems (mainly with transport mode)... Maybe if you built a GRE between VTI endpoints. I don't know that anyone has tried that yet, however.

[Chluz]

I tried the GRE tunnel over VTI, but it did not seem to want to route anything outside of the GRE tunnel IPs.
Here are the settings i used:

VTI
VTI Ip PfSense 1: 10.6.106.1
VTI Ip PfSense 2: 10.6.106.2

GRE PfSense1
Parent interface IpSec1000
GRE Remote Address 10.6.106.2
GRE tunnel local address 10.6.111.1
GRE tunnel remote address 10.6.111.2

GRE PfSense2
Parent interface IpSec2000
GRE Remote Address 10.6.106.1
GRE tunnel local address 10.6.111.2
GRE tunnel remote address 10.6.111.1

I allowed all on both sides for the firewall, enable the GRE interfaces on both sides and setup a gateway. after adding routes, I could ping 10.6.111.1/10.6.111.2 from both PfSense 1 and 2, but none of the local subnets from the routes I added.

[jimp]

Did you confirm with packet captures that things were taking the proper paths? And check the firewall logs? firewall states?