(IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP

kevindd992002 · Dec 20, 2020, 1:13 AM

I have two sites connected by an IKEv2/IPsec tunnel (routed VTI). Site A is the main site and has a static public IP assigned to it. Site B is behind a CGNAT. To access servers on Site B, I do:

external source IP -> Site A public IP -> Site A port forward (DNAT) to Site B server private IP ->Site A outbound NAT (SNAT) in IPsec interface for Site B server private IP using interface address

When I do a packet capture in Site A's IPsec interface, I see forward and return traffic properly as expected. This means that SNAT is working (at least that's what I thought). But we're sure that return traffic goes back properly from Site B to Site A. The problem is I don't see the destination IP's of the return traffic being translated back to the original source IP prior the SNAT'ting.

09:06:32.294093 IP 10.0.2.1.40083 > 192.168.20.101.62958: tcp 0
09:06:32.299220 IP 192.168.20.101.62958 > 10.0.2.1.40083: tcp 0
09:06:33.292736 IP 10.0.2.1.40083 > 192.168.20.101.62958: tcp 0
09:06:33.295234 IP 10.0.2.1.8026 > 192.168.20.101.62958: tcp 0
09:06:33.297741 IP 192.168.20.101.62958 > 10.0.2.1.40083: tcp 0
09:06:33.299987 IP 192.168.20.101.62958 > 10.0.2.1.8026: tcp 0
09:06:34.292689 IP 10.0.2.1.8026 > 192.168.20.101.62958: tcp 0
09:06:34.296383 IP 10.0.2.1.16857 > 192.168.20.101.62958: tcp 0
09:06:34.297697 IP 192.168.20.101.62958 > 10.0.2.1.8026: tcp 0
09:06:34.301195 IP 192.168.20.101.62958 > 10.0.2.1.16857: tcp 0
09:06:34.315697 IP 192.168.20.101.62958 > 10.0.2.1.40083: tcp 0
09:06:35.292730 IP 10.0.2.1.16857 > 192.168.20.101.62958: tcp 0
09:06:35.297737 IP 192.168.20.101.62958 > 10.0.2.1.16857: tcp 0
09:06:35.307705 IP 192.168.20.101.62958 > 10.0.2.1.8026: tcp 0
09:06:36.300054 IP 192.168.20.101.62958 > 10.0.2.1.16857: tcp 0
09:06:36.332062 IP 192.168.20.101.62958 > 10.0.2.1.40083: tcp 0
09:06:37.324060 IP 192.168.20.101.62958 > 10.0.2.1.8026: tcp 0

But when I do a packet capture on Site A's WAN interface, I don't see any return traffic:

09:09:17.769276 IP 198.199.98.246.41770 > {my public IP}.62958: tcp 0
09:09:18.766709 IP 198.199.98.246.41770 > {my public IP}.62958: tcp 0
09:09:18.770395 IP 198.199.98.246.41771 > {my public IP}.62958: tcp 0
09:09:19.766651 IP 198.199.98.246.41771 > {my public IP}.62958: tcp 0
09:09:19.771684 IP 198.199.98.246.41772 > {my public IP}.62958: tcp 0
09:09:20.770742 IP 198.199.98.246.41772 > {my public IP}.62958: tcp 0

So this tells me that the either the return traffic is lost somewhere in Site A's pfsense box or the outbound NAT table somehow doesn't preserve the original external source IP in its table and that the return traffic destination IP doesn't get translated back to that external IP?

This tool is what I usually use to test:

https://www.yougetsignal.com/tools/open-ports/

kevindd992002 · Dec 20, 2020, 1:14 AM

@jimp do you have any ideas? I know in your guides/hangouts you said that interface address outbound NAT should work through the IPsec tunnel. I have this same setup with OpenVPN and it was working fine with or without outbound NAT (because it supports reply-to's).

kevindd992002 · Dec 20, 2020, 2:41 AM

Outbound NAT table looks good:

🔒 Log in to view

But then their states are closed. I'm assuming because of the no return packets issue.

kevindd992002 · Dec 20, 2020, 1:22 PM

If I create an outbound NAT rule in the IPsec interface in Site A from any of its local subnet to any device in the Site B's subnet and do a test trying to access the servers in Site B from Site A, the same exact behavior happens. The return traffic reaches the Site A tunnel interface but is somewhat dropped preventing return traffic from being able to reach the source. Why is this happening?

kevindd992002 · Dec 26, 2020, 2:59 PM

BUMP! Anybody please?

kevindd992002 · Jan 4, 2021, 12:35 PM

Anybody can help, please?

stephenw10 · Jan 4, 2021, 5:44 PM

You can't outbound NAT like that on the IPSec interface the reply traffic will never hit it and will just get dropped. Which is what you're seeing.

If you need to NAT on IPSec it has to be done on the P2 policy but you can't do that for this situation because the source would have to be 'any'.

You can't do it with routed IPSec either becaise of no reply-to as you saw.

Use OpenVPN in a situation like that and you can port forward dircetly and use reply-to to avoid asymmetry.

Steve

kevindd992002 · Jan 4, 2021, 11:32 PM

@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:

You can't outbound NAT like that on the IPSec interface the reply traffic will never hit it and will just get dropped. Which is what you're seeing.

If you need to NAT on IPSec it has to be done on the P2 policy but you can't do that for this situation because the source would have to be 'any'.

You can't do it with routed IPSec either becaise of no reply-to as you saw.

Use OpenVPN in a situation like that and you can port forward dircetly and use reply-to to avoid asymmetry.

Steve

But in your routed IPsec guide, it specifically says that outbound NAT to interface address works. Is that clause not supposed to be there then? Some references even say that the workaround to the reply-to issue for IPsec is outbound NAT, just like what I'm doing.

What is the reason why traffic gets dropped? The return traffic does get to the IPsec interface as seen in the packet capture but get dropped on that interface itself.

I switched from OpenVPN to IPsec because of the sole purpose of needing more bandwidth to saturate my 100Mbps link between the two sites. I'm only using an APU2C4 for pfsense on two sites and with OpenVPN, I'm only getting 50Mbps so that won't cut it.

kevindd992002 · Jan 4, 2021, 11:37 PM

Also @jimp mentioned this to be a workaround but it isn't working for me. I've posted the issues I've experienced with that workaround in another thread starting with this one:

https://forum.netgate.com/post/953675

Thanks.

stephenw10 · Jan 5, 2021, 1:37 AM

Ah, sorry I missed you were using VTI here. You still can't do it by default as you found.

You can do outbound NAT from the tunnel subnet or from subnet routed over it. You can port forward to IPs a remopte subnet. You can't do it on the tunnel interfaces because the filtering does not parse traffic equally.

If you only have route based IPSec you could try setting those sysctls. They can be applied via System Tunables ion the GUI. It will break filtering for policy based tunnels though.

Steve

kevindd992002 · Jan 5, 2021, 2:00 AM

@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:

Ah, sorry I missed you were using VTI here. You still can't do it by default as you found.

Yeah, I'm not using the traditional policy-based IPsec. I'm using routed VTI. I thought that's what you meant by "route based" IPsec in your earlier reply? Aren't those one and the same?

You can do outbound NAT from the tunnel subnet or from subnet routed over it. You can port forward to IPs a remopte subnet. You can't do it on the tunnel interfaces because the filtering does not parse traffic equally.

Sorry, what do you mean? Yes, I do have a port forward to a remote subnet like so:

🔒 Log in to view

When your documentation said that outbound NAT to interface address in the tunnel interface works, what does it mean? I mean, it does work like I showed, it does translate the original source IP to the tunnel interface address, but the problem is the return traffic doesn't get handled properly.

If you only have route based IPSec you could try setting those sysctls. They can be applied via System Tunables ion the GUI. It will break filtering for policy based tunnels though.

Steve

I already did try setting those and it breaks the normal site-to-site routing as I explained here. And yes, that workaround would've been perfect for me because I do not use policy based tunnels so no issues in that. But I can't make it work. With the workaround in place, I can't even do normal pings from any device on either either site to any device on either site.

kevindd992002 · Jan 5, 2021, 2:20 AM

Most of the port forward and outbound NAT rules that I have are posted here:

https://forum.netgate.com/post/953857

kevindd992002 · Jan 7, 2021, 7:44 AM

@stephenw10 Do you have any more ideas regarding this?

stephenw10 · Jan 8, 2021, 3:24 PM

Yes, route-based IPSec is VTI,

No, you can't outbound NAT on a VTI interface because, as you found, the reply traffic will not be translated back.

You should use OpenVPN for this if you need to configure it as you have now.

Steve

kevindd992002 · Jan 8, 2021, 3:33 PM

@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:

Yes, route-based IPSec is VTI,

No, you can't outbound NAT on a VTI interface because, as you found, the reply traffic will not be translated back.

You should use OpenVPN for this if you need to configure it as you have now.

Steve

I actually was using OpenVPN without any issues except for the fact that I upgraded my ISP subscription to 100Mbps and OpenVPN cannot saturate that bandwidth as my APU2C4 does not a powerful single-core performance. This is where IPsec comes in as it does saturate the whole 100Mbps between the two sites.

So my question now is, why do those sysctl workarounds not work for me? They would've been perfect because I'm not using policy-based IPsec.

stephenw10 · Jan 8, 2021, 4:33 PM

Unclear. Check the firewall states that are created in each case at each end. Something is still mismatched I would suggest. Or not creating a state at all which should then show as blocked unless you have disabled logging default blocks or added your own block rule without logging.

kevindd992002 · Jan 8, 2021, 4:41 PM

@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:

Unclear. Check the firewall states that are created in each case at each end. Something is still mismatched I would suggest. Or not creating a state at all which should then show as blocked unless you have disabled logging default blocks or added your own block rule without logging.

Ok, I will check the firewall states and report back tomorrow (alreqdy midnight from where I am). I did not disable logging default blockd and did not add any block rule without logging so I should see the logs you're expecting.

But just to be clear though, the sysctl workarounds that I'm referring are the correct workarounds for what I'm trying to solve, yes?

stephenw10 · Jan 8, 2021, 5:24 PM

There is no 'correct' workaround here. This might work for you, it appears to have worked for others. It might break at upgrade etc, it's not something we test as NAT on VTI is expected to fail.

By moving the filtering from enc to if_ipsec it should mean traffic is passed outbound on ipsecX and replies are also passed there. No states on enc0.

Using OpenVPN here is the only thing expected to work.

Steve

kevindd992002 · Feb 19, 2021, 1:07 PM

@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:

There is no 'correct' workaround here. This might work for you, it appears to have worked for others. It might break at upgrade etc, it's not something we test as NAT on VTI is expected to fail.

By moving the filtering from enc to if_ipsec it should mean traffic is passed outbound on ipsecX and replies are also passed there. No states on enc0.

Using OpenVPN here is the only thing expected to work.

Steve

Since Wireguard is now available in pfsense 2.5, will it solve my issue with NAT over VPN?

stephenw10 · Feb 19, 2021, 3:56 PM

Yes, if you can use WireGuard you can happily route, NAT or whatever across it.