(IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP
-
I have two sites connected by an IKEv2/IPsec tunnel (routed VTI). Site A is the main site and has a static public IP assigned to it. Site B is behind a CGNAT. To access servers on Site B, I do:
external source IP -> Site A public IP -> Site A port forward (DNAT) to Site B server private IP ->Site A outbound NAT (SNAT) in IPsec interface for Site B server private IP using interface address
- When I do a packet capture in Site A's IPsec interface, I see forward and return traffic properly as expected. This means that SNAT is working (at least that's what I thought). But we're sure that return traffic goes back properly from Site B to Site A. The problem is I don't see the destination IP's of the return traffic being translated back to the original source IP prior the SNAT'ting.
09:06:32.294093 IP 10.0.2.1.40083 > 192.168.20.101.62958: tcp 0 09:06:32.299220 IP 192.168.20.101.62958 > 10.0.2.1.40083: tcp 0 09:06:33.292736 IP 10.0.2.1.40083 > 192.168.20.101.62958: tcp 0 09:06:33.295234 IP 10.0.2.1.8026 > 192.168.20.101.62958: tcp 0 09:06:33.297741 IP 192.168.20.101.62958 > 10.0.2.1.40083: tcp 0 09:06:33.299987 IP 192.168.20.101.62958 > 10.0.2.1.8026: tcp 0 09:06:34.292689 IP 10.0.2.1.8026 > 192.168.20.101.62958: tcp 0 09:06:34.296383 IP 10.0.2.1.16857 > 192.168.20.101.62958: tcp 0 09:06:34.297697 IP 192.168.20.101.62958 > 10.0.2.1.8026: tcp 0 09:06:34.301195 IP 192.168.20.101.62958 > 10.0.2.1.16857: tcp 0 09:06:34.315697 IP 192.168.20.101.62958 > 10.0.2.1.40083: tcp 0 09:06:35.292730 IP 10.0.2.1.16857 > 192.168.20.101.62958: tcp 0 09:06:35.297737 IP 192.168.20.101.62958 > 10.0.2.1.16857: tcp 0 09:06:35.307705 IP 192.168.20.101.62958 > 10.0.2.1.8026: tcp 0 09:06:36.300054 IP 192.168.20.101.62958 > 10.0.2.1.16857: tcp 0 09:06:36.332062 IP 192.168.20.101.62958 > 10.0.2.1.40083: tcp 0 09:06:37.324060 IP 192.168.20.101.62958 > 10.0.2.1.8026: tcp 0
- But when I do a packet capture on Site A's WAN interface, I don't see any return traffic:
09:09:17.769276 IP 198.199.98.246.41770 > {my public IP}.62958: tcp 0 09:09:18.766709 IP 198.199.98.246.41770 > {my public IP}.62958: tcp 0 09:09:18.770395 IP 198.199.98.246.41771 > {my public IP}.62958: tcp 0 09:09:19.766651 IP 198.199.98.246.41771 > {my public IP}.62958: tcp 0 09:09:19.771684 IP 198.199.98.246.41772 > {my public IP}.62958: tcp 0 09:09:20.770742 IP 198.199.98.246.41772 > {my public IP}.62958: tcp 0
So this tells me that the either the return traffic is lost somewhere in Site A's pfsense box or the outbound NAT table somehow doesn't preserve the original external source IP in its table and that the return traffic destination IP doesn't get translated back to that external IP?
This tool is what I usually use to test:
https://www.yougetsignal.com/tools/open-ports/
-
@jimp do you have any ideas? I know in your guides/hangouts you said that interface address outbound NAT should work through the IPsec tunnel. I have this same setup with OpenVPN and it was working fine with or without outbound NAT (because it supports reply-to's).
-
Outbound NAT table looks good:
But then their states are closed. I'm assuming because of the no return packets issue.
-
If I create an outbound NAT rule in the IPsec interface in Site A from any of its local subnet to any device in the Site B's subnet and do a test trying to access the servers in Site B from Site A, the same exact behavior happens. The return traffic reaches the Site A tunnel interface but is somewhat dropped preventing return traffic from being able to reach the source. Why is this happening?
-
BUMP! Anybody please?
-
Anybody can help, please?
-
You can't outbound NAT like that on the IPSec interface the reply traffic will never hit it and will just get dropped. Which is what you're seeing.
If you need to NAT on IPSec it has to be done on the P2 policy but you can't do that for this situation because the source would have to be 'any'.
You can't do it with routed IPSec either becaise of no reply-to as you saw.
Use OpenVPN in a situation like that and you can port forward dircetly and use reply-to to avoid asymmetry.
Steve
-
@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:
You can't outbound NAT like that on the IPSec interface the reply traffic will never hit it and will just get dropped. Which is what you're seeing.
If you need to NAT on IPSec it has to be done on the P2 policy but you can't do that for this situation because the source would have to be 'any'.
You can't do it with routed IPSec either becaise of no reply-to as you saw.
Use OpenVPN in a situation like that and you can port forward dircetly and use reply-to to avoid asymmetry.
Steve
But in your routed IPsec guide, it specifically says that outbound NAT to interface address works. Is that clause not supposed to be there then? Some references even say that the workaround to the reply-to issue for IPsec is outbound NAT, just like what I'm doing.
What is the reason why traffic gets dropped? The return traffic does get to the IPsec interface as seen in the packet capture but get dropped on that interface itself.
I switched from OpenVPN to IPsec because of the sole purpose of needing more bandwidth to saturate my 100Mbps link between the two sites. I'm only using an APU2C4 for pfsense on two sites and with OpenVPN, I'm only getting 50Mbps so that won't cut it.
-
-
Ah, sorry I missed you were using VTI here. You still can't do it by default as you found.
You can do outbound NAT from the tunnel subnet or from subnet routed over it. You can port forward to IPs a remopte subnet. You can't do it on the tunnel interfaces because the filtering does not parse traffic equally.
If you only have route based IPSec you could try setting those sysctls. They can be applied via System Tunables ion the GUI. It will break filtering for policy based tunnels though.
Steve
-
@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:
Ah, sorry I missed you were using VTI here. You still can't do it by default as you found.
Yeah, I'm not using the traditional policy-based IPsec. I'm using routed VTI. I thought that's what you meant by "route based" IPsec in your earlier reply? Aren't those one and the same?
You can do outbound NAT from the tunnel subnet or from subnet routed over it. You can port forward to IPs a remopte subnet. You can't do it on the tunnel interfaces because the filtering does not parse traffic equally.
Sorry, what do you mean? Yes, I do have a port forward to a remote subnet like so:
When your documentation said that outbound NAT to interface address in the tunnel interface works, what does it mean? I mean, it does work like I showed, it does translate the original source IP to the tunnel interface address, but the problem is the return traffic doesn't get handled properly.
If you only have route based IPSec you could try setting those sysctls. They can be applied via System Tunables ion the GUI. It will break filtering for policy based tunnels though.
Steve
I already did try setting those and it breaks the normal site-to-site routing as I explained here. And yes, that workaround would've been perfect for me because I do not use policy based tunnels so no issues in that. But I can't make it work. With the workaround in place, I can't even do normal pings from any device on either either site to any device on either site.
-
Most of the port forward and outbound NAT rules that I have are posted here:
https://forum.netgate.com/post/953857
-
@stephenw10 Do you have any more ideas regarding this?
-
Yes, route-based IPSec is VTI,
No, you can't outbound NAT on a VTI interface because, as you found, the reply traffic will not be translated back.
You should use OpenVPN for this if you need to configure it as you have now.
Steve
-
@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:
Yes, route-based IPSec is VTI,
No, you can't outbound NAT on a VTI interface because, as you found, the reply traffic will not be translated back.
You should use OpenVPN for this if you need to configure it as you have now.
Steve
I actually was using OpenVPN without any issues except for the fact that I upgraded my ISP subscription to 100Mbps and OpenVPN cannot saturate that bandwidth as my APU2C4 does not a powerful single-core performance. This is where IPsec comes in as it does saturate the whole 100Mbps between the two sites.
So my question now is, why do those sysctl workarounds not work for me? They would've been perfect because I'm not using policy-based IPsec.
-
Unclear. Check the firewall states that are created in each case at each end. Something is still mismatched I would suggest. Or not creating a state at all which should then show as blocked unless you have disabled logging default blocks or added your own block rule without logging.
-
@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:
Unclear. Check the firewall states that are created in each case at each end. Something is still mismatched I would suggest. Or not creating a state at all which should then show as blocked unless you have disabled logging default blocks or added your own block rule without logging.
Ok, I will check the firewall states and report back tomorrow (alreqdy midnight from where I am). I did not disable logging default blockd and did not add any block rule without logging so I should see the logs you're expecting.
But just to be clear though, the sysctl workarounds that I'm referring are the correct workarounds for what I'm trying to solve, yes?
-
There is no 'correct' workaround here. This might work for you, it appears to have worked for others. It might break at upgrade etc, it's not something we test as NAT on VTI is expected to fail.
By moving the filtering from enc to if_ipsec it should mean traffic is passed outbound on ipsecX and replies are also passed there. No states on enc0.
Using OpenVPN here is the only thing expected to work.
Steve
-
@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:
There is no 'correct' workaround here. This might work for you, it appears to have worked for others. It might break at upgrade etc, it's not something we test as NAT on VTI is expected to fail.
By moving the filtering from enc to if_ipsec it should mean traffic is passed outbound on ipsecX and replies are also passed there. No states on enc0.
Using OpenVPN here is the only thing expected to work.
Steve
Since Wireguard is now available in pfsense 2.5, will it solve my issue with NAT over VPN?
-
Yes, if you can use WireGuard you can happily route, NAT or whatever across it.
-
If you only have VTI IPsec (no tunnel mode P2 entries) you can also try the patch on https://redmine.pfsense.org/issues/11395 and switch the new option over to filter only on VTI interfaces. Then you can use NAT and rules on assigned VTI interfaces.
-
@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:
Yes, if you can use WireGuard you can happily route, NAT or whatever across it.
@jimp said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:
If you only have VTI IPsec (no tunnel mode P2 entries) you can also try the patch on https://redmine.pfsense.org/issues/11395 and switch the new option over to filter only on VTI interfaces. Then you can use NAT and rules on assigned VTI interfaces.
This is great! WireGuard as a solution and a patch with IPsec!
Is WireGuard a lot better than IPsec though? Is it recommended to switch over if I'll just need to use it for a single S2S connection?
-
Wireguard is perhaps slightly more flexible that IPSec if both sides support it. That's big if!
IPSec is pretty much universally supported.
You probably won't see much difference in speed. IPSec can be at least as fast.I would probably use Wireguard here because to use IPSec you will need to use that patch and doing so will prevent you using other IPSec tunnel mode connections. If you know you will never need to do that then either should be OK.
Steve
-
@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:
Wireguard is perhaps slightly more flexible that IPSec if both sides support it. That's big if!
IPSec is pretty much universally supported.
You probably won't see much difference in speed. IPSec can be at least as fast.I would probably use Wireguard here because to use IPSec you will need to use that patch and doing so will prevent you using other IPSec tunnel mode connections. If you know you will never need to do that then either should be OK.
Steve
I agree. Ok, time to try WireGuard. This is the first tutorial that's been published by a Youtuber since the release of 2.5. He admits that he is not a WireGuard expert and is open to corrections of his mistakes (if there are any).
I hope you guys can have an official documentation about WireGuard very soon.
-
Yeah, that looks good. I see nothing obviously wrong with that walk-through.
Steve
-
@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:
Yeah, that looks good. I see nothing obviously wrong with that walk-through.
Steve
Great! I have a few concerns though:
-
In the Peer setup, shouldn't the "Peer WireGuard Address" be just a single IP (tunnel interface of the far side peer), instead of the IP in CIDR notation that he did on the video?
-
When he created the WAN rule for 51820, he noted that he made a mistake about putting 52180 but it still works. Any idea why? Also, why is this WAN rule not automatically defined (but hidden) just like in IPSec? I know in OpenVPN, the explicit WAN rule is also required.
-
In my use case where I have one side with a static public IP (Main Site) and another side that is behind a CGNAT (so a private IP is assigned on my WAN interace; Remote Site), would it make sense to keep the "Endpoint Address" BLANK on the Main Site peer settings so that it will accept any remote IP AND will not initiate the connection?
-
-
The peer IP address is not actually required for WireGuard it routes internally using the allowed IPs vs the peer public keys.
Adding that there adds routes in pfSense automatically. It's a single IP address inside that subnet, I would not expect that to make any difference.You only actually need a firewall rule on one end to establish the tunnel. The reply traffic then uses that state so traffic goes both ways. It's better to have the rule correct at both ends though so the tunnels can establish from either end.
There are no hidden auto-rules like there are for IPSec. It's like OpenVPN in that regard. It would be better (IMO!) to have IPSec like that too but it would be a POLA violation to change that now.Yes, with one end behind NAT you can only establish outbound from there and the public IP address may change so I would leave it blank on the opposite end.
Steve
-
Ok, so are you saying that if I put the far end tunnel interface address in the Peer IP Address field but not in the Allowed IP's list (like how it was done it that video), it won't add a route for that tunnel interface address? If so, what is the use the case of the Peer IP Address field if it won't be used anyway?
-
If you add the peer IP address pfSense will add it as a gateway and you can use it for policy routing or static routes etc. You can do that manually but the scripts try to make it easier. More like OpenVPN behaves.
Steve
-
Ok, that makes sense. Then I would need that for my use case because I definitely will use the tunnel interface as gateways for routing.
Is it mandatory to put the the tunnel interface address in the allowed IP's though?
-
I figured it out. My observations are:
- Remote end tunnel interface IP in "Peer IP Address" field only (tunnel transit network not in "Allowed IP's" field)
Result: Tunnel works, route to far end subnets AND to transit network are automatically created, gateway automatically created, BUT gateway monitoring to remote tunnel interface IP does not work (it doesn't respond to ping).
- Remote end tunnel interface IP in "Peer IP Address" field AND tunnel transet network in "Allowed IP's" field
Result: Everything works!
The most important thing for me that solved my return traffic issue is to create a rule in the WireGuard interface tab and NOT on the WireGuard group tab (I left this empty) for reply-to to work. This is documented in the Netgate documentation pages.
My remaining concern now is why do I get these failures in my WireGuard "Client":
They were also in the server side while I'm trying to figure out everything but now they stopped. They continued to show in the client side even though everything seems to be working. Any ideas?
-
Also, I noticed that even though I get significant amount of hits in the WireGuard interface tabs on both sides, I only see 16KB for my 51820 allow rule on the server's WAN interface tab:
Server:
Client:
Shouldn't the WAN rule show approximately the same amount of traffic?
-
If the initial connection to the other side was outbound on that peer (there is no server/client in wg) then return traffic there will use the open state. That firewall rule would not be used to create a state that would that show traffic on it.
Steve
-
@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:
If the initial connection to the other side was outbound on that peer (there is no server/client in wg) then return traffic there will use the open state. That firewall rule would not be used to create a state that would that show traffic on it.
Steve
Yup, I'm aware that there is no server/client in WG but in my use case one side is "acting" like a server and the other side is acting like a client by initiating the outbound connection. No, the initial connection was outbound from the client side (the one behind an ISP NAT, naturally) so the WAN rule I'm showing is for inbound traffic (forward traffic, not return).
-
Hmm, well I would certainly expect to see traffic on that rule then.
Check the state table for a state open on WAN with that traffic on it.
Something else must be opening it. -
I do see an open state:
client dynamic public IP:random port -> server static public IP:51820
I then disabled the WAN rule and surprisingly the tunnel still worked. This is consistent with the behavior in the Youtube video above wherein he accidentally specified a wrong destination port in the WAN rule but it still worked. Although, the only difference in this case is the open state goes like the other way around:
server static public IP:51820 -> client dynamic public IP:random port
Why did the open state reverse?
I don't have any other rules for dst_port=51820. Here are all my WAN rules:
-
Run:
pfctl -ss -vvvv
Find the state(s) open with port 51820. Note the rule number that opened them.
Now run:
pfctl -sr -vvvv
Check what rule opened the state.
Steve
-
With the WAN rule enabled, running "pfctl -sr -vvvv" shows the user rule I created and like I said, the open state is as expected:
pfctl -ss -vvvv | grep 51820 igb0 udp {server public IP}:51820 <- {client public IP}:19988 MULTIPLE:MULTIPLE pfctl -sr -vvvv @68(1613966106) pass in quick on igb0 reply-to (igb0 {server WAN gateway IP}) inet proto udp from any to {server public IP} port = 51820 keep state label "USER_RULE: pfSense WireGuard Server" [ Evaluations: 21757 Packets: 6025 Bytes: 4078532 States: 1 ] [ Inserted: pid 28901 State Creations: 1 ]
Right after disabling the rule and deleting the open state on the server, it opens a new state but this time in reverse and naturally no rules are triggered because it becomes an outbound call:
pfctl -ss -vvvv | grep 51820 igb0 udp {server public IP}:51820 -> {client public IP}:19988 MULTIPLE:MULTIPLE pfctl -sr -vvvv @68(1613966106) pass in quick on igb0 reply-to (igb0 {server WAN gateway IP}) inet proto udp from any to {server public IP} port = 51820 keep state label "USER_RULE: pfSense WireGuard Server" [ Evaluations: 24229 Packets: 6933 Bytes: 4638020 States: 0 ] [ Inserted: pid 28901 State Creations: 1 ]
So how do both sides have outbound connections and still achieve a tunnel? I'm just curious here. The open state on the client side never changes direction. It's always client to server and the source and destination port is always 51820.
-
You have
client IP
twice in the second state. I assume that's a typo?Are you saying the state that opens without the WAN rule enabled shows it was created by the same rule number?
Steve
-
@stephenw10 said in (IPsec outbound NAT to interface address) Reply traffic destination IP not being translated back to original source IP:
You have
client IP
twice in the second state. I assume that's a typo?Are you saying the state that opens without the WAN rule enabled shows it was created by the same rule number?
Steve
Edited, sorry about that.
No. After I disable the WAN rule and delete the open state on the server side, a new state in the server side is recreated but seems to be outbound. So the same rule number does not show it was used because naturally it is outbound. But then after a while (like how I checked again now), the open state on the server seems to be as expected again BUT the rule state details under the WAN tab does not reflect the correct "bytes".