@tompark said in IPSec to USG behind NAT: It looks to be as if the connection between the USG and the PFSense I am connecting too, timesout. Is there a way that I can easierly check the traffic is being forwarded by the PFSense firewall? You can check pftop to see the state table. Doc is here

@diegoavelinogomes You need to add an IPSec phase 2 for the OpenVPN tunnel network: A: local: A clients subnet remote: OpenVPN tunnel subnet B: local: OpenVPN tunnel subnet remote: A clients subnet In the OpenVPN access server settings you have to add the "A clients subnet" to the "Local networks". However, OpenVPN clients might block access from the remote site by their own firewalls by default. You will have to configure their firewalls accordingly. Possibly masquerading the source IP with the OpenVPN interface IP can circumvent firewall restrictions.

@viragomann firstly thanks for the reply, forgive me for the dense paragraph. I did setup a P2 per subnet. In any case I have in the mean time found a solution (which happens to coincide with what you suggested): Firstly I switched from BiNAT to simple NAT for each target subnet and then I NATted on a single address per each subnet like this: Local Subnet Remote Subnet NATed IP 172.20.48.x/24 10.10.x.x/16 10.10.12.201/24 172.20.48.x/24 10.11.x.x/16 10.10.12.202/24 172.20.2.x/24 10.10.x.x/16 10.10.12.203/24 172.20.2.x/24 10.11.x.x/16 10.10.12.204/24 This way the packets from both subnets are routed and NATed through the IPSec tunnel correctly. However, even though I also added a local static route as per https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/access-firewall-over-ipsec.html to be able to be pinged by the other end for monitoring, the remote pings do not return. This may be a limitation of the above NAT setup in the IPSec. At the moment this isn't a major issue but I should find a solution for the remote monitoring setup. Notes: The BiNAT solution works only for a primary Network (the first P2 Network encountered in the list), any subsequent different specified P2 network subnet is ignored - packets reach the firewall LAN interface but are not routed to the IPSec interface. I have not attempted a routed VTI IPSec approach which I suppose will work fine also, but requires a more elaborate configuration to setup the relative IPSec enabled interfaces, firewall rules and NAT/routing. I may take a shot at this in the future as with the above setup it's hard if not impossible to have a fallback/secondary IPSec gateway configured if the main one dies - this would require the VTI routed approach if I'm not mistaken. Some additional references to IPSec / NAT issues and workarounds which seem to be relevant: https://forum.netgate.com/topic/155132/problems-with-routed-ipsec-vti/6 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248474 Cheers

Update 2: Fixed it. It is not so clear that vti interfaces ip addresses have to be routed also. To make it simple: use single /24 subnet for all vti tunnels and add this subnet to "Static routes" at every site

@TheStormsOfFury Okay - that means we should totally disregard the IPsec numbers and tests for now. Since we can’t reach Gbit in the public/public test, then we won’t be able to through IPsec either. When doing 4 streams? Are the numbers more or less the same in both directions when the client is in site 1 and server in site 2 (test from client with and without -R) Are the numbers more less the same if you run the server on site 1 and test both directions with the client on site 2? Is the packetloss still showing up in the “ramp up” intial phase of all 4 session tests above? (this causes TCP to not scale further) What it the PING latency between the sites? Anyways - like I said, these things are sometimes impossible to diagnose properly because there is so many factors in play. I’m guessing you probably have to do a packetcapture of the first 2 secs of an iPerf run to see if it’s just packetloss that causes TCP not to scale further, or if it is out-of-order delivery. I have started seeing the latter become a more and more prevalent issue on the Internet. I don’t know if ISPs are starting to use some new multipath loadbalancing mechanisms that are really suceptible to in-path latency causing a lot of out-of-order issues end-to-end. The point here is - your speedtests to internet does not pass the routers and paths that the test between your sites does. So it’s likely something in that path that is causing the problem - be it latency, out-of-order or packetdrops. Generally ISP’s focus and calibrate bandwidth and latency on vertical traffic (that is down/upload to the wider internet). Horisontal traffic between end user sites is of very little to no priority at all depending on the subscriptions used in both ends.

@mcury Copy. Will either write it up on the plane today or when I get to my destination.

@patient0 Fixed, I forgot to add the remote ID :). Thanks for help.

@Matt_Sharpe Two other options that come to mind Alert on syslog. So if using a syslog server such as Graylog, you can have it alert you by sending an email when it sees entries for the tunnel going down. Modify the scripts here - although meant for Zabbix i can see it easily being customizable for any monitoring tool

Seems like a perfectly good use case for Tailscale

@viragomann Okay, I understand! Thanks for your help! In two weeks the connection will be implemented, then I'll get back to you on this.

@plep Yeah, it can be a bit confusing. PfSense by default attempts to “combine” the subnets in P2s into one. That causes issues just like described where your other P2’s are “extended” from only the remote subnet to 0.0.0.0/0. To avoid this you need to tick the “Split Connections” box on your P1 for the tunnel then it will create several independant P2’s like the other end.

@NdubYu I appreciate this is an old topic, but i have the same problem Were you able to resolve ? What was the issue ? Thank you

Just for information to everyone, the problem was solved by changing (on both sides, of course) from IKE v1 to IKE v2.