Okay, after a long troubleshooting session. Problem is solved. For reference, I needed to recreate the firewall rules on both sites for ipsec, reload filters and reset states. I suspect it was a weird gimmick messing up the filters.

@viragomann @tinfoilmatt Based on your feedback I read up on asymmetric routing and ended up skipping pfSense for this setup altogether I always try to standardize on products but here it just lead to a level of network knowledge I do not fully understand. I configured a basic IKEv1 tunnel in the edge gateway, added necessary firewall rules and everything worked as intended. A sidenote is that IKEv2 did not work well in VMware NSX-V with the P2's being disconnected after 3600 seconds and not being able to reconnect without tearing down the P1 manually as well. [image: 1745913956839-ef365c84-1553-4c71-8c60-fbb1827af9a7-image.png] Thanks everyone for your input!

@stan-fergusonsmith Does your application really use a static source port?? That's very unusual. Most application use a random source port. So you probably have to set the source port to 'any' in the port forwarding and firewall rule.

I know this is an old topic but I got here from searching the error message. In our case the person adding the VPN didn't use the .ps1 file from pfSense to do it, and Windows 11 24H2 still apparently uses weak algos by default.

@michmoor I would love to give your reply a thumbs up, but apparently you have to have 5 something, and no clue on how to get that. Anyway, I'm going to look at wireguard; however, i upped my p1 timeout, rekey, and expiry times to 7 days then 10 under for rekey and 2 under for expiry and i've gone ahead and upped the p2 to 1 day and rekey at 5 minutes under. That was at 13:44 and we are now at 16:17 and we haven't had a drop yet.

@tompark said in IPSec to USG behind NAT: It looks to be as if the connection between the USG and the PFSense I am connecting too, timesout. Is there a way that I can easierly check the traffic is being forwarded by the PFSense firewall? You can check pftop to see the state table. Doc is here

@diegoavelinogomes You need to add an IPSec phase 2 for the OpenVPN tunnel network: A: local: A clients subnet remote: OpenVPN tunnel subnet B: local: OpenVPN tunnel subnet remote: A clients subnet In the OpenVPN access server settings you have to add the "A clients subnet" to the "Local networks". However, OpenVPN clients might block access from the remote site by their own firewalls by default. You will have to configure their firewalls accordingly. Possibly masquerading the source IP with the OpenVPN interface IP can circumvent firewall restrictions.

@viragomann firstly thanks for the reply, forgive me for the dense paragraph. I did setup a P2 per subnet. In any case I have in the mean time found a solution (which happens to coincide with what you suggested): Firstly I switched from BiNAT to simple NAT for each target subnet and then I NATted on a single address per each subnet like this: Local Subnet Remote Subnet NATed IP 172.20.48.x/24 10.10.x.x/16 10.10.12.201/24 172.20.48.x/24 10.11.x.x/16 10.10.12.202/24 172.20.2.x/24 10.10.x.x/16 10.10.12.203/24 172.20.2.x/24 10.11.x.x/16 10.10.12.204/24 This way the packets from both subnets are routed and NATed through the IPSec tunnel correctly. However, even though I also added a local static route as per https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/access-firewall-over-ipsec.html to be able to be pinged by the other end for monitoring, the remote pings do not return. This may be a limitation of the above NAT setup in the IPSec. At the moment this isn't a major issue but I should find a solution for the remote monitoring setup. Notes: The BiNAT solution works only for a primary Network (the first P2 Network encountered in the list), any subsequent different specified P2 network subnet is ignored - packets reach the firewall LAN interface but are not routed to the IPSec interface. I have not attempted a routed VTI IPSec approach which I suppose will work fine also, but requires a more elaborate configuration to setup the relative IPSec enabled interfaces, firewall rules and NAT/routing. I may take a shot at this in the future as with the above setup it's hard if not impossible to have a fallback/secondary IPSec gateway configured if the main one dies - this would require the VTI routed approach if I'm not mistaken. Some additional references to IPSec / NAT issues and workarounds which seem to be relevant: https://forum.netgate.com/topic/155132/problems-with-routed-ipsec-vti/6 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248474 Cheers

Update 2: Fixed it. It is not so clear that vti interfaces ip addresses have to be routed also. To make it simple: use single /24 subnet for all vti tunnels and add this subnet to "Static routes" at every site

@TheStormsOfFury Okay - that means we should totally disregard the IPsec numbers and tests for now. Since we can’t reach Gbit in the public/public test, then we won’t be able to through IPsec either. When doing 4 streams? Are the numbers more or less the same in both directions when the client is in site 1 and server in site 2 (test from client with and without -R) Are the numbers more less the same if you run the server on site 1 and test both directions with the client on site 2? Is the packetloss still showing up in the “ramp up” intial phase of all 4 session tests above? (this causes TCP to not scale further) What it the PING latency between the sites? Anyways - like I said, these things are sometimes impossible to diagnose properly because there is so many factors in play. I’m guessing you probably have to do a packetcapture of the first 2 secs of an iPerf run to see if it’s just packetloss that causes TCP not to scale further, or if it is out-of-order delivery. I have started seeing the latter become a more and more prevalent issue on the Internet. I don’t know if ISPs are starting to use some new multipath loadbalancing mechanisms that are really suceptible to in-path latency causing a lot of out-of-order issues end-to-end. The point here is - your speedtests to internet does not pass the routers and paths that the test between your sites does. So it’s likely something in that path that is causing the problem - be it latency, out-of-order or packetdrops. Generally ISP’s focus and calibrate bandwidth and latency on vertical traffic (that is down/upload to the wider internet). Horisontal traffic between end user sites is of very little to no priority at all depending on the subscriptions used in both ends.

@mcury Copy. Will either write it up on the plane today or when I get to my destination.

@patient0 Fixed, I forgot to add the remote ID :). Thanks for help.

@Matt_Sharpe Two other options that come to mind Alert on syslog. So if using a syslog server such as Graylog, you can have it alert you by sending an email when it sees entries for the tunnel going down. Modify the scripts here - although meant for Zabbix i can see it easily being customizable for any monitoring tool