I'm working on a new system two two Netgate 1541 appliances and need to do some iperf testing across the VPN to see which combinations of settings are ideal for IPsec and OpenVPN.

I configure the IPSec Phase 2 as VTI connections, then assign those to an Interface and use that Interface in OSPF.

This is all working and I can PING the remote firewall or remote hosts across the VPN connection, but I cannot make TCP connections across; for example, I can't pull up the pfSense web interface on the remote firewall.

I also have OpenVPN tunnels on these firewalls and I place them at a lower metric in OSPF, then I can pass ping traffic as well as TCP traffic, just fine.

I've never run into this and I've been banging my head against it all day and could use some help or advice of something else to check?

The reason I really need to get IPSec working, as opposed to just using OpenVPN is that these two firewalls will have 10 gigabit fiber WAN connections to each and I suspect that OpenVPN isn't going to give me as high of performance as IPSec should.

I am getting about 900 Mbps across OpenVPN with my two test laptops connected with gigabit interfaces. I have another machine on the way with 10 gig interfaces to test higher-than-1gbps connections.

I should also mention that with these IPSec tunnels, if I disable the VTI Phase 2 and create a standard tunnel Phase 2, I can pass traffic perfectly (ICMP and TCP). It seems to be related to IPSec and OSPF (via FRR) that things are falling down for some reason.