Poor performance over IPsec but not Internet
-
@stephenw10
Both Protectli units for now.. FW6B and FW2BSetting 1200 on both sides produces same results
-
Not sure what CPUs that is but 20Mbps is low for anything.
What encryption are you using? If you have everything turned up to 11 and an algorithm that cannot be offloaded ot will slow down significantly.
What does the per-core CPU usage look like during a test?
Try running:top -aSHSteve
-
For best performance, you are told to stop offloading anything.
For IPsec its important to do so in regards to performance.
I get shitty VPN speeds on anything. No more than 200mbps and I cant even watch videos on a share that located in a DC on a 10Gbps connection.
-
@cool_corona said in Poor performance over IPsec but not Internet:
No more than 200mbps
Assuming you mean Megabits per second there you are probably limited by latency.
However 200Mbps is enough for like 8 4K streams so you're probably hitting something else.
Steve
-
Output during the test is below.
CPU Type:
[22.01-RELEASE][admin@GA-FW1]/root: top -aSH last pid: 86070; load averages: 2.32, 1.13, 0.60 up 1+07:33:59 14:31:21 410 threads: 5 running, 385 sleeping, 1 zombie, 19 waiting CPU: 10.2% user, 0.0% nice, 7.0% system, 0.3% interrupt, 82.5% idle Mem: 10G Active, 4096B Inact, 3297M Laundry, 1546M Wired, 810M Buf, 60M Free Swap: 3656M Total, 3646M Used, 11M Free, 99% Inuse, 96K In, 8192B Out PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0B 64K CPU1 1 30.5H 90.93% [idle{idle: cpu1}] 11 root 155 ki31 0B 64K RUN 2 30.5H 82.33% [idle{idle: cpu2}] 11 root 155 ki31 0B 64K CPU3 3 30.6H 80.58% [idle{idle: cpu3}] 11 root 155 ki31 0B 64K CPU0 0 29.7H 76.42% [idle{idle: cpu0}] 28222 root 49 0 16G 13G pfault 2 5:28 39.41% /usr/local/zenarmor/zenarmor-agent/bin/zenarmor-agent -s{z 23 root -16 - 0B 48K laundp 0 0:16 13.67% [pagedaemon{laundry: dom0}] 50657 root 20 -20 1968M 46M select 1 5:20 3.95% eastpect: Eastpect Instance 0 (eastpect){Eastpect Main Eve 0 root -20 - 0B 1280K - 0 0:16 2.10% [kernel{crypto_0}] 0 root -20 - 0B 1280K - 2 0:16 1.94% [kernel{crypto_2}] 0 root -20 - 0B 1280K - 3 0:16 1.90% [kernel{crypto_1}] 0 root -20 - 0B 1280K - 3 0:16 1.52% [kernel{crypto_3}] 50867 root 20 -20 1633M 41M pfault 1 1:47 0.98% eastpect: Eastpect Instance 3 (eastpect){Eastpect Main Eve 6 root -16 - 0B 16K crypto 1 0:02 0.71% [crypto returns 3]Encryption used
-
@stephenw10 On a windows network share. Latency is 18ms.
I have tried everything.....
-
@michmoor said in Poor performance over IPsec but not Internet:
/usr/local/zenarmor/zenarmor-agent/bin/zenarmor-agent
What is that? Can you disable it as a test?
If your CPUs support AES-NI you should be using AES-GCM at P2. That's where it counts for tunnel bandwidth.
Steve
-
@stephenw10
/usr/local/zenarmor/zenarmor-agent/bin/zenarmor-agent- is a package i installed a few days ago but has nothing to do with the throughput issue as that has been on-going for a few months.
AES-GCM with no hash algo is now in play but unfortunately, the throughput is the same.
Its completely possible the fault is on the remote side firewall as well but unfortunately, I see no evidence of that at this time. I have another VPN(ZeroTier) from the OPNsense to a production VPC instance. For testing I set up iperf3 between the two and results are as expected.
~$ iperf3 -c 10.147.20.83 -P 10 -R
[SUM] 0.00-10.02 sec 152 MBytes 127 Mbits/sec 1434 sender
[SUM] 0.00-10.00 sec 142 MBytes 119 Mbits/sec receiverSo it seems the throughput issue is only between the two firewalls only. Each firewall has VPNs to other peers where throughput is very good - almost maxing out the local circuit bandwidth.
-
Re-reading; in that first test result are you actually testing between the two sites outside the tunnel?
I had though you were but now I'm not sure. In which case you could simply have something throttling the traffic in the route somehow.
Try running an iperf test between the sites directly.It could be restricting ESP traffic only. That unfortunately common. If neither side is behind NAT try forcing NAT-T at one end to encapsulate the traffic in UDP.
Steve
-
@stephenw10
The very first test,the one that started this post, is showing 2x tests. The first test is showing a speedtest from the Internet to the opnsense firewall. This is to illustrate that the site can achieve good throughput. The second test within the same opening thread is between both firewalls showing poor throughput.The last test i performed and posted last night is to show that each firewall has VPNs branching out to other vpn destinations other then between themselves. That test shows that when using a ZeroTier VPN (opnsense running zerotier) to a VPC i get great throughput.
On the other end, on the PFsense, I have several Wireguard tunnels to multiple VPCs and that has great throughput as well.
The problem is only between the two firewall systems only.You did mention that its possible that the ISP, one or both of them, could be throttling ESP traffic. Seeing how the other VPN technologies do not use ESP in the header that is a very interesting theory. Only way to satisfy that would be to run wireguard between the sites. I wanted to exhaust every avenue before i put effort to change the design.
edit: how do i force NAT-T ?
-
OK, so the next thing I would do here is test between the sites directly but outside the VPN.
There is no point digging into the VPN config if something in the route that traffic is taking is throttling it. That may not be at either ISP directly.
In the Phase 1 advanced options settings set 'NAT Traversal' to Force to test ESP specific throttling. Only one end needs to set that.
Steve
-
@stephenw10 You were spot on with the change to NAT-T. Encapsulating everything in UDP allowed throughput to shoot up to expected levels.
Ive never ran into this issue before so I can only assume home residential broadband providers do this while the more expensive plans such as DIA Internet lines have no such limitation.
Crazy...
-
@michmoor said in Poor performance over IPsec but not Internet:
I can only assume home residential broadband providers do this
I wish I could say it's limited to that. You might find it's not at either ISP directly but in some device that happens to be in the route between them.
It's not that uncommon to find routers that don't pass ESP at all or, worse, only pass it in one direction! The tunnel establishes using udp/500 traffic but cannot pass data at Phase2. Those are always fun.
Steve
