Poor performance over IPsec but not Internet

[michmoor]

Looking at the subject the inclination is MTU is in play but Im not entirely sure...

Background: I have a pfsense connected to a 500/500 Internet line. I have an OPNsense at another location connected to a 200/35 line. Each site is connected by IPsec. I have a speedtest docker container at the opnsense location. Results are below. I am hitting the container from the Internet through HA proxy so its going through CloudFlare and eventually down to my circuit so there is no issue there.
I have tried toggling the MTU on both sides all the way down to 1300 with no change. I am able to send ping packets with the DF bit set up to 1372 bytes so don't see MTU in play here.

Internet:

IPsec

[stephenw10]

I would try setting MSS and going to 1200.

What hardware are you running?

Steve

[michmoor]

@stephenw10
Both Protectli units for now.. FW6B and FW2B

Setting 1200 on both sides produces same results

[stephenw10]

Not sure what CPUs that is but 20Mbps is low for anything.

What encryption are you using? If you have everything turned up to 11 and an algorithm that cannot be offloaded ot will slow down significantly.

What does the per-core CPU usage look like during a test?
Try running: top -aSH

Steve

[Cool_Corona]

For best performance, you are told to stop offloading anything.

For IPsec its important to do so in regards to performance.

I get shitty VPN speeds on anything. No more than 200mbps and I cant even watch videos on a share that located in a DC on a 10Gbps connection.

[stephenw10]

@cool_corona said in Poor performance over IPsec but not Internet:

No more than 200mbps

Assuming you mean Megabits per second there you are probably limited by latency.

However 200Mbps is enough for like 8 4K streams so you're probably hitting something else.

Steve

[michmoor]

@stephenw10

Output during the test is below.

CPU Type:

[22.01-RELEASE][admin@GA-FW1]/root: top -aSH
last pid: 86070;  load averages:  2.32,  1.13,  0.60                                                  up 1+07:33:59  14:31:21
410 threads:   5 running, 385 sleeping, 1 zombie, 19 waiting
CPU: 10.2% user,  0.0% nice,  7.0% system,  0.3% interrupt, 82.5% idle
Mem: 10G Active, 4096B Inact, 3297M Laundry, 1546M Wired, 810M Buf, 60M Free
Swap: 3656M Total, 3646M Used, 11M Free, 99% Inuse, 96K In, 8192B Out

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
   11 root        155 ki31     0B    64K CPU1     1  30.5H  90.93% [idle{idle: cpu1}]
   11 root        155 ki31     0B    64K RUN      2  30.5H  82.33% [idle{idle: cpu2}]
   11 root        155 ki31     0B    64K CPU3     3  30.6H  80.58% [idle{idle: cpu3}]
   11 root        155 ki31     0B    64K CPU0     0  29.7H  76.42% [idle{idle: cpu0}]
28222 root         49    0    16G    13G pfault   2   5:28  39.41% /usr/local/zenarmor/zenarmor-agent/bin/zenarmor-agent -s{z
   23 root        -16    -     0B    48K laundp   0   0:16  13.67% [pagedaemon{laundry: dom0}]
50657 root         20  -20  1968M    46M select   1   5:20   3.95% eastpect: Eastpect Instance 0 (eastpect){Eastpect Main Eve
    0 root        -20    -     0B  1280K -        0   0:16   2.10% [kernel{crypto_0}]
    0 root        -20    -     0B  1280K -        2   0:16   1.94% [kernel{crypto_2}]
    0 root        -20    -     0B  1280K -        3   0:16   1.90% [kernel{crypto_1}]
    0 root        -20    -     0B  1280K -        3   0:16   1.52% [kernel{crypto_3}]
50867 root         20  -20  1633M    41M pfault   1   1:47   0.98% eastpect: Eastpect Instance 3 (eastpect){Eastpect Main Eve
    6 root        -16    -     0B    16K crypto   1   0:02   0.71% [crypto returns 3]

Encryption used

[Cool_Corona]

@stephenw10 On a windows network share. Latency is 18ms.

I have tried everything.....

[stephenw10]

@michmoor said in Poor performance over IPsec but not Internet:

/usr/local/zenarmor/zenarmor-agent/bin/zenarmor-agent

What is that? Can you disable it as a test?

If your CPUs support AES-NI you should be using AES-GCM at P2. That's where it counts for tunnel bandwidth.

Steve

[michmoor]

@stephenw10
/usr/local/zenarmor/zenarmor-agent/bin/zenarmor-agent

• is a package i installed a few days ago but has nothing to do with the throughput issue as that has been on-going for a few months.

AES-GCM with no hash algo is now in play but unfortunately, the throughput is the same.

Its completely possible the fault is on the remote side firewall as well but unfortunately, I see no evidence of that at this time. I have another VPN(ZeroTier) from the OPNsense to a production VPC instance. For testing I set up iperf3 between the two and results are as expected.

~$ iperf3 -c 10.147.20.83 -P 10 -R
[SUM] 0.00-10.02 sec 152 MBytes 127 Mbits/sec 1434 sender
[SUM] 0.00-10.00 sec 142 MBytes 119 Mbits/sec receiver

So it seems the throughput issue is only between the two firewalls only. Each firewall has VPNs to other peers where throughput is very good - almost maxing out the local circuit bandwidth.

[stephenw10]

Re-reading; in that first test result are you actually testing between the two sites outside the tunnel?

I had though you were but now I'm not sure. In which case you could simply have something throttling the traffic in the route somehow.
Try running an iperf test between the sites directly.

It could be restricting ESP traffic only. That unfortunately common. If neither side is behind NAT try forcing NAT-T at one end to encapsulate the traffic in UDP.

Steve

[michmoor]

@stephenw10
The very first test,the one that started this post, is showing 2x tests. The first test is showing a speedtest from the Internet to the opnsense firewall. This is to illustrate that the site can achieve good throughput. The second test within the same opening thread is between both firewalls showing poor throughput.

The last test i performed and posted last night is to show that each firewall has VPNs branching out to other vpn destinations other then between themselves. That test shows that when using a ZeroTier VPN (opnsense running zerotier) to a VPC i get great throughput.
On the other end, on the PFsense, I have several Wireguard tunnels to multiple VPCs and that has great throughput as well.
The problem is only between the two firewall systems only.

You did mention that its possible that the ISP, one or both of them, could be throttling ESP traffic. Seeing how the other VPN technologies do not use ESP in the header that is a very interesting theory. Only way to satisfy that would be to run wireguard between the sites. I wanted to exhaust every avenue before i put effort to change the design.

edit: how do i force NAT-T ?

[stephenw10]

OK, so the next thing I would do here is test between the sites directly but outside the VPN.

There is no point digging into the VPN config if something in the route that traffic is taking is throttling it. That may not be at either ISP directly.

In the Phase 1 advanced options settings set 'NAT Traversal' to Force to test ESP specific throttling. Only one end needs to set that.

Steve

[michmoor]

@stephenw10 You were spot on with the change to NAT-T. Encapsulating everything in UDP allowed throughput to shoot up to expected levels.
Ive never ran into this issue before so I can only assume home residential broadband providers do this while the more expensive plans such as DIA Internet lines have no such limitation.
Crazy...

[stephenw10]

@michmoor said in Poor performance over IPsec but not Internet:

I can only assume home residential broadband providers do this

I wish I could say it's limited to that. You might find it's not at either ISP directly but in some device that happens to be in the route between them.
It's not that uncommon to find routers that don't pass ESP at all or, worse, only pass it in one direction! The tunnel establishes using udp/500 traffic but cannot pass data at Phase2. Those are always fun.

Steve