Incredibly Slow transmission rates over Site-to-Site IPSEC VPN

[Maelstrom 0]

The Brief:

Trying to put in place a solution to allow more of our staff to work at home. Rather than simple mobile vpn, 'The Bosses' want permanent connections up that allow file share access/printing/backups. The workstations (that will be remote) are a mixture of Win10, MacOS, and Linux (mostly Ubuntu).

For the one off's and actual mobile workers, I've got IkeV2 Chap mobile vpn setup and working very well for Win10 and Linux users (mac users, however, no, but that's for another day) and the transfer speeds are in line with the remote workers access speeds (reasonable, with no complaints). The problem lies now with the site to site IPSEC configuration -- those speeds are miserable.

A Bit of Detail:

On the office side, we've a repurposed Dell Poweredge r220 (xeon E3-1220 v3 3.1 GHz - 4C4T, 8GB RAM) running pfSense 2.4.5-r-p1 on 2, 200MB dedicated fiber lines.

On the clients side, I've purchased a couple SG-1100's to test things out before I roll them to end users (with all sorts of different speeds and internet providers). I'm testing the client on a modest 10MB ADSL line. The ISP provider's router is a 'business' type which at least allows me to either port forward or put the SG-1100 in the DMZ, allowing direct access to the internet (this is what I have currently configured). I have been using the second incoming fiber line to host the Site to Site VPN (as if I use the primary, the mobile VPN stops working, something else to ask about later?)

Here's the current config I'm trying (and failing) with:

Office router details:

Interfaces WAN1/WAN2/LAN
Gateways WAN1 WAN2
Gateway Groups WAN_LoadBalancer
WAN_LinkFailover
Static Routes None

Phase 1:
Key Exchange IKEv2
IP IPv4
Interface Secondary WAN interface
Remote Gateway IP Address for my remote office (static)
Auth Method Mutual PSK
My Identifier - IP Address External IP address for 2nd WAN
Peer Identifier Peer IP address
Preshared Key Matches on both sides
Phase 1 Proposal AES128-GCM / 128 / SHA256 / DH14
NAT Traversal Auto
Mobike Enable
Enable Dead Peer Detection Active
Enable Max MSS - Active 1500
Auto Exclude LAN Address Active
Asynchronous Cryptography Active

Office Router Phase 2:
Mode Tunnel IP¨v4
Local Network LAN Subnet
NAT/BINAT Translation None
Remote Network : Network 10.0.20.0 / 24
Phase 2 Proposal Protocol ESP
Encryption Algorithms AES128-GCM / 128 / No Hash / PFS key grp 14

Remote office setup on the SG1100 is a copy of the above Phase1/Phase2 with the exception of Remote Gateway (being set to the external IP for my secondary WAN link) and the Remote Network being set to Network:10.0.1.0/24.

I've tried (originally) the encryption settings in the pfSense Guide (https://docs.netgate.com/pfsense/en/latest/recipes/ipsec-s2s-psk.html)

phase 1: AES / 256 / SHA256 / DH 2

phase 2 AES256-GCM / 128 / No Hash / PFS Off

The current settings I've tried after going over the article at https://medium.com/@dEad0r/measuring-performance-of-site-to-site-vpn-tunnels-between-pfsense-vms-b484ba425aff

I've tried MSS settings 1400 and deactivated as well as turning off Asynchronous Cryptography.

The Symptoms:

Accessing Web pages on remote (office site) servers (such as pfSense admin interface for the office installation of pfSense and other web administration portals) takes upwards of 30 seconds to load, with some just timing out.

Access File shares (smb) on remote files servers succeeds in listing the files/folders, but any attempt to open or copy results in a time out.

Iperf results:

Baseline from Remote office to ping.online.net (directly connected to internet):

------------------------------------------------------------
Client connecting to ping.online.net, TCP port 5001
TCP window size: 93.5 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.94 port 43356 connected with 62.210.18.40 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.9 sec  1280 KBytes   118 KBytes/sec

Baseline when remote office is connected to main office via Mobile VPN (IKEv2 Chap) and testing performance to file server:

iperf3 -c <file-srv-ip> -f K
Connecting to host <file-srv-ip>, port 5201
[  4] local 10.0.3.1 port 59152 connected to <file-srv-ip> port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.01   sec   256 KBytes   255 KBytes/sec
[  4]   1.01-2.00   sec  0.00 Bytes  0.00 KBytes/sec
[  4]   2.00-3.00   sec   128 KBytes   128 KBytes/sec
[  4]   3.00-4.01   sec   128 KBytes   128 KBytes/sec
[  4]   4.01-5.00   sec  0.00 Bytes  0.00 KBytes/sec
[  4]   5.00-6.00   sec   128 KBytes   128 KBytes/sec
[  4]   6.00-7.00   sec  0.00 Bytes  0.00 KBytes/sec
[  4]   7.00-8.02   sec   128 KBytes   126 KBytes/sec
[  4]   8.02-9.01   sec  0.00 Bytes  0.00 KBytes/sec
[  4]   9.01-10.00  sec   128 KBytes   129 KBytes/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   896 KBytes  89.6 KBytes/sec                  sender
[  4]   0.00-10.00  sec   693 KBytes  69.3 KBytes/sec                  receiver

iperf Done.

Results from remote office when connected to SG-1100 with Site-to-Site connection to Main Office Active and testing performance to the same file server:

iperf3 -c <file-srv-ip> -f K
Connecting to host <file-srv-ip>, port 5201
[  5] local 10.0.20.54 port 32934 connected to <file-srv-ip> port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   164 KBytes   164 KBytes/sec    0   17.0 KBytes       
[  5]   1.00-2.00   sec   124 KBytes   124 KBytes/sec    0   22.6 KBytes       
[  5]   2.00-3.00   sec  62.2 KBytes  62.2 KBytes/sec    1   24.0 KBytes       
[  5]   3.00-4.00   sec   124 KBytes   124 KBytes/sec    4   22.6 KBytes       
[  5]   4.00-5.00   sec  62.2 KBytes  62.2 KBytes/sec    9   11.3 KBytes       
[  5]   5.00-6.00   sec  62.2 KBytes  62.2 KBytes/sec    0   15.6 KBytes       
[  5]   6.00-7.00   sec   124 KBytes   125 KBytes/sec    1   14.1 KBytes       
[  5]   7.00-8.00   sec  62.2 KBytes  62.2 KBytes/sec    0   17.0 KBytes       
[  5]   8.00-9.00   sec  62.2 KBytes  62.2 KBytes/sec    3   15.6 KBytes       
[  5]   9.00-10.00  sec  62.2 KBytes  62.2 KBytes/sec    3   15.6 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   911 KBytes  91.1 KBytes/sec   21             sender
[  5]   0.00-10.25  sec   829 KBytes  80.8 KBytes/sec                  receiver

iperf Done.

I've posted the results from when I am connected via the Mobile VPN (where the file transfer speed and access to web portals is acceptable for my overall connection speed) to compare the iperf result to those where the speed is hugely diminished (when connected to the Site-to-Site) The iperf results are very similar.

Now, knowing, and reading that SMB suffers from high latency connections I've tried a scp of a small file (190kb) while connected via Mobile VPN and the site-to-site with those results below:

MobileVPN:

scp <USERNAME>@<file-srv>:/<PATH>/<REMOTE-FILE-NAME> ./test.pdf
<USERNAME>@<file-srv>'s password:
<REMOTE-FILE-NAME>          100%  190KB 277.5KB/s   00:00

Site-to-Site

scp <USERNAME>@<file-srv>:/<PATH>/<REMOTE-FILE-NAME> ./test.pdf
<USERNAME>@<file-srv>'s password:
<REMOTE-FILE-NAME>          100%  190KB   5.0KB/s   00:38

Obviously a huge, and unacceptable difference, and this is using scp. I have not yet tried wget, but I expect similar, disappointing results.

In Conclusion :

I'm definitely using this experience to build my experience around this topic and have done countless searches (probably using the wrong terms) and have seen other people with similar issues and have seen the conclusion that IPSEC site to site just does not work with SMB, with no real constructive takeaways (and I've shown above, it's affecting more than just SMB, so I'm thinking I'm missing something) so I'm turning to you kind folks to help me grok where I am going wrong (and hopefully before I've torn out all my hair and resign myself to self doubt!).

Thanks for any feedback/advice you may have, even if it is pointing to an article I may have missed or some settings I may or may have not yet tried!!

EDIT: Cross-posted at /rPFSENSE as well

[Morlock]

Why Max MSS 1500? The default is 1400 and that might be too big. I'd work my way up from 1300. I got also slow speeds from a VTI connection until I set not more than 1390 MTU/MSS in the interface settings (I guess the VTI interface settings are equvialent to the Max MSS advanced setting in tunnel mode).

[Maninblack]

I have a similar problem, if not quite as serious.

One side has a 300/300 Mbit/s connection, the other side a 1000/50 Mbit/s connection.

However, I am unable to transmit more than 35 Mbit/s using iperf3.

I set the MSS value to 1420 on the WAN connection and the clamping to 1380 in the IPSec settings.
However, this does not change anything.
If I set the MSS clamping to 1400 in the IPSec settings, then suddenly only about 3 Mbit/s go through the tunnel, so I think I'm on the right path.

Does anyone else have a tip?

Thank you all.

[Maelstrom 0]

I was eventually able to solve it by backing off the encryption levels; seems the hardware in the SG-1100 is not able to handle the settings in the guide (Netgear, you may want to address this). I haven't had the chnace to dial in the exact settings I am happy with yet (performance vs security) but once I do, I'll be sure to post here.

[Maninblack]

@maelstrom-0 said in Incredibly Slow transmission rates over Site-to-Site IPSEC VPN:

I was eventually able to solve it by backing off the encryption levels

thanks for the tip. Unfortunately no improvement for me.