Incredibly Slow transmission rates over Site-to-Site IPSEC VPN
Trying to put in place a solution to allow more of our staff to work at home. Rather than simple mobile vpn, 'The Bosses' want permanent connections up that allow file share access/printing/backups. The workstations (that will be remote) are a mixture of Win10, MacOS, and Linux (mostly Ubuntu).
For the one off's and actual mobile workers, I've got IkeV2 Chap mobile vpn setup and working very well for Win10 and Linux users (mac users, however, no, but that's for another day) and the transfer speeds are in line with the remote workers access speeds (reasonable, with no complaints). The problem lies now with the site to site IPSEC configuration -- those speeds are miserable.
A Bit of Detail:
On the office side, we've a repurposed Dell Poweredge r220 (xeon E3-1220 v3 3.1 GHz - 4C4T, 8GB RAM) running pfSense 2.4.5-r-p1 on 2, 200MB dedicated fiber lines.
On the clients side, I've purchased a couple SG-1100's to test things out before I roll them to end users (with all sorts of different speeds and internet providers). I'm testing the client on a modest 10MB ADSL line. The ISP provider's router is a 'business' type which at least allows me to either port forward or put the SG-1100 in the DMZ, allowing direct access to the internet (this is what I have currently configured). I have been using the second incoming fiber line to host the Site to Site VPN (as if I use the primary, the mobile VPN stops working, something else to ask about later?)
Here's the current config I'm trying (and failing) with:
Office router details:
Gateways WAN1 WAN2
Gateway Groups WAN_LoadBalancer
Static Routes None
Key Exchange IKEv2
Interface Secondary WAN interface
Remote Gateway IP Address for my remote office (static)
Auth Method Mutual PSK
My Identifier - IP Address External IP address for 2nd WAN
Peer Identifier Peer IP address
Preshared Key Matches on both sides
Phase 1 Proposal AES128-GCM / 128 / SHA256 / DH14
NAT Traversal Auto
Enable Dead Peer Detection Active
Enable Max MSS - Active 1500
Auto Exclude LAN Address Active
Asynchronous Cryptography Active
Office Router Phase 2:
Mode Tunnel IP¨v4
Local Network LAN Subnet
NAT/BINAT Translation None
Remote Network : Network 10.0.20.0 / 24
Phase 2 Proposal Protocol ESP
Encryption Algorithms AES128-GCM / 128 / No Hash / PFS key grp 14
Remote office setup on the SG1100 is a copy of the above Phase1/Phase2 with the exception of Remote Gateway (being set to the external IP for my secondary WAN link) and the Remote Network being set to Network:10.0.1.0/24.
I've tried (originally) the encryption settings in the pfSense Guide (https://docs.netgate.com/pfsense/en/latest/recipes/ipsec-s2s-psk.html)
phase 1: AES / 256 / SHA256 / DH 2
phase 2 AES256-GCM / 128 / No Hash / PFS Off
The current settings I've tried after going over the article at https://medium.com/@dEad0r/measuring-performance-of-site-to-site-vpn-tunnels-between-pfsense-vms-b484ba425aff
I've tried MSS settings 1400 and deactivated as well as turning off Asynchronous Cryptography.
Accessing Web pages on remote (office site) servers (such as pfSense admin interface for the office installation of pfSense and other web administration portals) takes upwards of 30 seconds to load, with some just timing out.
Access File shares (smb) on remote files servers succeeds in listing the files/folders, but any attempt to open or copy results in a time out.
Baseline from Remote office to ping.online.net (directly connected to internet):
------------------------------------------------------------ Client connecting to ping.online.net, TCP port 5001 TCP window size: 93.5 KByte (default) ------------------------------------------------------------ [ 3] local 10.0.0.94 port 43356 connected with 220.127.116.11 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.9 sec 1280 KBytes 118 KBytes/sec
Baseline when remote office is connected to main office via Mobile VPN (IKEv2 Chap) and testing performance to file server:
iperf3 -c <file-srv-ip> -f K Connecting to host <file-srv-ip>, port 5201 [ 4] local 10.0.3.1 port 59152 connected to <file-srv-ip> port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.01 sec 256 KBytes 255 KBytes/sec [ 4] 1.01-2.00 sec 0.00 Bytes 0.00 KBytes/sec [ 4] 2.00-3.00 sec 128 KBytes 128 KBytes/sec [ 4] 3.00-4.01 sec 128 KBytes 128 KBytes/sec [ 4] 4.01-5.00 sec 0.00 Bytes 0.00 KBytes/sec [ 4] 5.00-6.00 sec 128 KBytes 128 KBytes/sec [ 4] 6.00-7.00 sec 0.00 Bytes 0.00 KBytes/sec [ 4] 7.00-8.02 sec 128 KBytes 126 KBytes/sec [ 4] 8.02-9.01 sec 0.00 Bytes 0.00 KBytes/sec [ 4] 9.01-10.00 sec 128 KBytes 129 KBytes/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 896 KBytes 89.6 KBytes/sec sender [ 4] 0.00-10.00 sec 693 KBytes 69.3 KBytes/sec receiver iperf Done.
Results from remote office when connected to SG-1100 with Site-to-Site connection to Main Office Active and testing performance to the same file server:
iperf3 -c <file-srv-ip> -f K Connecting to host <file-srv-ip>, port 5201 [ 5] local 10.0.20.54 port 32934 connected to <file-srv-ip> port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 164 KBytes 164 KBytes/sec 0 17.0 KBytes [ 5] 1.00-2.00 sec 124 KBytes 124 KBytes/sec 0 22.6 KBytes [ 5] 2.00-3.00 sec 62.2 KBytes 62.2 KBytes/sec 1 24.0 KBytes [ 5] 3.00-4.00 sec 124 KBytes 124 KBytes/sec 4 22.6 KBytes [ 5] 4.00-5.00 sec 62.2 KBytes 62.2 KBytes/sec 9 11.3 KBytes [ 5] 5.00-6.00 sec 62.2 KBytes 62.2 KBytes/sec 0 15.6 KBytes [ 5] 6.00-7.00 sec 124 KBytes 125 KBytes/sec 1 14.1 KBytes [ 5] 7.00-8.00 sec 62.2 KBytes 62.2 KBytes/sec 0 17.0 KBytes [ 5] 8.00-9.00 sec 62.2 KBytes 62.2 KBytes/sec 3 15.6 KBytes [ 5] 9.00-10.00 sec 62.2 KBytes 62.2 KBytes/sec 3 15.6 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 911 KBytes 91.1 KBytes/sec 21 sender [ 5] 0.00-10.25 sec 829 KBytes 80.8 KBytes/sec receiver iperf Done.
I've posted the results from when I am connected via the Mobile VPN (where the file transfer speed and access to web portals is acceptable for my overall connection speed) to compare the iperf result to those where the speed is hugely diminished (when connected to the Site-to-Site) The iperf results are very similar.
Now, knowing, and reading that SMB suffers from high latency connections I've tried a scp of a small file (190kb) while connected via Mobile VPN and the site-to-site with those results below:
scp <USERNAME>@<file-srv>:/<PATH>/<REMOTE-FILE-NAME> ./test.pdf <USERNAME>@<file-srv>'s password: <REMOTE-FILE-NAME> 100% 190KB 277.5KB/s 00:00
scp <USERNAME>@<file-srv>:/<PATH>/<REMOTE-FILE-NAME> ./test.pdf <USERNAME>@<file-srv>'s password: <REMOTE-FILE-NAME> 100% 190KB 5.0KB/s 00:38
Obviously a huge, and unacceptable difference, and this is using scp. I have not yet tried wget, but I expect similar, disappointing results.
In Conclusion :
I'm definitely using this experience to build my experience around this topic and have done countless searches (probably using the wrong terms) and have seen other people with similar issues and have seen the conclusion that IPSEC site to site just does not work with SMB, with no real constructive takeaways (and I've shown above, it's affecting more than just SMB, so I'm thinking I'm missing something) so I'm turning to you kind folks to help me grok where I am going wrong (and hopefully before I've torn out all my hair and resign myself to self doubt!).
Thanks for any feedback/advice you may have, even if it is pointing to an article I may have missed or some settings I may or may have not yet tried!!
EDIT: Cross-posted at /rPFSENSE as well
Why Max MSS 1500? The default is 1400 and that might be too big. I'd work my way up from 1300. I got also slow speeds from a VTI connection until I set not more than 1390 MTU/MSS in the interface settings (I guess the VTI interface settings are equvialent to the Max MSS advanced setting in tunnel mode).
I have a similar problem, if not quite as serious.
One side has a 300/300 Mbit/s connection, the other side a 1000/50 Mbit/s connection.
However, I am unable to transmit more than 35 Mbit/s using iperf3.
I set the MSS value to 1420 on the WAN connection and the clamping to 1380 in the IPSec settings.
However, this does not change anything.
If I set the MSS clamping to 1400 in the IPSec settings, then suddenly only about 3 Mbit/s go through the tunnel, so I think I'm on the right path.
Does anyone else have a tip?
Thank you all.
I was eventually able to solve it by backing off the encryption levels; seems the hardware in the SG-1100 is not able to handle the settings in the guide (Netgear, you may want to address this). I haven't had the chnace to dial in the exact settings I am happy with yet (performance vs security) but once I do, I'll be sure to post here.
I was eventually able to solve it by backing off the encryption levels
thanks for the tip. Unfortunately no improvement for me.