OpenVPN client VPN slow, especially with higher latency
-
Currently, I'm using OpenVPN AS (commercial openvpn "appliance") installed on a virtual machine inside our network, pfsense as firewall is only doing the port forwarding to that machine. This works fine so far, users get quite good speeds, easily over 100Mbit/s (Depending on their line speed of course, the line pfsense is connected to is 1Gbit/s).
Now I wanted to replace above setup with an OpenPVN running on pfsense 2.4.4-RELEASE-p3, however, the OpenVPN connection with pfsense is much slower than with the OpenVPN AS server.
What I've tried so far:
-
All kinds of sysctls, buffers, clusters, sendspace, recvspace, you name it, I've tried it. Some settings make a small effect, but don't really make it fast
-
All kinds of hardware related things, like tcp segment offloading, tso, interrupt related msix things, etc. etc. The only thing that I've not tried yet is to disable checksum offloading
-
All kinds of openvpn options, sndbuf, rcvbuf, zero, large value, small value, medium value, tcp mss fix, mtu settings etc. etc. doesn't make a real difference
-
Tried both TCP and UDP modes
-
Tried different client versions, both on Linux and Windows
-
Different encryption methods, including no encryption
-
Different compression methods, also no compression
-
Setting tcp-nodelay improves things, but it's still a lot slower than with the OpenVPN AS
But, it's still slow. Especially with higher latencies between client and server it seems to get way worse, i.e. the OpenVPN AS setup is like 10x faster than the pfsense OpenVPN setup (yes, I'm aware of the effects of latency on TCP connections, window sizing, BDP, etc. ...).
The machine that pfsense runs on (Dell R610 with 3.3GHz Xeon) is actually faster than the virtual machine the openvpn AS runs on.
Maybe somebody has an idea on how to further debug this (?) I would really like to use OpenVPN on pfsense.
-
-
To give some numbers:
OpenVPN AS server, no additional delay configured:
# ping 172.16.40.52 PING 172.16.40.52 (172.16.40.52) 56(84) bytes of data. 64 bytes from 172.16.40.52: icmp_seq=1 ttl=63 time=21.1 ms 64 bytes from 172.16.40.52: icmp_seq=2 ttl=63 time=20.8 ms 64 bytes from 172.16.40.52: icmp_seq=3 ttl=63 time=19.8 ms ^C --- 172.16.40.52 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 19.825/20.612/21.185/0.587 ms# wget http://172.16.40.52/testfile.html --2019-10-15 16:17:17-- http://172.16.40.52/testfile.html Connecting to 172.16.40.52:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1048576000 (1000M) [text/html] Saving to: ‘testfile.html.13’ testfile.html.13 12%[========> ] 121,32M 11,1MB/s eta 83s ^COpenVPN AS server, with 50ms additional delay:
# tc qdisc add dev enxd481d720804b root netem delay 50ms # ping 172.16.40.52 PING 172.16.40.52 (172.16.40.52) 56(84) bytes of data. 64 bytes from 172.16.40.52: icmp_seq=1 ttl=63 time=68.9 ms 64 bytes from 172.16.40.52: icmp_seq=2 ttl=63 time=68.8 ms ^C --- 172.16.40.52 ping statistics --- 3 packets transmitted, 2 received, 33% packet loss, time 2003ms rtt min/avg/max/mdev = 68.893/68.898/68.904/0.262 ms# wget http://172.16.40.52/testfile.html --2019-10-15 16:29:34-- http://172.16.40.52/testfile.html Connecting to 172.16.40.52:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1048576000 (1000M) [text/html] Saving to: ‘testfile.html’ testfile.html 23%[================> ] 239,76M 6,59MB/s eta 2m 14s ^COpenVPN AS server, with 250ms additional delay:
# tc qdisc del dev enxd481d720804b root netem # tc qdisc add dev enxd481d720804b root netem delay 250ms# ping 172.16.40.52 PING 172.16.40.52 (172.16.40.52) 56(84) bytes of data. 64 bytes from 172.16.40.52: icmp_seq=1 ttl=63 time=268 ms 64 bytes from 172.16.40.52: icmp_seq=2 ttl=63 time=269 ms ^C --- 172.16.40.52 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 268.639/269.060/269.481/0.421 ms# wget http://172.16.40.52/testfile.html --2019-10-15 16:31:51-- http://172.16.40.52/testfile.html Connecting to 172.16.40.52:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1048576000 (1000M) [text/html] Saving to: ‘testfile.html.1’ testfile.html.1 5%[==> ] 50,46M 2,45MB/s eta 8m 15s ^Cpfsense openpvn, with zero additional delay:
Not exactly slow, but slower:
wget http://172.16.40.52/testfile.html --2019-10-15 16:40:45-- http://172.16.40.52/testfile.html Connecting to 172.16.40.52:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1048576000 (1000M) [text/html] Saving to: ‘testfile.html.3’ testfile.html.3 30%[=====================> ] 302,08M 10,2MB/s eta 68spfsense openpvn, with additional 50ms delay:
wget http://172.16.40.52/testfile.html --2019-10-15 16:51:58-- http://172.16.40.52/testfile.html Connecting to 172.16.40.52:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1048576000 (1000M) [text/html] Saving to: ‘testfile.html’ testfile.html 8%[=====> ] 82,58M 1,64MB/s eta 12m 30s^Cpfsense openpvn, with additional 250ms delay:
# wget http://172.16.40.52/testfile.html --2019-10-15 16:32:50-- http://172.16.40.52/testfile.html Connecting to 172.16.40.52:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1048576000 (1000M) [text/html] Saving to: ‘testfile.html.2’ testfile.html.2 1%[ ] 13,00M 817KB/s eta 30m 5s ^CAnother thing I've noticed it, that the roundtrip times go through the roof while downloading:
Pfsense openvpn, with additional 250ms delay, while download is running:
64 bytes from 172.16.40.52: icmp_seq=19 ttl=63 time=2944 ms 64 bytes from 172.16.40.52: icmp_seq=20 ttl=63 time=2484 ms 64 bytes from 172.16.40.52: icmp_seq=21 ttl=63 time=2280 ms 64 bytes from 172.16.40.52: icmp_seq=22 ttl=63 time=2623 ms 64 bytes from 172.16.40.52: icmp_seq=23 ttl=63 time=2432 ms 64 bytes from 172.16.40.52: icmp_seq=24 ttl=63 time=2503 ms 64 bytes from 172.16.40.52: icmp_seq=25 ttl=63 time=2310 ms 64 bytes from 172.16.40.52: icmp_seq=26 ttl=63 time=2114 ms 64 bytes from 172.16.40.52: icmp_seq=27 ttl=63 time=1658 msOpenVPN AS server, with additional 250ms delay, while download is running:
64 bytes from 172.16.40.52: icmp_seq=35 ttl=63 time=603 ms 64 bytes from 172.16.40.52: icmp_seq=36 ttl=63 time=367 ms 64 bytes from 172.16.40.52: icmp_seq=37 ttl=63 time=584 ms 64 bytes from 172.16.40.52: icmp_seq=38 ttl=63 time=435 ms 64 bytes from 172.16.40.52: icmp_seq=39 ttl=63 time=499 ms 64 bytes from 172.16.40.52: icmp_seq=40 ttl=63 time=500 ms -
Did some more tests to verify my suspicion that the issue is somehow due to latency, not openpvn on pfsense being generally slower.
Configured a router on the outside leg of the pfsense, i.e. without any ISPs or internet in-between, ping around 1.5ms over the tunnel.
result with openvpn on pfsense is around 11,7Megabyte/s
# ping 172.16.40.52 PING 172.16.40.52 (172.16.40.52) 56(84) bytes of data. 64 bytes from 172.16.40.52: icmp_seq=1 ttl=63 time=1.34 ms 64 bytes from 172.16.40.52: icmp_seq=2 ttl=63 time=1.57 ms 64 bytes from 172.16.40.52: icmp_seq=3 ttl=63 time=1.51 ms 64 bytes from 172.16.40.52: icmp_seq=4 ttl=63 time=1.16 ms 64 bytes from 172.16.40.52: icmp_seq=5 ttl=63 time=1.43 ms 64 bytes from 172.16.40.52: icmp_seq=6 ttl=63 time=1.47 ms ^C --- 172.16.40.52 ping statistics --- 6 packets transmitted, 6 received, 0% packet loss, time 5008ms rtt min/avg/max/mdev = 1.166/1.418/1.573/0.135 ms wget http://172.16.40.52/testfile.html --2019-10-15 18:21:48-- http://172.16.40.52/testfile.html Connecting to 172.16.40.52:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1048576000 (1000M) [text/html] Saving to: ‘testfile.html.20’ testfile.html.20 11%[=========> ] 112,36M 11,7MB/s eta 80s ^Cresult with OpenVPN AS server, it's slighthly slower when used with very low latency:
# wget http://172.16.40.52/testfile.html --2019-10-15 18:24:42-- http://172.16.40.52/testfile.html Connecting to 172.16.40.52:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1048576000 (1000M) [text/html] Saving to: ‘testfile.html.21’ testfile.html.21 10%[========> ] 104,87M 11,0MB/s eta 81s ^C -
Made another test to see if pfsense behaves different.
Downloaded a testfile on a machine with additional 250ms delay configured. All machines on a local LAN with Gigabit switches in-between:
Downloading on a Linux machine gives around 12Mbyte per second:
$ curl http://172.16.34.206/testfile.img --output testfile.img % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 4 3320M 4 155M 0 0 9.8M 0 0:05:36 0:00:15 0:05:21 12.0M^CDownload on the pfsense machine, gives only around 6.5Mbyte per second:
: curl http://172.16.34.206/testfile.img --output testfile.img % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 3 3320M 3 118M 0 0 6125k 0 0:09:15 0:00:19 0:08:56 6578k^CHowever, it's not that the pfsense machine is generally slower, when removing the artificial latency, the download on the pfsense reaches the expected >100Mbyte/s on a gigabit network:
: curl http://172.16.34.206/testfile.img --output testfile.img % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 9 3320M 9 318M 0 0 111M 0 0:00:29 0:00:02 0:00:27 111M^C