Improve OpenVPN throughput
-
Hello Folks,
I'm trying to find a way to improve the throughput on a single threaded connections across our OpenVPN links. We're running 500Mbps sequential fiber at out data center, and 300/20 on a remote site. The highest I was able to tune my OpenVPN settings was ~85 Mbps on a single thread iperf3 test. With "-P 4" on the iperf, I can hit 300, but that's not my goal. I'd like to try and improve the single thread to get at least 150Mb or better.
The client router is an old Dell PC with an i5-2400 and 4GB of DDR3 ram, Intel Pro 1000 PT Quad Port NIC.
The server router is an old Lenovo server, Xeon E3-1230, 16GB Ram, Intel Pro 1000 PT Quad port NIC.
Both routers running pfSense 2.4.3, checksum and TCP seg offloading enabled.Here's my OpenVPN Client config:
[2.4.3-RELEASE][admin@XXXXXXXXXXXX.com]/var/etc/openvpn: cat client2.conf dev ovpnc2 verb 1 dev-type tun dev-node /dev/tun2 writepid /var/run/openvpn_client2.pid #user nobody #group nobody script-security 3 daemon keepalive 10 60 ping-timer-rem persist-tun persist-key proto udp4 cipher AES-256-CBC auth SHA256 up /usr/local/sbin/ovpn-linkup down /usr/local/sbin/ovpn-linkdown local XXX.XXX.XXX.XXX engine cryptodev lport 0 management /var/etc/openvpn/client2.sock unix remote XXX.XXX.XXX.XXX ifconfig 10.XXX.XXX.X 10.XXX.XXX.X route 192.XXX.XXX.0 255.XXX.XXX.X route 10.0.XXX.X 255.XXX.XXX.X secret /var/etc/openvpn/client2.secret compress lzo resolv-retry infinite fast-io sndbuf 2097152 rcvbuf 2097152Here's my Server config:
dev ovpns2 verb 1 dev-type tun dev-node /dev/tun2 writepid /var/run/openvpn_server2.pid #user nobody #group nobody script-security 3 daemon keepalive 10 60 ping-timer-rem persist-tun persist-key proto udp4 cipher AES-256-CBC auth SHA256 up /usr/local/sbin/ovpn-linkup down /usr/local/sbin/ovpn-linkdown local XXX.XXX.XXX.XXX engine cryptodev ifconfig XXX.XXX.XXX.XXX XXX.XXX.XXX.XXX lport 1195 management /var/etc/openvpn/server2.sock unix push "route XXX.XXX.XXX.XXX XXX.XXX.XXX.XXX" route XXX.XXX.XXX.XXX XXX.XXX.XXX.XXX route XXX.XXX.XXX.XXX XXX.XXX.XXX.XXX secret /var/etc/openvpn/server2.secret compress lzo fast-io sndbuf 2097152 rcvbuf 2097152Here's some 'tunables' that I've tweaked to improve my throughput (started at ~55Mbps, settled at 84 after tuning)
net.inet.tcp.recvbuf_inc, increased to 131072
net.inet.tcp.sendbuf_inc, increased to 131072 (these two settings gave the most increase, higher numbers made no change)net.inet.tcp.recvspace , 262144 (no change in speed)
net.inet.tcp.sendspace, 262144 (no change in speed)
kern.bufcachepercent (Percent of System RAM kernel can use for Cache) 50 (no change in speed)
kern.ipc.nmbclusters (Network Mbufs available to system) 32786 (no change in speed)
kern.ipc.maxsockbuf (Maximum socket buffer size) 16777216 (no change in speed)I did try 'tun-mtu 9000', but this didn't change the numbers at all.
Does anyone have any more suggestions I can try to improve the performance?
Thanks in advance! -
Have you tried to remove engine cryptodev ?
Sometimes it make things slower not faster.
-Rico
-
Just tried it, lost about 8-10 Mbps. I also tried removing the AES-256-GCM (only AES-128-GCM now) from the negotiable's, made no change.
-
I'd play around with the compression in the next step, disable it completely or try lz4 to check for any impact.
-Rico
-
Didn't seem to make much difference. LZ4 v2 gave ~1-2 Mbps increase over several tests, but nothing quite like what I was hoping for.
Turning off compression didn't seem to make any difference at all, was still between 78-84 Mbps. -
@drkrieger said in Improve OpenVPN throughput:
Didn't seem to make much difference. LZ4 v2 gave ~1-2 Mbps increase over several tests, but nothing quite like what I was hoping for.
Turning off compression didn't seem to make any difference at all, was still between 78-84 Mbps.Using an SG-3100 here so low power device. I am using 128-GCM, SHA1, DH-2048. If you want to know anymore about my config I can share. I would export your profile again and test. GCM should use multiple cores. On gigabit I have topped out at 110Mbps but that is about it with this setup. IPSec on the other hand would likely provide the speeds you're looking for but I prefer to stay with OpenVPN and the lower overall throughput.
-
Weird stuff, my 11 old Xeon 3000 can easy make around 160 MBit/s OpenVPN AES-256-GCM.
Please try this test from the commandline/SSH.openvpn --genkey --secret /tmp/secret time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-gcm( 3200 / execution_time_seconds ) = Projected Maximum OpenVPN Performance in Mbps
-Rico
-
@gsmornot
I don't think GCM use multiple cores...
With my SG-3100 the max in real world tests is around 83 MBit/s OpenVPN AES-256-GCM.
But you can share your config if you want. :-)-Rico
-
@rico Thanks for that tip, good to know!
This was my output:Thu Sep 27 12:33:53 2018 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode 10.288u 0.007s 0:10.30 99.8% 814+178k 0+0io 0pf+0wSo with ~10 seconds, I'm guessing my peak would be theoretical 320 Mbps which is the maximum of the pipe on the remote site. I'm wondering how I can tune to get closer to that number.
I noticed in the test that MTU's are set to 20k. I'm guessing that I can't do that with OpenVPN's tunnel settings? -
@drkrieger said in Improve OpenVPN throughput:
I noticed in the test that MTU's are set to 20k. I'm guessing that I can't do that with OpenVPN's tunnel settings?
https://forum.netgate.com/post/619858
-Rico
-
I'm finding something interesting about this whole situation: iperf single thread only hits ~85 Mbps, yet a SMB file transfer (Windows Share) is able to hit 38.5 MB/s. Flaw with iperf, or just protocol differences?
-
@rico said in Improve OpenVPN throughput:
@gsmornot
I don't think GCM use multiple cores...
With my SG-3100 the max in real world tests is around 83 MBit/s OpenVPN AES-256-GCM.
But you can share your config if you want. :-)-Rico
Nothing special in the config, just offering to reply with any detail asked for in my current settings.
Looking, I thought it was multicore but I think what I want to say is capable of parallel processing. I am a bit more basic in my understanding. I tested the performance versus CBC and found it to be much better, maybe double, so I stuck with it. IPSec was much faster but I kept reading people say they would always pick OpenVPN first so I stuck with it. So far no issues. My main use is access through my firewall while mobile so I can make use of public wifi at times and even bigger for me block ad's with pfBlocker.
