Improve OpenVPN throughput

drkrieger

Hello Folks,

I'm trying to find a way to improve the throughput on a single threaded connections across our OpenVPN links. We're running 500Mbps sequential fiber at out data center, and 300/20 on a remote site. The highest I was able to tune my OpenVPN settings was ~85 Mbps on a single thread iperf3 test. With "-P 4" on the iperf, I can hit 300, but that's not my goal. I'd like to try and improve the single thread to get at least 150Mb or better.
The client router is an old Dell PC with an i5-2400 and 4GB of DDR3 ram, Intel Pro 1000 PT Quad Port NIC.
The server router is an old Lenovo server, Xeon E3-1230, 16GB Ram, Intel Pro 1000 PT Quad port NIC.
Both routers running pfSense 2.4.3, checksum and TCP seg offloading enabled.

Here's my OpenVPN Client config:

[2.4.3-RELEASE][admin@XXXXXXXXXXXX.com]/var/etc/openvpn: cat client2.conf
dev ovpnc2
verb 1
dev-type tun
dev-node /dev/tun2
writepid /var/run/openvpn_client2.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
cipher AES-256-CBC
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local XXX.XXX.XXX.XXX
engine cryptodev
lport 0
management /var/etc/openvpn/client2.sock unix
remote XXX.XXX.XXX.XXX
ifconfig 10.XXX.XXX.X 10.XXX.XXX.X
route 192.XXX.XXX.0 255.XXX.XXX.X
route 10.0.XXX.X 255.XXX.XXX.X
secret /var/etc/openvpn/client2.secret
compress lzo
resolv-retry infinite
fast-io
sndbuf 2097152
rcvbuf 2097152

Here's my Server config:

dev ovpns2
verb 1
dev-type tun
dev-node /dev/tun2
writepid /var/run/openvpn_server2.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
cipher AES-256-CBC
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local XXX.XXX.XXX.XXX
engine cryptodev
ifconfig XXX.XXX.XXX.XXX XXX.XXX.XXX.XXX
lport 1195
management /var/etc/openvpn/server2.sock unix
push "route XXX.XXX.XXX.XXX XXX.XXX.XXX.XXX"
route XXX.XXX.XXX.XXX XXX.XXX.XXX.XXX
route XXX.XXX.XXX.XXX XXX.XXX.XXX.XXX 
secret /var/etc/openvpn/server2.secret
compress lzo
fast-io
sndbuf 2097152
rcvbuf 2097152

Here's some 'tunables' that I've tweaked to improve my throughput (started at ~55Mbps, settled at 84 after tuning)
net.inet.tcp.recvbuf_inc, increased to 131072
net.inet.tcp.sendbuf_inc, increased to 131072 (these two settings gave the most increase, higher numbers made no change)

net.inet.tcp.recvspace , 262144 (no change in speed)
net.inet.tcp.sendspace, 262144 (no change in speed)
kern.bufcachepercent (Percent of System RAM kernel can use for Cache) 50 (no change in speed)
kern.ipc.nmbclusters (Network Mbufs available to system) 32786 (no change in speed)
kern.ipc.maxsockbuf (Maximum socket buffer size) 16777216 (no change in speed)

I did try 'tun-mtu 9000', but this didn't change the numbers at all.

Does anyone have any more suggestions I can try to improve the performance?
Thanks in advance!

Rico

Have you tried to remove engine cryptodev ?
Sometimes it make things slower not faster.

-Rico

drkrieger

Just tried it, lost about 8-10 Mbps. I also tried removing the AES-256-GCM (only AES-128-GCM now) from the negotiable's, made no change.

Rico

I'd play around with the compression in the next step, disable it completely or try lz4 to check for any impact.

-Rico

drkrieger

Didn't seem to make much difference. LZ4 v2 gave ~1-2 Mbps increase over several tests, but nothing quite like what I was hoping for.
Turning off compression didn't seem to make any difference at all, was still between 78-84 Mbps.

gsmornot

@drkrieger said in Improve OpenVPN throughput:

Didn't seem to make much difference. LZ4 v2 gave ~1-2 Mbps increase over several tests, but nothing quite like what I was hoping for.
Turning off compression didn't seem to make any difference at all, was still between 78-84 Mbps.

Using an SG-3100 here so low power device. I am using 128-GCM, SHA1, DH-2048. If you want to know anymore about my config I can share. I would export your profile again and test. GCM should use multiple cores. On gigabit I have topped out at 110Mbps but that is about it with this setup. IPSec on the other hand would likely provide the speeds you're looking for but I prefer to stay with OpenVPN and the lower overall throughput.

Rico

Weird stuff, my 11 old Xeon 3000 can easy make around 160 MBit/s OpenVPN AES-256-GCM.
Please try this test from the commandline/SSH.

openvpn --genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-gcm

( 3200 / execution_time_seconds ) = Projected Maximum OpenVPN Performance in Mbps

-Rico

Rico

@gsmornot
I don't think GCM use multiple cores...
With my SG-3100 the max in real world tests is around 83 MBit/s OpenVPN AES-256-GCM.
But you can share your config if you want. :-)

-Rico

drkrieger

@rico Thanks for that tip, good to know!
This was my output:

Thu Sep 27 12:33:53 2018 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
10.288u 0.007s 0:10.30 99.8%    814+178k 0+0io 0pf+0w

So with ~10 seconds, I'm guessing my peak would be theoretical 320 Mbps which is the maximum of the pipe on the remote site. I'm wondering how I can tune to get closer to that number.
I noticed in the test that MTU's are set to 20k. I'm guessing that I can't do that with OpenVPN's tunnel settings?

Rico

@drkrieger said in Improve OpenVPN throughput:

I noticed in the test that MTU's are set to 20k. I'm guessing that I can't do that with OpenVPN's tunnel settings?

https://forum.netgate.com/post/619858

-Rico

drkrieger

I'm finding something interesting about this whole situation: iperf single thread only hits ~85 Mbps, yet a SMB file transfer (Windows Share) is able to hit 38.5 MB/s. Flaw with iperf, or just protocol differences?

gsmornot

@rico said in Improve OpenVPN throughput:

@gsmornot
I don't think GCM use multiple cores...
With my SG-3100 the max in real world tests is around 83 MBit/s OpenVPN AES-256-GCM.
But you can share your config if you want. :-)

-Rico

Nothing special in the config, just offering to reply with any detail asked for in my current settings.

Looking, I thought it was multicore but I think what I want to say is capable of parallel processing. I am a bit more basic in my understanding. I tested the performance versus CBC and found it to be much better, maybe double, so I stuck with it. IPSec was much faster but I kept reading people say they would always pick OpenVPN first so I stuck with it. So far no issues. My main use is access through my firewall while mobile so I can make use of public wifi at times and even bigger for me block ad's with pfBlocker.