OpenVPN Slow - local network test

spyder0552

Spent a long time reading so hopefully I can get this detailed and the group can help me.
Just testing a OpenVPN setup on a local 1Gb Lan.
Server is running the latest version.
This is on a Dell R610 with 48Gb Ram
Doing an IPERF to the Pfsense (natted to a Iperf server on the other side..so no vpn) shows 939 - 978 Mbits/s. This is expected as they are all in the same building.
Same test, same machines, however, through the vpn tunnel (specifying the Iperf server on the other side of the wall) shows 118-135 Mbits/s
I know from reading that there is overhead and OpenVPN is slower then IPSEC. However, it cant be that much slower. I have to be doing something wrong.

Any help would be appreciated. Love the product...so I must be doing something silly.
I have tried mucking around with buffers MTU....ect...but just no luck.

Final installation will be using a full 1GB Fiber line so that is why I am trying to get as much speed as possible.
Thanks all.

My specs:
24 CPU X5680 Xeons
48 GB Ram
AES-N1
UDP tunel

Server Config:
dev ovpns3
verb 1
dev-type tun
dev-node /dev/tun3
writepid /var/run/openvpn_server3.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp4
cipher AES-128-CBC
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
client-connect /usr/local/sbin/openvpn.attributes.sh
client-disconnect /usr/local/sbin/openvpn.attributes.sh
local 172.18.19.24
engine cryptodev
tls-server
server 192.168.27.0 255.255.255.0
client-config-dir /var/etc/openvpn-csc/server3
username-as-common-name
plugin /usr/local/lib/openvpn/plugins/openvpn-plugin-auth-script.so /usr/local/sbin/ovpn_auth_verify_async user UkFESVVT false server3 1194
tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'test.com' 1"
lport 1194
management /var/etc/openvpn/server3.sock unix
push "route 172.25.0.0 255.255.0.0"
push "dhcp-option DNS 172.25.21.10"
push "dhcp-option DNS 8.8.8.8"
client-to-client
ca /var/etc/openvpn/server3.ca
cert /var/etc/openvpn/server3.cert
key /var/etc/openvpn/server3.key
dh /etc/dh-parameters.2048
tls-auth /var/etc/openvpn/server3.tls-auth 0
ncp-ciphers AES-128-GCM
persist-remote-ip
float
topology subnet
fast-io

Client Config:
dev tun
persist-tun
persist-key
cipher AES-128-CBC
ncp-ciphers AES-128-GCM
auth SHA256
tls-client
client
resolv-retry infinite
remote 172.18.19.24 1194 udp4
lport 0
verify-x509-name "test.com" name
auth-user-pass
ca pfSense-UDP4-1194-VPNUsers-ca.crt
cryptoapicert "Myprivatestuffgoeshere.LOL."
tls-auth pfSense-UDP4-1194-VPNUsers-tls.key 1
remote-cert-tls server

Rico

Try with AES-128-GCM
AES-NI is selected in System > Advanced > Miscellaneous > Cryptographic Hardware ? And does it show active (widget)?
Leave the Hardware Crypto setting in the OpenVPN instance set to No Hardware Crypto.

-Rico

spyder0552

@Rico
Yes, Hardwareware Crypto shows AES-CBC,AES-XTS,AES-GCM,AES-ICM in the dashboard widget.
OpenVPN was showing BSD Cryptodev engine.....Turning that to none and testing again.

BRB.

spyder0552

Rico,
So that gave me an ever so slight increase. I think I am up to 165 Mb/s.

So still has to be something weird. You can see the first set is going through the VPN while the other is just hitting the wan and nat to the iperf server

No VPN.png

Rico

Did you try with GCM or still CBC?
Also try to set

sndbuf 524288
rcvbuf 524288

-Rico

spyder0552

@Rico
Buffers go in the server config, client config or both?

Rico

Set the Send/Receive Buffer on both.

-Rico

spyder0552

Thank you for trying to help btw.
Changed to use GCM and set the buffers on both sides...actually made it a little worse.

I am totally baffled by this. I know it can work and is not the network. Just something in OpenVpn is throttling it all down.

spyder0552

@Rico
Also, wondering if I need to set the tunnel size.
Pinging at 1500 shows fragmentation. Setting it to 1472 shows no frag. (hitting a box on the remote network of course).

What do you think?

johnpoz

I would take a look at this
https://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux

spyder0552

@johnpoz
Thanks, I did read this but going to go over again to see if I missed something.
I know it talks about jumbo frames....will try that and let everyone know what i find.

Rico

Pls run those two via the command line or SSH and post the output here

openvpn --genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-128-gcm

-Rico

spyder0552

@Rico

[2.4.5-RELEASE][admin@pfSense.seradex.local]/root: openvpn --genkey --secret /tm p/secret
[2.4.5-RELEASE][admin@pfSense.seradex.local]/root: time openvpn --test-crypto -- secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-128-gcm
Fri Jun 19 14:47:16 2020 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
10.007u 0.000s 0:10.00 100.0% 843+177k 0+0io 0pf+0w
[2.4.5-RELEASE][admin@pfSense.seradex.local]/root:

spyder0552

@spyder0552
Also just did a test one using UDP and setting the tunnel size to 9000....but still same results

Rico

Hmm with this output I'd expect more than ~165 Mbps but you will never get to 1 Gbps with this CPU and OpenVPN, not even close.
Run again your iperf via the tunnel and check top -aSH - does the OpenVPN process run close to 100%?
Also for testing try with Encryption Algorithm set to None and try iperf via the tunnel again.

-Rico

spyder0552

@Rico
From what I see, no where near any cpu usage

spyder0552

@spyder0552
Back on this again now. Anyone have suggestions?
This is all on the same network using Cat6 cables. Just testing from a laptop to the wall to a device on the other side.
No VPN - Speed is fantastic.
OpenVPN tunnel - speed is pathetic.

Any help would be appreciated before I have to dump this and look to another solution. I just can't believe that OpenVPN can't handle at least 400-600 Mbs.

Thanks all.

Rico

Did you try with Encryption Algorithm set to None for testing?
If you really aim for highspeed VPN traffic I think you need to try with IPsec.

-Rico

johnpoz

I'm about ready to head out - but I I want to try and see what my 4860 can do via this local sort of testing.. If find some time will do a local only test..

spyder0552

@johnpoz
Hi all.
So, still having the same issue. I have now setup a home lab and get the exact same results.
Test Lab:
Intel i5 with 32Gb Ram.
Using Windows HyperV as the host.
Pfsense and Windows10 as guest machines.

Test 1 = laptop -> Firewall -> Windows 10
Get ~900 Mb/s

Test 2 = Laptop with OpenVP -> Firewall -> Windows 10
Get ~ 170 Mb/s

This is the same results I am seeing at my office using Dell R610 with 24 CPU Xenon's.

Things I have tried:

Enable/disable the network options Hardware TCP Seg and Large Receive Offload (check box set to disable them).
Enable fast I/O (with it turned off, I go down to ~155-160)...so currently enabled.
Snd/rcv buffers. Tried default and 2Mb. No significant change.
Encryption at AES-128-CBC and GCM. No significant change.

So for anyone that wants to setup my test enviroment:

Just setup a HyperV box.
Add 2 virtual switches (one external and one internal).
Create a VM for Pfsense and give it both virtual switches (WAN = External, LAN = Internal switch).
Create a VM for a test box
Put Iperf on the test box
Create your PFsense with default OpenVPN (using TLS). Just run through the wizard.

I am sure if someone did this they would get to the same spot I am where the speed is just no where at what it should be capable of.
You can rule the hardware out to some extent as testing without the VPN shows near full speed (meaning the nics are talking to each other properly).

Anyone willing to try my test?