OpenVPN performance tests don't match up

dirtyfreebooter

Re: Asus N3050I-C for OpenVPN (100MBIT WAN)

Hi. Trying to understand my OpenVPN performance.

Setup:
Supermicro 5019D-FN8TP
CPU: Intel Xeon D-2146NT

Proxmox 6.2 hypervisor.
pfSense is guest, with PCI through of 2 NICs, 4 vCPUs, 16GiB memory, and CPU type is set to "host" in Proxmox

Following this test:

openvpn --genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-gcm

If i do this on pfSense, the performance is terrible. If I do the same test on the hypervisor, its great! Trying to understand the differences or if this is even a valid smoke test for openvpn performance.

pfSense VM:

time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-gcm
Thu Oct 29 16:16:26 2020 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
8.428u 0.007s 0:08.45 99.6%     843+177k 0+0io 0pf+0w

Proxmox Hypervisor:

time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-gcm
Thu Oct 29 16:16:02 2020 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode

real    0m0.685s
user    0m0.672s
sys     0m0.012s

According to that post, it means OpenVPN performance would be as follows:

pfSense = 3200 / 8.45 seconds = ~378 Mbps
Proxmox = 3200 / 0.685s = ~ 4672 Mbps

That is crazy difference and also the 4000+ Mbps on Linux seems wrong to me...

dirtyfreebooter

I've also done the same test on a few other machines, all the results match, even pfSense 2.5 in VM on same hypervisor.

It seems like pfSense 2.4.5 is the problem...

Ubuntu 20.04 VM on same Proxmox hypervisor with same CPU settings:

time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-gcm
Thu Oct 29 22:50:42 2020 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode

real    0m0.710s
user    0m0.698s
sys     0m0.012s

FreeBSD 12 VM on same Proxmox hypervisor

time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-gcm
Thu Oct 29 17:00:54 2020 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
        0.89 real         0.87 user         0.01 sys

pfSense 2.5 nightly on same Proxmox hypervisor

time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-gcm
Thu Oct 29 23:20:08 2020 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
0.580u 0.007s 0:00.58 100.0%     561+178k 3+0io 0pf+0w

TrueNAS Core 12.0-RELEASE with AMD 7402p

time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-128-gcm
Thu Oct 29 16:38:02 2020 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher   0.55s user 0.00s system 99% cpu 0.548 total

Ubuntu 18.04 with i7-7800X overclocked

time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-gcm
Thu Oct 29 16:47:27 2020 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode

real    0m0.467s
user    0m0.463s
sys     0m0.004s

Ubuntu 20.04 on WSL 2 with i9=9900KF

$ time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-gcm
Thu Oct 29 16:47:52 2020 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
real: 0.43s user: 0.43s sys: 0.00s

All the tests seem similar except the pfSense one is order of magnitude slower... Even when compared against the same CPU in hypervisor.

dirtyfreebooter

Even a Raspberry Pi 4 (running 32-bit Raspbian GNU/Linux 10 (buster)) performs better than pfSense 2.4.5

 time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-gcm
Fri Oct 30 00:07:07 2020 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode

real    0m8.052s
user    0m8.008s
sys     0m0.031s

dirtyfreebooter

and double checking AES-NI in pfSense 2.4.5 VM

dmesg | grep -i aes
aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
  Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard

dirtyfreebooter

I so install OPNsense 20.7 and compared a different isolated setup.

Hypervisor
Proxmox 6.2
CPU: Intel Xeon E-2287G (3.4Ghz, 5.0Ghz turbo)
Memory: 64 GiB
Disk: 1TB Samsung 970 Pro ZFS mirror

Guest 1
pfSense 2.4.5-p1
vCPU: 4 (host type)
Memory: 8 GiB
Disk: 16 GiB Virtual Block

Guest 2
OPNsense 20.7
vCPU: 4 (host type)
Memory: 8 GiB
Disk: 16 GiB Virtual Block

Test Script

#!/usr/bin/env sh
openvpn --genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --ncp-disable --cipher aes-128-gcm
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --ncp-disable --cipher aes-256-gcm
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --ncp-disable --cipher aes-128-cbc
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --ncp-disable --cipher aes-256-cbc

pfSense

openvpn --version
OpenVPN 2.4.9 amd64-portbld-freebsd11.3 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on May  4 2020
library versions: OpenSSL 1.0.2u-freebsd  20 Dec 2019, LZO 2.10
Originally developed by James Yonan
Copyright (C) 2002-2018 OpenVPN Inc <sales@openvpn.net>
Compile time defines: enable_async_push=no enable_comp_stub=no enable_crypto=yes enable_crypto_ofb_cfb=yes enable_debug=yes enable_def_auth=yes enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=needless enable_fragment=yes enable_iproute2=no enable_libtool_lock=yes enable_lz4=yes enable_lzo=yes enable_management=yes enable_multihome=yes enable_pam_dlopen=no enable_pedantic=no enable_pf=yes enable_pkcs11=no enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_selinux=no enable_server=yes enable_shared=yes enable_shared_with_static_runtimes=no enable_silent_rules=no enable_small=no enable_static=yes enable_strict=yes enable_strict_options=no enable_systemd=no enable_werror=no enable_win32_dll=yes enable_x509_alt_username=no with_aix_soname=aix with_crypto_library=openssl with_gnu_ld=yes with_mem_check=no with_sysroot=no

OPNsense

openvpn --version
OpenVPN 2.4.9 amd64-portbld-freebsd12.1 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Jul 28 2020
library versions: OpenSSL 1.1.1h  22 Sep 2020, LZO 2.10
Originally developed by James Yonan
Copyright (C) 2002-2018 OpenVPN Inc <sales@openvpn.net>
Compile time defines: enable_async_push=no enable_comp_stub=no enable_crypto=yes enable_crypto_ofb_cfb=yes enable_debug=yes enable_def_auth=yes enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=needless enable_fragment=yes enable_iproute2=no enable_libtool_lock=yes enable_lz4=yes enable_lzo=yes enable_management=yes enable_multihome=yes enable_pam_dlopen=no enable_pedantic=no enable_pf=yes enable_pkcs11=no enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_selinux=no enable_server=yes enable_shared=yes enable_shared_with_static_runtimes=no enable_silent_rules=no enable_small=no enable_static=yes enable_strict=yes enable_strict_options=no enable_systemd=no enable_werror=no enable_win32_dll=yes enable_x509_alt_username=no with_aix_soname=aix with_crypto_library=openssl with_gnu_ld=yes with_mem_check=no with_sysroot=no

Results
pfSense

./openvpn_test.sh
        5.01 real         4.99 user         0.00 sys
        5.04 real         5.02 user         0.00 sys
        5.38 real         5.36 user         0.00 sys
        5.46 real         5.42 user         0.01 sys

OPNsense

./openvpn_test.sh
        0.29 real         0.28 user         0.00 sys
        0.31 real         0.31 user         0.00 sys
        0.76 real         0.75 user         0.00 sys
        0.82 real         0.81 user         0.01 sys

So its clear, something is broken with pfSense OpenVPN or OpenSSL binaries. Not getting anywhere near the same speeds as all the other platforms on the same hardware I tested.

It would be really nice to see a response from Netgate on this.

Pippin

If you read back your topic, doesn't it occur to you that the test is flawed?

Don't draw conclusions like "something is broken with pfSense OpenVPN or OpenSSL binaries" as it will bar you from advancing insight...
;)

all the other platforms on the same hardware I tested

Now try on real hardware...

dirtyfreebooter

@Pippin wow, good insight! thanks! But I did experiments and I am drawing conclusions from those experiments. That is how scientific method works. You are welcome to dispute those findings, but you can't just say "you're wrong", either provide your wisdom or please stay out of this one-way post.

I did try this installing 2.4.5-p1 on bare Xeon D-2146NT and got the exact same results.

The D-2146NT is faster and stronger than any hardware offered by Netgate and does not match their advertised speeds.

dirtyfreebooter

Trying the 2.5.0 nightly again on the same hypervisor.

openvpn --version
OpenVPN 2.5.0 amd64-portbld-freebsd12.2 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Nov 11 2020
library versions: OpenSSL 1.1.1h-freebsd  22 Sep 2020, LZO 2.10
Originally developed by James Yonan
Copyright (C) 2002-2018 OpenVPN Inc <sales@openvpn.net>
Compile time defines: enable_async_push=yes enable_comp_stub=no enable_crypto_ofb_cfb=yes enable_debug=yes enable_def_auth=yes enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=needless enable_fragment=yes enable_iproute2=no enable_libtool_lock=yes enable_lz4=yes enable_lzo=yes enable_management=yes enable_multihome=yes enable_pam_dlopen=no enable_pedantic=no enable_pf=yes enable_pkcs11=no enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_selinux=no enable_shared=yes enable_shared_with_static_runtimes=no enable_silent_rules=no enable_small=no enable_static=yes enable_strict=yes enable_strict_options=no enable_systemd=no enable_werror=no enable_win32_dll=yes enable_x509_alt_username=yes with_aix_soname=aix with_crypto_library=openssl with_gnu_ld=yes with_mem_check=no with_sysroot=no

[2.5.0-DEVELOPMENT][root@pfSense.lan]/root: ./openvpn_test.sh
        0.34 real         0.33 user         0.00 sys
        0.35 real         0.34 user         0.00 sys
        0.81 real         0.80 user         0.00 sys
        0.86 real         0.86 user         0.00 sys

So pfSense nightly matches OPNsense.

Pippin

So you read over the ;)

It's not about

D-2146NT is faster and stronger than any hardware offered by Netgate

but about the test.
Anyway:

[2.5.0-DEVELOPMENT][root@pfSense.lan]/root: ./openvpn_test.sh
0.34 real 0.33 user 0.00 sys
0.35 real 0.34 user 0.00 sys
0.81 real 0.80 user 0.00 sys
0.86 real 0.86 user 0.00 sys

If we take 0.34 for aes-128-gcm and do the calculation to get an estimated maximum OpenVPN performance, as in line speed:

3200/0.34 ~ 9411 Mbps

something is wrong.
Don't test in VM, results will vary/be skewed.

dirtyfreebooter

VM is perfectly fine here. You literally don't know what you are talking about if you think otherwise.

The fact that multiple OSes bare or VM all produce the same results and its only pfSense 2.4.5 (non even pfSense 2.5.0) is not suspect to you? How do you explain that? Why is pfSense 2.4.5 ok and all others Linux, FreeBSD, OPNsense, pfSense 2.5 all wrong?

Its a 5.0Ghz CPU with AES-NI, that is about 10gbe of raw crypto, so i did say I found that strange in post #1... but instead of offering wisdom on that, you just say incorrect things like dont run on VM. hahahahhahahahahahahhahahahahahahahahahahahahahahah. so dumb

elbuit

In March I did similar test pfsense and OPNsense in a proxmox passing all AES-NI instructions from to the VM.
Both performs similar and I got maximum performance of 170Mbps (iperf over VPN).

When I moved to a non virtualized, I got close to a 1Gbps.
That is the max I expected because openvpn isn't a multithreaded server.

I don't know why that poor performance, I guess that It doesn't use aes-ni instructions although I enabled in OpenVPN conf, system conf, cryptodev, ...

The good news are that both pfsense and opnsense solved this.

dirtyfreebooter

I just don't get your logic here. I have no clue how you setup your VM or what hypervisor you were using, but maybe you setup your VMs incorrectly? I dunno.

I have test 2 systems:

Intel Xeon D-2146NT (3.0 Ghz)
Intel Xeon E-2278G (5.0 Ghz)

Both times I tested these in isolation (nothing else is running on the systems, just setup specifically for test)
Both times I tested bare metal install and with Proxmox and is all 4 cases (2 different systems * bare metal/vm), the results are exactly the same, whether run bare metal or run as a VM. pfSense 2.4.5-p1 is slower/different than pfSense 2.5, OPNsense, FreeBSD, Linux (Debian 10.6 / Ubuntu 20.04) is ALL cases. No matter the configuration.

So its clear to me, people offering "advise" are not reading and understanding the results I posted. I still feel something funny is going on here, but pfSense in a VM does seem to be a problem.. according to the direct evidence presented here.

heper

so what you are saying is that it's already fixed in a soon-to-be-released version?

dirtyfreebooter

@heper said in OpenVPN performance tests don't match up:

so what you are saying is that it's already fixed in a soon-to-be-released version?

Yea, it does appear that 2.5.0 nightly seems match the output from all the other OSes. Is 2.5 coming soon? :)

heper

https://forum.netgate.com/topic/146195/pfsense-2-5-release-date-news/72#

dirtyfreebooter

So I got bored and built a few systems to test, all bare metal.

X11SCL-IF (G5400)
X11SCM-LN8F (Xeon E-2278G)
A2SDi-4C-HLN4F (Atom C3358)
X10SDV-2C-TLN2F (Xeon D 1508)
X10SDV-6C-TLN4F (Xeon D 1528)

Turns out the G5400 is great, idles at 16w (4w is supermicro IPMI/BMC), 3.7Ghz is great for OpenVPN, AES-NI (no AVX or turbo boost) and for a homelab the 2c/4t is great and can handle all the things pfSense can possibly do.

Here is the thing. I think this OpenVPN test that the entire internet says to do

openvpn --genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-gcm

is not correct, or accurate, or whatever. not sure who came up with it, but it is not correct.

For example, on the G5400 cpu, the test shows 6.85 seconds, which translates to 3200 / 6.85 = 467 Mbps ... but I am sitting here with this machine in a test environment and running iperf3 through an actual OpenVPN tunnel and I am getting 800 Mbps ... and as high as 850 Mbps if I enabled fast-io and 512 KiB buffers.

So this internet suggested OpenVPN quick performance test is totally bogus in my opinion, backed by testing 5 different setups. It was inaccurate for every one of those systems, most where off by more than 2x ... I didn't expect it to be crazy accurate, but 467 Mbps vs 850 Mbps isn't something to base any decisions off of.

bingo600

@dirtyfreebooter

Are you running those (slow tests) on the pfSense box ?

AFAIK
The general recommendation for iperf and others , is to generate the traffic on another node (than the pfSense).
pfSense is much faster at forwarding packets , than generating them.

/Bingo

dirtyfreebooter

@bingo600 TL;DR; iperf3 is never being run on pfSense.

I the iperf3 tests are

i7-9800X (iperf client) -> WAN -> pfSense -> LAN -> i9-9900KF (iperf server)

Also this is being done in an isolated environment, I setup this simple 3 machine environment without any other hardware, switches, etc. Just 3 machines, swapping the pfSense hardware, directly connecting with cat 6a ethernet cables.

To make sure everything is setup correctly, I also made a simple port forward to test firewall/forwarding speed and every pfSense box I tested was able to forward at 1 Gbps (941 Mbits/s), before I involved OpenVPN at all. Then when I added OpenVPN, it is the same setup, but instead of going through the port forward (NAT), things are directly connected to AES-256-GCM OpenVPN tunnel.

dirtyfreebooter

doing more and more testing. two systems now. both 1151 based. both setups have the same memory, 32GB (16GB x2 of DDR4-2666Mhz ECC UDIMM)

pfSense Hardware

Supermicro 1019C-FHTN8 with Intel Xeon E-2278G (8c/16t, 3.4Ghz, 5.0Ghz turbo), idles at ~26w
Supermicro 505-203B / X11SCL-IF with Intel Pentium Gold G5400 (2c/4t, 3.7Ghz, no turbo), idles at ~16w

Both systems have Intel I210 NICs, but I also tested an Intel X710-DA2 10g dual port SFP+ NIC (on the LAN side only). The 1019C-FHTN8 is fun because it has 8 i210 NICs!

OpenVPN Clients

i9-9900KF running Ubuntu 20.04
i7-7800X running Ubuntu 20.04

Both clients are AIO water-cooled and slightly overclocked, so there should be no client-side bottlenecks with 1 Gbps links.

Testing Matrix

pfSense 2.4.5-p1 vs pfSense 2.5.0-nightly
VM vs Bare metal installs
PCIe pass-through of NICs vs VirtIO

Again, is all cases, this OpenVPN test is totally bogus and is wildly off from real world numbers.

openvpn --genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-gcm

Observations

Proxmox KVM adds about 10-20% overhead
VirtIO NIC perform nearly identical for 1 Gbps vs PCI pass-through (probably due to both CPUs being fairly powerful)
pfSense 2.5 is about 4% faster than pfSense 2.4 in iperf3 tests
OpenSSL could be used to compare
- openssl speed -elapsed -evp aes-256-gcm the results of this test matched nearly the differences in each iperf3 test, percentage-wise
X710-DA2 NIC adds about 4-5 watts to each system's total idle power

Bare Metal Results

Intel Xeon E-2278G through using OpenVPN with AES-256-GCM was ~810-850 Mbps
Intel Pentium Gold G5400 using OpenVPN with AES-256-GCM was ~760-800 Mbps

Before I sent back my Supermicro A2SDi-4C-HLN4F, Intel Atom C3558, I managed to do some quick testing

Observations

Idles at 22w, but maxed out at 26w, whereas the 1151 systems maxed at at 40w and 110w when CPUs are loaded with stress-ng --matrix 0
Under Proxmox as a guest, OpenVPN performed at nearly 50% loss in total throughput using a simple iperf3 test

C3558 was just not great under Hypervisor/Guest situation, even though pfSense was the only guest on an otherwise idle system. I have no explanation, other than it was repeatable and what I observed.

Conclusions

If you are using some embedded CPU like Intel Atom, than bare metal setup is the way to go. If you are using a fairly fast CPU, even the Pentium Gold series, it seems like for gigabit speeds on firewall, CPU is not the bottleneck. For OpenVPN itself, I was unable to achieve 1 Gbps AES-256-GCM even with the E-2278G @ 5Ghz.

The convenience of VM, being able to easily snapshot VM before a major upgrade, etc, probably outweighs the OpenVPN performance hit, plus the power savings if you are already running a Proxmox setup. I would love for pfSense with ZFS to support taking a snapshot of itself before an upgrade so you can easily rollback if it goes south. If you needed real serious OpenVPN performance, you'd probably wouldn't be doing it on your router anyway and using a VPN appliance.

I did not test any VLAN performance, which is all done on the CPU with pfSense, but I would imagine the VM overhead would exist there as well.

I have CenturyLink Fiber, so it uses PPPoE and the FreeBSD bug (although pfSense won't call it bug for some odd reason, which it does not exist in Linux), basically only uses 1 of the WAN NIC's queues, so when testing outside of my lab and actually hooking this up to the internet, my overall speeds were even worst, given its basically singled thread now inside the kernel. Documented here, here, and here.

Thoughts

FreeBSD has become a toy compared to Linux over the past decade. The Linux device drivers, kernel, applications, etc all have eclipsed BSDs at this point and with nftables replacing iptables on Linux, I would love to see pfSense router based on Linux instead of FreeBSD :)

I also tested Wireguard on Debian 10.6 and Ubuntu 20.04, behind pfSense, and in each case, Wireguard was easily able to achieve 1 Gbps. Wireguard is probably the future of VPNs at this point :)