doing more and more testing. two systems now. both 1151 based. both setups have the same memory, 32GB (16GB x2 of DDR4-2666Mhz ECC UDIMM)
pfSense Hardware
Supermicro
1019C-FHTN8 with
Intel Xeon E-2278G (8c/16t, 3.4Ghz, 5.0Ghz turbo), idles at ~26w
Supermicro
505-203B /
X11SCL-IF with
Intel Pentium Gold G5400 (2c/4t, 3.7Ghz, no turbo), idles at ~16w
Both systems have Intel I210 NICs, but I also tested an Intel X710-DA2 10g dual port SFP+ NIC (on the LAN side only). The 1019C-FHTN8 is fun because it has 8 i210 NICs!
aaaf76da-644d-4dc7-a768-d6f05bb91d92-image.png
OpenVPN Clients
i9-9900KF running Ubuntu 20.04
i7-7800X running Ubuntu 20.04
Both clients are AIO water-cooled and slightly overclocked, so there should be no client-side bottlenecks with 1 Gbps links.
Testing Matrix
pfSense 2.4.5-p1 vs pfSense 2.5.0-nightly
VM vs Bare metal installs
PCIe pass-through of NICs vs VirtIO
Again, is all cases, this OpenVPN test is totally bogus and is wildly off from real world numbers.
openvpn --genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-gcm
Observations
Proxmox KVM adds about 10-20% overhead
VirtIO NIC perform nearly identical for 1 Gbps vs PCI pass-through (probably due to both CPUs being fairly powerful)
pfSense 2.5 is about 4% faster than pfSense 2.4 in iperf3 tests
OpenSSL could be used to compare
openssl speed -elapsed -evp aes-256-gcm the results of this test matched nearly the differences in each iperf3 test, percentage-wise
X710-DA2 NIC adds about 4-5 watts to each system's total idle power
Bare Metal Results
Intel Xeon E-2278G through using OpenVPN with AES-256-GCM was ~810-850 Mbps
Intel Pentium Gold G5400 using OpenVPN with AES-256-GCM was ~760-800 Mbps
Before I sent back my Supermicro A2SDi-4C-HLN4F, Intel Atom C3558, I managed to do some quick testing
Observations
Idles at 22w, but maxed out at 26w, whereas the 1151 systems maxed at at 40w and 110w when CPUs are loaded with stress-ng --matrix 0
Under Proxmox as a guest, OpenVPN performed at
nearly 50% loss in total throughput using a simple iperf3 test
C3558 was just not great under Hypervisor/Guest situation, even though pfSense was the only guest on an otherwise idle system. I have no explanation, other than it was repeatable and what I observed.
Conclusions
If you are using some embedded CPU like Intel Atom, than bare metal setup is the way to go. If you are using a fairly fast CPU, even the Pentium Gold series, it seems like for gigabit speeds on firewall, CPU is not the bottleneck. For OpenVPN itself, I was unable to achieve 1 Gbps AES-256-GCM even with the E-2278G @ 5Ghz.
The convenience of VM, being able to easily snapshot VM before a major upgrade, etc, probably outweighs the OpenVPN performance hit, plus the power savings if you are already running a Proxmox setup. I would love for pfSense with ZFS to support taking a snapshot of itself before an upgrade so you can easily rollback if it goes south. If you needed real serious OpenVPN performance, you'd probably wouldn't be doing it on your router anyway and using a VPN appliance.
I did not test any VLAN performance, which is all done on the CPU with pfSense, but I would imagine the VM overhead would exist there as well.
I have CenturyLink Fiber, so it uses PPPoE and the FreeBSD bug (although pfSense won't call it bug for some odd reason, which it does not exist in Linux), basically only uses 1 of the WAN NIC's queues, so when testing outside of my lab and actually hooking this up to the internet, my overall speeds were even worst, given its basically singled thread now inside the kernel. Documented here, here, and here.
Thoughts
FreeBSD has become a toy compared to Linux over the past decade. The Linux device drivers, kernel, applications, etc all have eclipsed BSDs at this point and with nftables replacing iptables on Linux, I would love to see pfSense router based on Linux instead of FreeBSD :)
I also tested Wireguard on Debian 10.6 and Ubuntu 20.04, behind pfSense, and in each case, Wireguard was easily able to achieve 1 Gbps. Wireguard is probably the future of VPNs at this point :)