Best 2017 hardware for gigabit fiber + VPN
-
Explain what this bottle neck is clearly then. Can I just use one PFsense router and run multiple instances of the VPN. Then tell the same Pfsense router to merge it as a multi-WAN connection?
-
Somewhere in openvpn software it simply does not scale to gigabit.
So you create multiple instances, which will utilize multiple cores/threads. Create a gateway group and you can bypass the restriction for some types of traffic but not all. I.e., anything that uses only one connection will be limited to the max throughout of one openvpn instance.
-
Explain what this bottle neck is clearly then.
On OpenVPN the TUN/TAP architecture for sure.
Can I just use one PFsense router and run multiple instances of the VPN.
On OpenVPN you might be able to set up several tunnels and they all can be running on one cpu core each!
Over IPsec you might be able to set up also more then one IPsec tunnel too, but with the need of more IP addresses. -
@BlueKobold:
Explain what this bottle neck is clearly then.
On OpenVPN the TUN/TAP architecture for sure.
Not really. There are more fundamental problems with the openvpn protocol that prevent it from approaching the limits of tun/tap. In my experience when it maxes out on a high speed link, it will do so before it runs out of CPU. Fundamentally, the problem is that it can't keep enough packets in flight to saturate a higher speed link. Too much of the code is synchronous: in a simplified view, the receiver will get a packet, process it, send it on, tell the sender it's ready for another one, etc. In a more asynchronous/threaded model the receiver would get a packet, tell the sender it's ready for another one, start processing the first one, get a second one, tell the sender it's ready for another one, start processing the second one, tell the sender it's ready for another one, send the first packet on, etc. At that point the tun interface becomes a bottleneck, but one you could throw hardware at (throwing hardware at openvpn now doesn't really change things much).
-
Does the attached diagram I made shed light on the subject?
-
Yeah, you can get in the neighborhood of 300Mbps AES-128 with an SoC J3355 Celeron @ 2.0 GhZ.
Throwing a 4.2 GhZ i3-7350k at it only gets you in the 650Mbps range. Beyond that it didn't get much faster.
While that may seem like linear scaling, it isn't. One part is an SoC Celeron architecture, the other is an actively cooled desktop part with a very high clock meant to be overclocked.
-
Or you could just use a higher performance VPN such as IKEv2 with AES GCM to get you 600-700 Mbps with much more modest HW requirements if you're using your box as a VPN server.
I've yet to find a public network that won't connect through IPSec IKEv2 but I do have an OpenVPN tcp 443 server running as backup just in case.
If you're using your box as a vpn client your best bet is using gateway groups to run multiple OpenVPN client connections. That will get you again about 6-700 Mbps inbound on multi connection traffic.
-
Yeah, gateway groups are the best answer for most people. It won't work for all types of traffic but will work for a lot of it.
With gateway groups you can get probably 900Mbps from a low power $75 J3455-ITX.
-
Interesting initial question, would anyone post final (I know never is final) choice for best 2017 hardware, either mob or barebone (future proof/w AES-NI) - appreciate
-
There isn't a simple X hardware is best answer.
It depends on how much $ you want to spend, if gateway groups are a solution for your use case, if IPSec is a solution for you or are you set on OpenVPN, etc.
The simple answer for the best hardware will always be whatever modern CPU with the highest clock speed. So that would be a very expensive overclocked CPU. But since that's ridiculous, and there are diminishing returns no one does that.
IMO the i3-7350k would be the absolute upper limit for 99.9% of home use pfSense (and probably 90% of commercial) installations @$150.
That CPU would also be massive Overkill for almost all pfSense setups. You simply aren't going to exceed the limits of 2 cores with hyperthreading @ 4.2GhZ pushing packets at home. If you do find a way to do that then you are probably doing something entirely unnecessary.Most home users are probably best suited by a modern (Apollo lake as of now) SoC, or an old eBay SFF workstation desktop.
-
Interesting initial question, would anyone post final (I know never is final) choice for best 2017 hardware, either mob or barebone (future proof/w AES-NI) - appreciate
Get a Chinese Qotom mini PC which has 4 Intel LAN ports, Core i5 or for slightly less money an i3. Both have AES- NI.
https://www.aliexpress.com/store/product/Latest-New-core-I5-5250U-4-LAN-Home-computer-router-server-support-pfsense-linux-firewall-Cent/108231_32798137911.html?spm=2114.12010608.0.0.XFsGIe
-
Somewhere in openvpn software it simply does not scale to gigabit.
So you create multiple instances, which will utilize multiple cores/threads. Create a gateway group and you can bypass the restriction for some types of traffic but not all. I.e., anything that uses only one connection will be limited to the max throughout of one openvpn instance.
I've never used ipsec but I would guess you'd have no problem with that. You can use the oldest most broken / compromised encryption you want for that.
Your ISP will not attempt to decrypt your encrypted traffic no matter how easy it might be to do so.
What aspect of the software is maxing out the hardware? I want to see if I can find hardware that can handle the problem the software is causing. I can't do that search if I don't know how the software is maxing out the hardware.
For example. Is the CPU fast enough but there is a cache limitation problem when it comes to 1 Gbps? Is it a bus speed problem with how the software is sending the data? The software has to hit some hardware limitation, otherwise it would be going faster.
-
At this time you are not going to find hardware to solve the problem.
Someday if/when OpenVPN is updated, but not now.
If you want to try though, go buy an i7-7740X, put it on liquid helium, overclock it and let us know how close you get to gigabit with AES-128-GCM. Please post pics! ;) My bet is 780Mbps!
-
Interesting initial question, would anyone post final (I know never is final) choice for best 2017 hardware, either mob or barebone (future proof/w AES-NI) - appreciate
Get a Chinese Qotom mini PC which has 4 Intel LAN ports, Core i5 or for slightly less money an i3. Both have AES- NI.
https://www.aliexpress.com/store/product/Latest-New-core-I5-5250U-4-LAN-Home-computer-router-server-support-pfsense-linux-firewall-Cent/108231_32798137911.html?spm=2114.12010608.0.0.XFsGIe
yeah that looks a decent package.
-
I just need to know how the software is capping the hardware so I can try to find the best hardware for handling the problem.
-
I just need to know how the software is capping the hardware so I can try to find the best hardware for handling the problem.
This might help you understand the limitations of OpenVPN, it certainly helped me :)
https://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux
Here you will see some proper tests on OpenVPN and ways to optimise it with the right hardware. Don't forget if you are using mobile devices its unlikely that you can support the fragment command hence the above won't work and you'll be stuck with an unoptimised OpenVPN connection. However for point to point server connections or connections from Laptops/desktops to "home" it should work. Also this WILL NOT work for connections as client to VPN providers as they do not allow you to alter the connection parameters (tun size, fragment etc). In that case multi-openvpn gateways is the answer and you will be comfortably hitting 500-700mbps with a dual core quad connection OpenVPN configuration. Finally the tests are done on Linux so your milage may vary with FreeBSD which PFSense is based on.
Summary from the link above:
1. First bottleneck is the OpenSSL encryption / decryption routines perform better with larger packet sizes due to the way the algorithm works. This also helps reducing the context switching between user space and kernel space as more data are fed in one packet hence reducing the switching overhead (less switching is done)
2. Second is AES NI acceleration on the CPU and support being compiled into the OpenSSL library
3. Encryption itself. Without encryption they managed to hit almost gigabit speeds with jumbo frames in the TUNIn general you will need a CPU with the highest possible CPU clock as OpenVPN is not multithreaded. Even with that though you will NOT hit gigabit speeds due to the encryption overhead.
From my personal experience with the above settings I am hitting about 300mbps from my Digital Ocean web server to my gigabit connection at home. CPU utilisation on the Digital Ocean Ubuntu box is about 90% on the OpenVPN process so it could be the virtual CPU limiting me or the network stack/virtualisation drivers they are using. On my personal devices I use IPSec where I get a comfortable 400-500 mbps throughput and I would strongly advise you the same unless the IPSec ports are blocked for whatever reason.
