AES-NI / Cryptodev / OpenVPN – help a n00b understand
-
Now, something seems wrong with your numbers unless you're either on a 40Mbps link or your VPN provider is having trouble keeping up–an APU2 can do more than 40Mbps.
Interesting you should should say that. I have a 100/10 Mbps connection, which at speedtest.net gives me 115/12.
Another user in this thread shows a method of benchmarking openvpn at the command line (not through an actual internet connection) and estimating the "max" throughput achievable. He gets 41 Mbps for the APU2. I repeated his test on my APU2C4 and got the same result. His method:
His method is a heuristic, good for rough estimates of performance. People are putting too much into it. That said, while I'm sure I saw at least a bit more than 40Mbps on an APU2, I don't currently have one on the end of a link fast enough to confirm that, and it was almost certainly on Linux so it might not be comparable. Also, try AES-128 instead of AES-256.
-
Just wanted to follow up on APU2 VPN performance for future searchers:
I turned OFF every mention of crypto hardware acceleration, which did show an increase in performance.
To PIA, I was able to get as high as 72 Mbps and can get reliably above 60 Mbps. I haven't had time to set up a test to a server of my own where I can eliminate the internet/PIA servers as a factor, but I am happy with this performance. If/when I get around to setting up a controlled test I will report back.
-
A controlled test of what?
-
A controlled test of what?
"Controlled test" as in I can control all of the variables.
My test was done to Private Internet Access servers. I haven't had time to set up a test where I put my own VPN server on the WAN interface using a high-powered desktop so that I'm truely testing the APU's OpenVPN throughput. As it was my speeds could have been limited by the load on the PIA servers, slow internet to my house, or some other bottleneck along the way.
-
Don't waste your time. If your CPU supports AES-NI, use it, it's used by default without you selecting BSD cryptodev.
It looks like I found the source of at least one of this forums users incorrect claim that "AES-NI didn't work well prior to 2.4 and only works on AEAD ciphers" , an unsupported reddit post… ::). That line is a bunch of crap. AES-NI works great on pfSense prior to 2.4.
https://forum.pfsense.org/index.php?topic=129246.msg713031#msg713031You can't believe everything you read on the internet. If something goes against the grain, you are probably better off not taking it seriously unless that something can provide a solid reference, or at least a strong argument.
Reddit/Forum posts ≠ Solid Reference
This is true of my posts also, but you'll find a whitepaper linked in my post.
-
The origin actually is much older than that.
With the BSD Cryptodev engine loaded along with the AES-NI module, OpenVPN would latch onto that instead of using AES-NI, resulting in lower speeds because the BSD Cryptodev hooks for AES-NI only supported AES-GCM, while claiming to support more. Before 2.4, you could not run without the BSD cryptodev engine active, and on 2.4 you can.
Now if you didn't have the AES-NI module loaded, it wouldn't matter, OpenVPN would latch onto it and use it to accelerate anything it could. But you couldn't accelerate AES-GCM with IPsec without the AES-NI module loaded.
For a proper set of tests on 2.4 you'd need to run with a variety of settings in OpenVPN while also testing with the modules in their various states (aesni.ko loaded vs unloaded, cryptodev.ko loaded vs unloaded, both loaded, neither loaded).
-
Don't waste your time. If your CPU supports AES-NI, use it, it's used by default without you selecting BSD cryptodev.
I know it's being used… what I'm interested in is what the max openVPN capability is using the PC Engines APU2 board with the AES-128-CBC cipher.
For that I need a a controlled test where I remove the internet from the equation.
-
https://blog.cloudflare.com/aes-cbc-going-the-way-of-the-dodo/
-
Don't waste your time. If your CPU supports AES-NI, use it, it's used by default without you selecting BSD cryptodev.
I know it's being used… what I'm interested in is what the max openVPN capability is using the PC Engines APU2 board with the AES-128-CBC cipher.
For that I need a a controlled test where I remove the internet from the equation.
I'm beating on this myself right now and I'm a little disappointed, or perhaps I have it setup poorly.
I tried a single connection to PIA w/ OpenVPN, APU2d, AES-NI is not turned on anyway (because this thread says not to). I have multiple IPsec connections to remote clients, I use OpenVPN for outbound Internet activity.
I've tried every mix of cryptographic hardware settings.
System/Advanced/Miscellaneous - Cryptographic Hardware - AES-NI (haven't tried AMD Gecode LX yet but I don't think it applies)
In OpenVPN I've turned it on and off.I direct clients out the OpenVPN by way of an alias, that is referenced in a FW rule that points them to the PIA / OpenVPN Interface gateway. I even tried 4 OpenVPN connections, 4 interfaces, 4 gateways all in a gateway group to try to get the OpenVPN processes to run on different cores in the APU2d. I still moved around the same amount of traffic. No matter what I get 10-20mbit/sec. On top of that, I only see CPU utilizations in the 20% range and usually only on one core. The others are around 5%.
I'd love to find a way to max out my Internet connection (300/200), I'm thinking I just need to install an OpenVPN client on the few server that need to move a ton of traffic but I'd really like my router/fw to do all the networking tasks.
-
in the APU2d…
I'd love to find a way to max out my Internet connection (300/200)
You aren't going to max out 500Mbps of OpenVPN throughput on 4 x 1GHz cores from 2014.
You won't get close. Not even with gateway groups.
-
in the APU2d…
I'd love to find a way to max out my Internet connection (300/200)
You aren't going to max out 500Mbps of OpenVPN throughput on 4 x 1GHz cores from 2014.
You won't get close. Not even with gateway groups.
Of course. But I'd like to get more than 5-7%
-
OpenVPN is single threaded which is why you only see it on one core. Even running for clients in a gateway group not all traffic can utilize this so you will be stuck to one cores performance sometimes (often).
~40Mbps seems to be about the max for an APU2 with OpenVPN.
-
~40Mbps seems to be about the max for an APU2 with OpenVPN.
I seem to be able to do better than that on my APU2C4. I'm using AES-128-CBC / SHA1 with a 100/10 connection (112/12 actual), I have all mention of hardware crypto turned off (i.e., in both System/Advanced/Misc and In the OpenVPN Client Settings).
I was getting ~40 Mbit with this setup until i added the following three lines my OpenVPN Custom Options:
fast-io sndbuf 524288 rcvbuf 524288At that point I am able to get in the 90's of Mbps. I systematically add/removed them and it's the sndbuf/rcvbuf settings that are making the difference.
-
~40Mbps seems to be about the max for an APU2 with OpenVPN.
I seem to be able to do better than that on my APU2C4. I'm using AES-128-CBC / SHA1 with a 100/10 connection (112/12 actual), I have all mention of hardware crypto turned off (i.e., in both System/Advanced/Misc and In the OpenVPN Client Settings).
I was getting ~40 Mbit with this setup until i added the following three lines my OpenVPN Custom Options:
fast-io sndbuf 524288 rcvbuf 524288At that point I am able to get in the 90's of Mbps. I systematically add/removed them and it's the sndbuf/rcvbuf settings that are making the difference.
Holy crap thank you!!!!
I just got 67/21 on a single PIA OpenVPN connection using these settings on my APU2D. Much better!
-
…
In current and prior versions of pfsense there was a gotcha: the buttons in gui basically made it so that you couldn't get AES-NI for IPSEC (a desirable thing) without also getting AES-NI+/dev/crypto for OpenVPN (an undesirable thing). In 2.4 they've made changes so the two things aren't coupled and you can get AES-NI for IPSEC without screwing up OpenVPN. (It's been this way in upstream FreeBSD--by default you get cryptodev [the kernel interface that IPSEC and other kernel modules use] without /dev/crypto [the userspace interface that hurts performance on modern platforms].)VAMike - thank you for taking the time to explain this!
Just to be explicit, you recommend running with AES-NI selected in System->Advanced->Misc and "No Hardware Crypto Acceleration" in my OpenVPN settings? (pfSense 2.3.3_1)
Is this the recommended settings for pfSense prior to 2.4? I don't think the question has been explicitly answered.
-
~40Mbps seems to be about the max for an APU2 with OpenVPN.
I seem to be able to do better than that on my APU2C4. I'm using AES-128-CBC / SHA1 with a 100/10 connection (112/12 actual), I have all mention of hardware crypto turned off (i.e., in both System/Advanced/Misc and In the OpenVPN Client Settings).
I was getting ~40 Mbit with this setup until i added the following three lines my OpenVPN Custom Options:
fast-io sndbuf 524288 rcvbuf 524288At that point I am able to get in the 90's of Mbps. I systematically add/removed them and it's the sndbuf/rcvbuf settings that are making the difference.
I was about to chime in with my similar experience but it looks like you've already found what I was going to offer.
Those three settings made a huge difference for me as well. My setup (bare metal to ESXi on the same hardware) and WAN connection have changed in the meantime, but i've settled on leaving my PIA clients set with "no hardware acceleration."
Here's my initial thread on the subject; the replies led me down a rabbit hole of multiple PIA tunnels and config tweaks. https://forum.pfsense.org/index.php?topic=115992.msg643637#msg643637
-
Just a heads up to anyone adding the "fast-io" option to their OpenVPN client config. I'm pretty sure from reading the documentation that this option only applies to UDP. I'm not sure whether it would just be ignored for a client config using TCP (which is my guess) or mess it up in some way. But I don't believe there's any point to adding it to a TCP client config. Also, I realize that if you only have one client connection, you probably want to use UDP anyway. But in my situation, for example, I maintain two client connections to the same provider, and need to have one UDP and one TCP.
And if I can tack on a bump to mifronte's question . . . is there a definitive answer to whether hardware crypto acceleration should be selected in System->Advanced->Misc? It seems clear that we do not want it enabled for OpenVPN client configs, but is the system-wide setting useful in any way?
-
FYI- I added GUI knobs for fast-io and sndbuf/rcvbuf to 2.4, will be in snaps soon. See https://forum.pfsense.org/index.php?topic=130350.0 and https://redmine.pfsense.org/issues/7507
-
…
In current and prior versions of pfsense there was a gotcha: the buttons in gui basically made it so that you couldn't get AES-NI for IPSEC (a desirable thing) without also getting AES-NI+/dev/crypto for OpenVPN (an undesirable thing). In 2.4 they've made changes so the two things aren't coupled and you can get AES-NI for IPSEC without screwing up OpenVPN. (It's been this way in upstream FreeBSD--by default you get cryptodev [the kernel interface that IPSEC and other kernel modules use] without /dev/crypto [the userspace interface that hurts performance on modern platforms].)VAMike - thank you for taking the time to explain this!
Just to be explicit, you recommend running with AES-NI selected in System->Advanced->Misc and "No Hardware Crypto Acceleration" in my OpenVPN settings? (pfSense 2.3.3_1)
Is this the recommended settings for pfSense prior to 2.4? I don't think the question has been explicitly answered.
I just got my Core i5 Box , and am using pfSense 2.4.0 Beta , and would like to use/enable AES-NI
After install it defaulted to : cryptodev
In System->Advanced->Misc->Cryptographic Hardware : Should i select None or AES-NI ?
If i use "None" the AES-NI shows up on the Main page as : AES-NI CPU Crypto: Yes (inactive)
If i use "AES-NI" the AES-NI shows up on the Main page as : AES-NI CPU Crypto: Yes (active)And i get an extra line :
Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICMCould someone capable , give a definitive ansver ??
It seems like the performance is the same with all of the 3 selections , using : openssl speed -evp aes-128-cbc
openssl speed -evp aes-128-cbc Doing aes-128-cbc for 3s on 16 size blocks: 94684757 aes-128-cbc's in 2.99s Doing aes-128-cbc for 3s on 64 size blocks: 25963476 aes-128-cbc's in 3.05s Doing aes-128-cbc for 3s on 256 size blocks: 6553759 aes-128-cbc's in 3.01s Doing aes-128-cbc for 3s on 1024 size blocks: 1642176 aes-128-cbc's in 3.00s Doing aes-128-cbc for 3s on 8192 size blocks: 206252 aes-128-cbc's in 3.01s OpenSSL 1.0.2k-freebsd 26 Jan 2017 built on: date not available options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) compiler: clang The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-128-cbc 506303.87k 543971.34k 557801.49k 560529.41k 561742.59kTIA
/Bingo
-
Greetings-
Re-posting here as this is an akin topic:
Greetings!
Long-time listener, first-time caller.
I have been running pfSense in Azure (not the Netgate addition, sorry Netgate on a tight budget right now…) for sometime and and just upgraded to pfSense 2.4 and noticed that speeds from the appliance itself get 250-300 Mbps download tested with iperf (client) against he.net and scottlinux.com (public iperf servers), but my openvpn 2.4 (not to be confused with pfSense 2.4) clients are only getting a symmetric MAX 6 Mbps download and upload "capped".
I have no limiters in place:
ipfw show pipe - blank.
XML - none.My /temp/rules.limits:
set limit table-entries 2000000
set optimization conservative
set timeout { udp.first 300, udp.single 150, udp.multiple 900 }
set limit states 1429000
set limit src-nodes 1429000(which I am assuming is default, as I have no limits pushed to XML via the GUI).
Note: AES-NI Accel is noted:
CPU Type Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
4 CPUs: 1 package(s) x 4 core(s)
AES-NI CPU Crypto: Yes (active) -----------> CHECK!
Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICMOpenvpn Crypto used: AES-256-CBC (CHECK!)
OpenVPN config (Screen in GUI): Hardware Crypto: BSD Cryptodev......
Checked kernel mods loaded:
kldstat
Id Refs Address Size Name
1 8 0xffffffff80200000 2c3e9a0 kernel
2 1 0xffffffff83019000 46c6 cryptodev.ko
3 1 0xffffffff8301e000 7f92 aesni.koOn-board speed test:
openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 1240941 aes-256-cbc's in 0.11s
Doing aes-256-cbc for 3s on 64 size blocks: 1143048 aes-256-cbc's in 0.13s
Doing aes-256-cbc for 3s on 256 size blocks: 877391 aes-256-cbc's in 0.07s
Doing aes-256-cbc for 3s on 1024 size blocks: 500204 aes-256-cbc's in 0.07s
Doing aes-256-cbc for 3s on 8192 size blocks: 95778 aes-256-cbc's in 0.02s
OpenSSL 1.0.2k-freebsd 26 Jan 2017
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 181531.94k 550814.66k 3194483.14k 7284748.74k 33476837.38kBaffled. <shrugs shoulders="">.... :-\
This thread proved extremely insightful, however I am still not breaking the 6 Mbps barrier <sheds tear...=""> :'(
Any insight or corrections appreciated!
Thanks much!
C0l. P.</sheds></shrugs>
