Pfsense hardware for home
-
Update my previous post to include loaded module in pfSense.
As you can see from the results, the best is achieved when selecting no crypto in pfSense.
It loads a module, aesni.ko, which is not needed in case one has a CPU with AES-NI support.In OpenVPN you don
t need to set anything either, as said, OpenSSL will use AES-NI whenever its available on SOC. -
AES-GCM is where AES-NI really shines, not so much AES-CBC. Check out the difference here…
Without AES-NI...
: env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-gcm You have chosen to measure elapsed time instead of user CPU time. Doing aes-256-gcm for 3s on 16 size blocks: 4318531 aes-256-gcm's in 3.01s Doing aes-256-gcm for 3s on 64 size blocks: 1237576 aes-256-gcm's in 3.00s Doing aes-256-gcm for 3s on 256 size blocks: 324193 aes-256-gcm's in 3.00s Doing aes-256-gcm for 3s on 1024 size blocks: 82041 aes-256-gcm's in 3.00s Doing aes-256-gcm for 3s on 8192 size blocks: 10292 aes-256-gcm's in 3.00sWith AES-NI…
: openssl speed -elapsed -evp aes-256-gcm You have chosen to measure elapsed time instead of user CPU time. Doing aes-256-gcm for 3s on 16 size blocks: 20466923 aes-256-gcm's in 3.01s Doing aes-256-gcm for 3s on 64 size blocks: 8766278 aes-256-gcm's in 3.00s Doing aes-256-gcm for 3s on 256 size blocks: 2775125 aes-256-gcm's in 3.00s Doing aes-256-gcm for 3s on 1024 size blocks: 748960 aes-256-gcm's in 3.00s Doing aes-256-gcm for 3s on 8192 size blocks: 95348 aes-256-gcm's in 3.00sThe difference is quite a bit more visible… 5 to 9 times faster depending on the block size, and I have AES-NI selected under System > Advanced > Miscellaneous > Cryptographic Hardware.
AES-CBC still needs the hash to be calculated for authentication, so it might be fast to encrypt, but it's lost in computing the hash to go along with it... AES-GCM has that all wrapped in so there's no additional processing needed. That's why it can be so much faster when accelerated.
But AES-GCM is not available as an option in the pfSense OpenVPN settings right now... so hopefully Paint is correct regarding support coming in a future version. I don't see anything in the OpenVPN category in Redmine that asks for AES-GCM support to be added, but maybe it happens with one of the other updates/fixes there?
-
AES-GCM is where AES-NI really shines, not so much AES-CBC.
Still, AES-CBC does benefit compared to not using AES-NI, I would say, use it if have it.
AES-GCM has that all wrapped in so there's no additional processing needed
Yes, no separate authentication.
I have AES-NI selected under System > Advanced > Miscellaneous > Cryptographic Hardware
Since version 1.0+ OpenSSL automatically detects if AES-NI support is available, not any module is needed to activate it.
When loading the aesni.ko module in WebUI, crypto will take place in kernel, not in hardware.
When not loading the aesni.ko module in WebUI, crypto takes place on crypto hardware if there is any, not in kernel.
But AES-GCM is not available as an option in the pfSense OpenVPN settings right now
OpenVPN will add AES-GCM (AEAD) support in version 2.4 so pfSense has to wait until OpenVPN releases version 2.4.
-
I have AES-NI selected under System > Advanced > Miscellaneous > Cryptographic Hardware
…
When loading the aesni.ko module in WebUI, crypto will take place in kernel, not in hardware.When not loading the aesni.ko module in WebUI, crypto takes place on crypto hardware if there is any, not in kernel.
I'm confused about this. Are you saying, for openVPN specifically, that turning OFF the "crypto h/w" option in pfsense results in hardware crypto working, but turning ON the h/w option in pfsense results in h/w crypto NOT working?
Thanks
Gary -
Yes, we talk about OpenVPN (which uses OpenSSL for the crypto part).
And yes, that`s what I'm saying :)Will try to explain my findings:
System/ Advanced/ Miscellaneous
Cryptographic Hardware: None
^^^
–- When selecting an option a kernel module gets loaded, but we don`t need
–- it because OpenSSL will use AES-NI on SOC automatically.VPN/ OpenVPN/ Servers--> Edit Server
Hardware Crypto: No Hardware Crypto Acceleration
^^^
--- When selecting an option then it will set the directive "engine xxxxx" in server config file, but we
–- dont need it because OpenSSL will use AES-NI on SOC automatically. --- If it should be selectable? Yes I think so because when theres no AES-NI on SOC available, one could
--- possibly benefit from the kernel module that gets loaded by one of these options.
--- In that case Cryptographic Hardware needs to be set appropriately.Now the test to confirm and for who is interested to compare to mine:
Make the two settings as written above, selecting None and No Hardware Crypto Acceleration.
For the first setting above to take immediate effect without a restart, dokldunload aesni.koin console.
Close the WebUI as it will influence the test and if can disable installed packages.
The following however I did with packages running because didn`t feel like disabling them :)Then this command will tell OpenSSL to not use AES-NI support on SOC
1.env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-256-cbc 23756.57k 28643.61k 29828.52k 75748.35k 76480.51kAnd then this command will tell OpenSSL to use AES-NI on SOC
2.openssl speed -elapsed -evp aes-256-cbc type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-256-cbc 125684.70k 211853.44k 244509.53k 254263.64k 256703.33kCompare 1 and 2 and see the improvement.
Now to compare against aesni.ko loaded, do
kldload aesni.koThis command will load the AES-NI kernel module, it
s the same as selecting it in the WebUI, "Cryptographic Hardware: None" but on console its more convenient as one no need to restart pfSense.
To see if it`s loaded dokldstatTo unload it do
kldunload aesni.koDo the same as in 1 and 2
3.env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-256-cbc 4026.26k 17522.26k 56889.00k 135259.14k 218065.68kopenssl speed -elapsed -evp aes-256-cbc type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-256-cbc 4689.60k 17543.15k 58354.77k 136928.60k 217352.87kIn 3 and 4 we can see that there is no difference because aesni.ko is loaded and therefore AES-NI on SOC is not used.
We also see that 2, AES-NI on SOC gives best result. -
I hope you understand that I believe you, but I wanted to make sure that I understood you. Obviously, what you describe sounds like it's working backwards.
Enable h/w AES support and it's NOT used, but if you disable h/w AES support, then it IS used. Backwards, right?
In the linux world, such a kernel module would probably provide canned routines that make use of AES instructions. Afterall, it wouldn't make much sense to have an entire kernel module just to enable the instructions. So, the only way I can make sense of your experience is that, perhaps, the openVPN code has more efficient use of the AES instructions when compared to the canned routines in the kernel module (for the usage that openVPN makes of them) (assuming my guess on what the module does is correct.) Of course, THAT doesn't make sense either, because even poorly written canned routines using AES should perform at a significantly different speed than no use of AES-NI to begin with.
Is the AES-NI kernel module just an emulator and not really h/w support?
I wonder if the same backwards behavior occurs with IPSec.. (I haven't configured VPN yet, so haven't had a chance to play with it.)
Sadly, I'm not familiar with BSD-based kernels whatsoever, so I can't really contribute anything more than startled exclamations and questions based on knowledge of a different platform.
-
I hope you understand that I believe you, but I wanted to make sure that I understood you. Obviously, what you describe sounds like it's working backwards.
Enable h/w AES support and it's NOT used, but if you disable h/w AES support, then it IS used. Backwards, right?I think the thing to understand is that there is nothing needed to enable AES-NI support because OpenSSL will detect/support it automatically.
It`s possible that the built-in code in OpenSSL is doing a better job then the kernel module.
An answer I got a while back from a OpenVPN dev:
–Using AES-NI via kernel cryptodev is almost always a bad idea - because
--it is much slower than just using the same AES-NI instructions in openssl
--userland ("same CPU opcodes, less jump-to-kernel-and-back").--So "just don't do that"..."
the openVPN code has more efficient use of the AES instructions…........
.............
Of course, THAT doesn't make sense eitherIt`s actually the OpenSSL code, OpenVPN just makes a call and gets a encrypted/decrypted buffer back.
But i think it can make sense, see "same CPU opcodes, less jump-to-kernel-and-back"Is the AES-NI kernel module just an emulator and not really h/w support?
About deeper/inner workings my knowledge is limited, only just about seeing the big picture and my results.
Most of it is from trying out myself and trying to filter correct from incorrect info from reading.I wonder if the same backwards behavior occurs with IPSec..
Never used IPsec, it doesn`t see userland i think? That could possibly mean aesni.ko needs to be loaded?
If IPsec requires aesni.ko module and one also uses OpenVPN then i can imagine that OpenVPN performance is, to some degree, degraded, crypto-wise.
I would think so because when aesni.ko is loaded the OpenSSL built-in engine is not used.One can repeat the above test and see what works best…
-
Im sorry Pippin, I know you are trying to help here…. but I want to make sure a future pfSense user is not confused by this thread.
This is correct as of pfSense 2.3.2:
-
OpenVPN will not show a large benefit from AES-NI until the next version of OpenVPN
-
AES-NI should be enabled via the pfSense GUI so that the kernel module is loaded.
-
IPsec VPNs will show a speed improvement with AES-NI enabled,if your processor supports it.
-
OpenVPN will not be slower if AES-NI is enabled.
-
OpenSSL will still be able to use AES-NI with the kernel modules loaded - its not an either or situation.
-
-
- OpenVPN will not be slower if AES-NI is enabled.
Based on the data above, that's a false statement. His tests show openVPN performing SLOWER when then AES-NI module is loaded (enabled in pfSense) when compared to it NOT being loaded.
- IPsec VPNs will show a speed improvement with AES-NI enabled,if your processor supports it.
Is it? Has that statement been verified recently? Sure, it seems reasonable to assume IPSec would be faster with AES-NI enabled, but it ALSO seems reasonable to assume that enabling AES-NI would help, not hinder, openVPN (which Pippen shows is false.)
-
I did the test suggested by Pippin
These are the results:System/ Advanced/ Miscellaneous
Cryptographic Hardware: AES-NI CPU-based Accelerationopenssl speed -elapsed -evp aes-256-cbc
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 4872.05k 18312.96k 59575.30k 138123.61k 219373.57kSystem/ Advanced/ Miscellaneous
Cryptographic Hardware: Noneopenssl speed -elapsed -evp aes-256-cbc
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 155467.39k 211837.63k 244339.11k 254131.88k 256329.17k -
Thanks @mauroman33, that seems to confirm it once more.
Im sorry Pippin, I know you are trying to help here…. but I want to make sure a future pfSense user is not confused by this thread.
Glad to help and I
m trying to take away confusion, for myself too, I not understand/know it all ;) If you have data showing otherwise, test method described and better explanations, maybe clear up the mystic. To me its a complex thing to understand, especially having no background (at all) in IT whatsoever.This is correct as of pfSense 2.3.2:
-
1. OpenVPN will not show a large benefit from AES-NI until the next version of OpenVPN
-
2. AES-NI should be enabled via the pfSense GUI so that the kernel module is loaded.
-
3. IPsec VPNs will show a speed improvement with AES-NI enabled,if your processor supports it.
-
4. OpenVPN will not be slower if AES-NI is enabled.
-
5. OpenSSL will still be able to use AES-NI with the kernel modules loaded - its not an either or situation.
First part is not true, see my results.
The way i understand it the culprit is hashing not supported on hardware crypto, having bigger impact on performance.
Second part yes, it will improve even more when OpenVPN 2.4 is ready due to support for AES-GCM which has the hashing included, so to speak.
AES-GCM is not debated here though and would not be correct to compare to AES-CBC for the latter test not includes the hashing.
To test including hashing one could do: openssl speed -evp aes-256-cbc-hmac-sha1But just do test 1 and 2 exactly as above, one will see the improvement.
Statement in itself is correct to get the aesni.ko module loaded but only for in kernel crypto which is less performing compared to using AES-NI on the SOC.
Understand the difference between kernel and userland…..which I don`t fully :)I`ll stay away from IPSec, no experience.
I assume you mean when AES-NI module (aesni.ko) is loaded?
On a system under load, it`s better to use AES-NI on SOC (hardware acceleration), looks kind of logical to me.
Meaning not selecting anything in WebUI.Yes, it will.
But the question is: Is crypto performance better using aesni.ko or handled by OpenSSLs built-in code for hardware crypto device support. My results show, better use OpenSSLs built-in code in case AES-NI on SOC is supported.
Also think about a system under load using CPU power that cannot be used for crypto, then better use crypto hardware. -
-
@Pippin and @mauroman33
Have a look at the results I found while testing on an APU1D and an APU2C4, especially the heatmap in the attachment and particularly the APU2C4.
https://forum.pfsense.org/index.php?topic=106444.msg646667#msg646667Enabling the aes-ni in the GUI has tremendous impact, usually negatively, on the new unit. For example:
openssl speed -elapsed -evp aes-128-cbc Without aes-ni: 116,857.16 167,172.30 205,183.44 216,286.74 219,179.69 With aes-ni: 1,455.86 5,778.35 21,179.49 64,385.85 158,815.65 openssl speed -elapsed -evp aes-256-cbc Without aes-ni: 96,810.10 129,034.06 150,190.10 156,638.07 158,143.28 With aes-ni: 1,404.00 5,528.13 19,735.86 55,687.85 119,758.85I guess I'm only adding to the confusion. I would expect the encryption to work better with aes-ni loaded, but it definitely doesn't appear to.
-
Also the following test seems to be affected by that setting.
System/ Advanced/ Miscellaneous
Cryptographic Hardware: AES-NI CPU-based Accelerationopenvpn –genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc
3200/27.41=116.74 Mbps OpenVPN performance (estimate)System/ Advanced/ Miscellaneous
Cryptographic Hardware: Noneopenvpn –genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc
3200/26.94=118.78 Mbps OpenVPN performance (estimate) -
I guess I'm only adding to the confusion. I would expect the encryption to work better with aes-ni loaded, but it definitely doesn't appear to.
Yes, maybe add to confusion but you seem to confirm it again.
The way I understand it/picture it in my head, your result could be expected.
When loading the module which, for what I understand means crypto in kernel, then it boils down to what the CPU is capable of.If you are willing, you could do as described in Reply: #47 and post the four results.
-
I'm really trying to understand what the end result of all this is: Is AES-NI h/w "broken" in freebsd (and therefore pfsense), or do these results only impact openssl/openVPN?
-
I'm really trying to understand what the end result of all this is: Is AES-NI h/w "broken" in freebsd (and therefore pfsense), or do these results only impact openssl/openVPN?
I think these results only impact OpenSSL/OpenVPN, since it will automatically detect when AES-NI is available, while other crypto capabilities (IPSEC?) may not do so. So other things may need the kernel module to be loaded for full benefit.
-
@virgiliomi:
I think these results only impact OpenSSL/OpenVPN, since it will automatically detect when AES-NI is available
Yes, not only think but I'm pretty sure.
while other crypto capabilities (IPSEC?) may not do so. So other things may need the kernel module to be loaded for full benefit.
Yes, indeed it`s probably because IPSec is L2, but there is also a but :) in case one uses IPsec and OpenVPN simultaneously.
As I wrote before, OpenVPN/OpenSSL wil not use hardware crypto device if the module is loaded.
Therefore it will perform less.Since I not use IPsec, never did, I will stick to my findings and use settings as described.
Edit: Forgot the simultaneous part
-
I guess I'm only adding to the confusion. I would expect the encryption to work better with aes-ni loaded, but it definitely doesn't appear to.
Yes, maybe add to confusion but you seem to confirm it again.
The way I understand it/picture it in my head, your result could be expected.
When loading the module which, for what I understand means crypto in kernel, then it boils down to what the CPU is capable of.If you are willing, you could do as described in Reply: #47 and post the four results.
I can run those tests on Monday but it's my understanding that setting the aes-ni in the GUI is really just loading/unloading aesni.ko.
-
Here is the result of my system which does not support aes-ni.
Intel(R) Celeron(R) CPU 1037U @ 1.80GHz
2 CPUs: 1 package(s) x 2 core(s)env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 8821510 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 2379023 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 606261 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 334626 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 42234 aes-256-cbc's in 3.01s
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 47048.05k 50752.49k 51734.27k 114219.01k 115027.43kopenssl speed -elapsed -evp aes-256-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 22248563 aes-256-cbc's in 3.01s
Doing aes-256-cbc for 3s on 64 size blocks: 5986655 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 1515738 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 382211 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 47915 aes-256-cbc's in 3.00s
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 118350.80k 127715.31k 129342.98k 130461.35k 130839.89k -
Added:
CPU spend on userland
CPU spend on kernel/system
aes-256-cbc-hmac-sha1
8 threads -multi 8Command for top in second SSH window:
top -s 1 -aSCHIPNo module loaded:
1. Run without hardware crypto supportenv OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc -multi 8 evp 97659.74k 117140.11k 123383.16k 307494.50k 339199.79kCPU user 99-100% on all cores
CPU kernel 0-0,3% on all coresNo module loaded:
2. Run with hardware crypto supportopenssl speed -elapsed -evp aes-256-cbc -multi 8 evp 618920.09k 882027.93k 1003271.00k 1041262.20k 1150214.31kCPU user 98-100% on all cores
CPU kernel 0-0,3% on all cores^^^ Compare 1 and 2. Improvement using hardware crypto support
No module loaded:
3. Run with hardware crypto support and hmac-sha1openssl speed -elapsed -evp aes-256-cbc-hmac-sha1 -multi 8 evp 214362.25k 408669.56k 440537.10k 468988.33k 505701.81kCPU user 98-100% on all cores
CPU kernel 0-0,3% on all cores^^^ Compare 2 and 3. Decrease when including SHA1 HMAC.
Compare 1 and 3. Improvement, that would be more if I could test 1 with
aes-256-cbc-hmac-sha1, but it doesn`t let me.kldload aesni.koModule loaded.
4. Run with module loadedopenssl speed -elapsed -evp aes-256-cbc -multi 8 evp 15317.98k 62119.05k 211655.98k 516341.52k 917814.69kCPU user 1-10% on all cores
CPU kernel 89-99% on all coresModule loaded.
5. Run with module loaded and hmac-sha1openssl speed -elapsed -evp aes-256-cbc-hmac-sha1 -multi 8 evp 244592.48k 407765.54k 444189.26k 458533.81k 501885.04kCPU user 98-100% on all cores
CPU kernel 0-0,3% on all coresCompare 2 and 4. Decrease when using module.
Compare 3 and 5. About the same.
Compare 4 and 5. Decrease when including SHA1 HMAC.See the difference between CPU in 4 and 5.
Why?
