Pfsense hardware for home
-
Yes, we talk about OpenVPN (which uses OpenSSL for the crypto part).
And yes, that`s what I'm saying :)Will try to explain my findings:
System/ Advanced/ Miscellaneous
Cryptographic Hardware: None
^^^
–- When selecting an option a kernel module gets loaded, but we don`t need
–- it because OpenSSL will use AES-NI on SOC automatically.VPN/ OpenVPN/ Servers--> Edit Server
Hardware Crypto: No Hardware Crypto Acceleration
^^^
--- When selecting an option then it will set the directive "engine xxxxx" in server config file, but we
–- dont need it because OpenSSL will use AES-NI on SOC automatically. --- If it should be selectable? Yes I think so because when theres no AES-NI on SOC available, one could
--- possibly benefit from the kernel module that gets loaded by one of these options.
--- In that case Cryptographic Hardware needs to be set appropriately.Now the test to confirm and for who is interested to compare to mine:
Make the two settings as written above, selecting None and No Hardware Crypto Acceleration.
For the first setting above to take immediate effect without a restart, dokldunload aesni.koin console.
Close the WebUI as it will influence the test and if can disable installed packages.
The following however I did with packages running because didn`t feel like disabling them :)Then this command will tell OpenSSL to not use AES-NI support on SOC
1.env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-256-cbc 23756.57k 28643.61k 29828.52k 75748.35k 76480.51kAnd then this command will tell OpenSSL to use AES-NI on SOC
2.openssl speed -elapsed -evp aes-256-cbc type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-256-cbc 125684.70k 211853.44k 244509.53k 254263.64k 256703.33kCompare 1 and 2 and see the improvement.
Now to compare against aesni.ko loaded, do
kldload aesni.koThis command will load the AES-NI kernel module, it
s the same as selecting it in the WebUI, "Cryptographic Hardware: None" but on console its more convenient as one no need to restart pfSense.
To see if it`s loaded dokldstatTo unload it do
kldunload aesni.koDo the same as in 1 and 2
3.env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-256-cbc 4026.26k 17522.26k 56889.00k 135259.14k 218065.68kopenssl speed -elapsed -evp aes-256-cbc type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-256-cbc 4689.60k 17543.15k 58354.77k 136928.60k 217352.87kIn 3 and 4 we can see that there is no difference because aesni.ko is loaded and therefore AES-NI on SOC is not used.
We also see that 2, AES-NI on SOC gives best result. -
I hope you understand that I believe you, but I wanted to make sure that I understood you. Obviously, what you describe sounds like it's working backwards.
Enable h/w AES support and it's NOT used, but if you disable h/w AES support, then it IS used. Backwards, right?
In the linux world, such a kernel module would probably provide canned routines that make use of AES instructions. Afterall, it wouldn't make much sense to have an entire kernel module just to enable the instructions. So, the only way I can make sense of your experience is that, perhaps, the openVPN code has more efficient use of the AES instructions when compared to the canned routines in the kernel module (for the usage that openVPN makes of them) (assuming my guess on what the module does is correct.) Of course, THAT doesn't make sense either, because even poorly written canned routines using AES should perform at a significantly different speed than no use of AES-NI to begin with.
Is the AES-NI kernel module just an emulator and not really h/w support?
I wonder if the same backwards behavior occurs with IPSec.. (I haven't configured VPN yet, so haven't had a chance to play with it.)
Sadly, I'm not familiar with BSD-based kernels whatsoever, so I can't really contribute anything more than startled exclamations and questions based on knowledge of a different platform.
-
I hope you understand that I believe you, but I wanted to make sure that I understood you. Obviously, what you describe sounds like it's working backwards.
Enable h/w AES support and it's NOT used, but if you disable h/w AES support, then it IS used. Backwards, right?I think the thing to understand is that there is nothing needed to enable AES-NI support because OpenSSL will detect/support it automatically.
It`s possible that the built-in code in OpenSSL is doing a better job then the kernel module.
An answer I got a while back from a OpenVPN dev:
–Using AES-NI via kernel cryptodev is almost always a bad idea - because
--it is much slower than just using the same AES-NI instructions in openssl
--userland ("same CPU opcodes, less jump-to-kernel-and-back").--So "just don't do that"..."
the openVPN code has more efficient use of the AES instructions…........
.............
Of course, THAT doesn't make sense eitherIt`s actually the OpenSSL code, OpenVPN just makes a call and gets a encrypted/decrypted buffer back.
But i think it can make sense, see "same CPU opcodes, less jump-to-kernel-and-back"Is the AES-NI kernel module just an emulator and not really h/w support?
About deeper/inner workings my knowledge is limited, only just about seeing the big picture and my results.
Most of it is from trying out myself and trying to filter correct from incorrect info from reading.I wonder if the same backwards behavior occurs with IPSec..
Never used IPsec, it doesn`t see userland i think? That could possibly mean aesni.ko needs to be loaded?
If IPsec requires aesni.ko module and one also uses OpenVPN then i can imagine that OpenVPN performance is, to some degree, degraded, crypto-wise.
I would think so because when aesni.ko is loaded the OpenSSL built-in engine is not used.One can repeat the above test and see what works best…
-
Im sorry Pippin, I know you are trying to help here…. but I want to make sure a future pfSense user is not confused by this thread.
This is correct as of pfSense 2.3.2:
-
OpenVPN will not show a large benefit from AES-NI until the next version of OpenVPN
-
AES-NI should be enabled via the pfSense GUI so that the kernel module is loaded.
-
IPsec VPNs will show a speed improvement with AES-NI enabled,if your processor supports it.
-
OpenVPN will not be slower if AES-NI is enabled.
-
OpenSSL will still be able to use AES-NI with the kernel modules loaded - its not an either or situation.
-
-
- OpenVPN will not be slower if AES-NI is enabled.
Based on the data above, that's a false statement. His tests show openVPN performing SLOWER when then AES-NI module is loaded (enabled in pfSense) when compared to it NOT being loaded.
- IPsec VPNs will show a speed improvement with AES-NI enabled,if your processor supports it.
Is it? Has that statement been verified recently? Sure, it seems reasonable to assume IPSec would be faster with AES-NI enabled, but it ALSO seems reasonable to assume that enabling AES-NI would help, not hinder, openVPN (which Pippen shows is false.)
-
I did the test suggested by Pippin
These are the results:System/ Advanced/ Miscellaneous
Cryptographic Hardware: AES-NI CPU-based Accelerationopenssl speed -elapsed -evp aes-256-cbc
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 4872.05k 18312.96k 59575.30k 138123.61k 219373.57kSystem/ Advanced/ Miscellaneous
Cryptographic Hardware: Noneopenssl speed -elapsed -evp aes-256-cbc
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 155467.39k 211837.63k 244339.11k 254131.88k 256329.17k -
Thanks @mauroman33, that seems to confirm it once more.
Im sorry Pippin, I know you are trying to help here…. but I want to make sure a future pfSense user is not confused by this thread.
Glad to help and I
m trying to take away confusion, for myself too, I not understand/know it all ;) If you have data showing otherwise, test method described and better explanations, maybe clear up the mystic. To me its a complex thing to understand, especially having no background (at all) in IT whatsoever.This is correct as of pfSense 2.3.2:
-
1. OpenVPN will not show a large benefit from AES-NI until the next version of OpenVPN
-
2. AES-NI should be enabled via the pfSense GUI so that the kernel module is loaded.
-
3. IPsec VPNs will show a speed improvement with AES-NI enabled,if your processor supports it.
-
4. OpenVPN will not be slower if AES-NI is enabled.
-
5. OpenSSL will still be able to use AES-NI with the kernel modules loaded - its not an either or situation.
First part is not true, see my results.
The way i understand it the culprit is hashing not supported on hardware crypto, having bigger impact on performance.
Second part yes, it will improve even more when OpenVPN 2.4 is ready due to support for AES-GCM which has the hashing included, so to speak.
AES-GCM is not debated here though and would not be correct to compare to AES-CBC for the latter test not includes the hashing.
To test including hashing one could do: openssl speed -evp aes-256-cbc-hmac-sha1But just do test 1 and 2 exactly as above, one will see the improvement.
Statement in itself is correct to get the aesni.ko module loaded but only for in kernel crypto which is less performing compared to using AES-NI on the SOC.
Understand the difference between kernel and userland…..which I don`t fully :)I`ll stay away from IPSec, no experience.
I assume you mean when AES-NI module (aesni.ko) is loaded?
On a system under load, it`s better to use AES-NI on SOC (hardware acceleration), looks kind of logical to me.
Meaning not selecting anything in WebUI.Yes, it will.
But the question is: Is crypto performance better using aesni.ko or handled by OpenSSLs built-in code for hardware crypto device support. My results show, better use OpenSSLs built-in code in case AES-NI on SOC is supported.
Also think about a system under load using CPU power that cannot be used for crypto, then better use crypto hardware. -
-
@Pippin and @mauroman33
Have a look at the results I found while testing on an APU1D and an APU2C4, especially the heatmap in the attachment and particularly the APU2C4.
https://forum.pfsense.org/index.php?topic=106444.msg646667#msg646667Enabling the aes-ni in the GUI has tremendous impact, usually negatively, on the new unit. For example:
openssl speed -elapsed -evp aes-128-cbc Without aes-ni: 116,857.16 167,172.30 205,183.44 216,286.74 219,179.69 With aes-ni: 1,455.86 5,778.35 21,179.49 64,385.85 158,815.65 openssl speed -elapsed -evp aes-256-cbc Without aes-ni: 96,810.10 129,034.06 150,190.10 156,638.07 158,143.28 With aes-ni: 1,404.00 5,528.13 19,735.86 55,687.85 119,758.85I guess I'm only adding to the confusion. I would expect the encryption to work better with aes-ni loaded, but it definitely doesn't appear to.
-
Also the following test seems to be affected by that setting.
System/ Advanced/ Miscellaneous
Cryptographic Hardware: AES-NI CPU-based Accelerationopenvpn –genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc
3200/27.41=116.74 Mbps OpenVPN performance (estimate)System/ Advanced/ Miscellaneous
Cryptographic Hardware: Noneopenvpn –genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc
3200/26.94=118.78 Mbps OpenVPN performance (estimate) -
I guess I'm only adding to the confusion. I would expect the encryption to work better with aes-ni loaded, but it definitely doesn't appear to.
Yes, maybe add to confusion but you seem to confirm it again.
The way I understand it/picture it in my head, your result could be expected.
When loading the module which, for what I understand means crypto in kernel, then it boils down to what the CPU is capable of.If you are willing, you could do as described in Reply: #47 and post the four results.
-
I'm really trying to understand what the end result of all this is: Is AES-NI h/w "broken" in freebsd (and therefore pfsense), or do these results only impact openssl/openVPN?
-
I'm really trying to understand what the end result of all this is: Is AES-NI h/w "broken" in freebsd (and therefore pfsense), or do these results only impact openssl/openVPN?
I think these results only impact OpenSSL/OpenVPN, since it will automatically detect when AES-NI is available, while other crypto capabilities (IPSEC?) may not do so. So other things may need the kernel module to be loaded for full benefit.
-
@virgiliomi:
I think these results only impact OpenSSL/OpenVPN, since it will automatically detect when AES-NI is available
Yes, not only think but I'm pretty sure.
while other crypto capabilities (IPSEC?) may not do so. So other things may need the kernel module to be loaded for full benefit.
Yes, indeed it`s probably because IPSec is L2, but there is also a but :) in case one uses IPsec and OpenVPN simultaneously.
As I wrote before, OpenVPN/OpenSSL wil not use hardware crypto device if the module is loaded.
Therefore it will perform less.Since I not use IPsec, never did, I will stick to my findings and use settings as described.
Edit: Forgot the simultaneous part
-
I guess I'm only adding to the confusion. I would expect the encryption to work better with aes-ni loaded, but it definitely doesn't appear to.
Yes, maybe add to confusion but you seem to confirm it again.
The way I understand it/picture it in my head, your result could be expected.
When loading the module which, for what I understand means crypto in kernel, then it boils down to what the CPU is capable of.If you are willing, you could do as described in Reply: #47 and post the four results.
I can run those tests on Monday but it's my understanding that setting the aes-ni in the GUI is really just loading/unloading aesni.ko.
-
Here is the result of my system which does not support aes-ni.
Intel(R) Celeron(R) CPU 1037U @ 1.80GHz
2 CPUs: 1 package(s) x 2 core(s)env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 8821510 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 2379023 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 606261 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 334626 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 42234 aes-256-cbc's in 3.01s
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 47048.05k 50752.49k 51734.27k 114219.01k 115027.43kopenssl speed -elapsed -evp aes-256-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 22248563 aes-256-cbc's in 3.01s
Doing aes-256-cbc for 3s on 64 size blocks: 5986655 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 1515738 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 382211 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 47915 aes-256-cbc's in 3.00s
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 118350.80k 127715.31k 129342.98k 130461.35k 130839.89k -
Added:
CPU spend on userland
CPU spend on kernel/system
aes-256-cbc-hmac-sha1
8 threads -multi 8Command for top in second SSH window:
top -s 1 -aSCHIPNo module loaded:
1. Run without hardware crypto supportenv OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc -multi 8 evp 97659.74k 117140.11k 123383.16k 307494.50k 339199.79kCPU user 99-100% on all cores
CPU kernel 0-0,3% on all coresNo module loaded:
2. Run with hardware crypto supportopenssl speed -elapsed -evp aes-256-cbc -multi 8 evp 618920.09k 882027.93k 1003271.00k 1041262.20k 1150214.31kCPU user 98-100% on all cores
CPU kernel 0-0,3% on all cores^^^ Compare 1 and 2. Improvement using hardware crypto support
No module loaded:
3. Run with hardware crypto support and hmac-sha1openssl speed -elapsed -evp aes-256-cbc-hmac-sha1 -multi 8 evp 214362.25k 408669.56k 440537.10k 468988.33k 505701.81kCPU user 98-100% on all cores
CPU kernel 0-0,3% on all cores^^^ Compare 2 and 3. Decrease when including SHA1 HMAC.
Compare 1 and 3. Improvement, that would be more if I could test 1 with
aes-256-cbc-hmac-sha1, but it doesn`t let me.kldload aesni.koModule loaded.
4. Run with module loadedopenssl speed -elapsed -evp aes-256-cbc -multi 8 evp 15317.98k 62119.05k 211655.98k 516341.52k 917814.69kCPU user 1-10% on all cores
CPU kernel 89-99% on all coresModule loaded.
5. Run with module loaded and hmac-sha1openssl speed -elapsed -evp aes-256-cbc-hmac-sha1 -multi 8 evp 244592.48k 407765.54k 444189.26k 458533.81k 501885.04kCPU user 98-100% on all cores
CPU kernel 0-0,3% on all coresCompare 2 and 4. Decrease when using module.
Compare 3 and 5. About the same.
Compare 4 and 5. Decrease when including SHA1 HMAC.See the difference between CPU in 4 and 5.
Why? -
I guess I'm only adding to the confusion. I would expect the encryption to work better with aes-ni loaded, but it definitely doesn't appear to.
Yes, maybe add to confusion but you seem to confirm it again.
The way I understand it/picture it in my head, your result could be expected.
When loading the module which, for what I understand means crypto in kernel, then it boils down to what the CPU is capable of.If you are willing, you could do as described in Reply: #47 and post the four results.
I can run those tests on Monday but it's my understanding that setting the aes-ni in the GUI is really just loading/unloading aesni.ko.
kldload aesni.ko openssl speed -elapsed -evp aes-256-cbc -multi 4 evp 5616.59k 21923.05k 78318.57k 221039.27k 460087.30k kldunload aesni.ko openssl speed -elapsed -evp aes-256-cbc -multi 4 evp 383593.98k 500903.62k 577343.57k 599402.27k 597577.82kThose are reproducible.
-
Yeah, reproducible here too.
The difference I pretty noticeable.But difficult thing to dissect. :)
-
However, as confusing as it is, I think we can reliably state that AES-128 (at over 567MB/s) and aes-256 (Over 442MB/s) will both perform well (by the benchmarks that I ran) regardless if the aes-ni box is checked in the GUI. I don't think any of us are using the APU2 boxes in environments where we are pushing more than 400MB/s encrypted traffic. And if they can do greater than 400MB/s encrypted then they can certainly do faster than that on pure NAT. While my iperf numbers are consistently low no matter what I try (gave up on it), speed tests show I have no problem cracking 200MB/s with Squid+SquidGuard+AV+Snort running and that shows a peak of 33% utilization in the pfSense GUI.
Can they to GB Line speed with just NAT? Probably. I can't really test it. We can safely say, however, that the APU2D4 can certainly do at least 500MB/s UTM which places it within arms reach of the SonicWall TZ400 for 2/3 less and no annual fees. Aside from expandability and convenience factors it's better than all Cisco ISA models and many Cisco ASA models. (Although I fear saying that since I'm sure some CCNA will find this thread and jump in to talk about how nothing can touch a Cisco, ever!)
Do I wish it could do full GB no matter what we throw at it? Absolutely! But since the whole kit is $200 I'm not sure there's room to complain. Does this need to be fixed/clarified? Certainly! Something needs to be addressed here but as long as we know the limitations, it's just something to work around for now. Any other thoughts?
-
However, as confusing as it is, I think we can reliably state that AES-128 (at over 567MB/s) and aes-256 (Over 442MB/s) will both perform well (by the benchmarks that I ran) regardless if the aes-ni box is checked in the GUI.
It depends I would think.
On a system under load where the load is taking CPU cycles it could be better to use the hardware crypto support (not loading module).400MB/s
200MB/sThink you mean Mbit/s…
Any other thoughts?
Saw a ticket last week about they will change something about the crypto module, maybe they try to address something, I don`t know.
