Pfsense hardware for home

garyd9

I have AES-NI selected under System > Advanced > Miscellaneous > Cryptographic Hardware

…
When loading the aesni.ko module in WebUI, crypto will take place in kernel, not in hardware.

When not loading the aesni.ko module in WebUI, crypto takes place on crypto hardware if there is any, not in kernel.

I'm confused about this. Are you saying, for openVPN specifically, that turning OFF the "crypto h/w" option in pfsense results in hardware crypto working, but turning ON the h/w option in pfsense results in h/w crypto NOT working?

Thanks
Gary

Pippin

Yes, we talk about OpenVPN (which uses OpenSSL for the crypto part).
And yes, that`s what I'm saying :)

Will try to explain my findings:

System/ Advanced/ Miscellaneous
Cryptographic Hardware: None
^^^
–- When selecting an option a kernel module gets loaded, but we don`t need
–- it because OpenSSL will use AES-NI on SOC automatically.

VPN/ OpenVPN/ Servers--> Edit Server
Hardware Crypto: No Hardware Crypto Acceleration
^^^
--- When selecting an option then it will set the directive "engine xxxxx" in server config file, but we
–- dont need it because OpenSSL will use AES-NI on SOC automatically. --- If it should be selectable? Yes I think so because when theres no AES-NI on SOC available, one could
--- possibly benefit from the kernel module that gets loaded by one of these options.
--- In that case Cryptographic Hardware needs to be set appropriately.

Now the test to confirm and for who is interested to compare to mine:
Make the two settings as written above, selecting None and No Hardware Crypto Acceleration.
For the first setting above to take immediate effect without a restart, do

kldunload aesni.ko

in console.
Close the WebUI as it will influence the test and if can disable installed packages.
The following however I did with packages running because didn`t feel like disabling them :)

Then this command will tell OpenSSL to not use AES-NI support on SOC
1.

env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc      23756.57k    28643.61k    29828.52k    75748.35k    76480.51k

And then this command will tell OpenSSL to use AES-NI on SOC
2.

openssl speed -elapsed -evp aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc     125684.70k   211853.44k   244509.53k   254263.64k   256703.33k

Compare 1 and 2 and see the improvement.

Now to compare against aesni.ko loaded, do

kldload aesni.ko

This command will load the AES-NI kernel module, its the same as selecting it in the WebUI, "Cryptographic Hardware: None" but on console its more convenient as one no need to restart pfSense.
To see if it`s loaded do

kldstat

To unload it do

kldunload aesni.ko

Do the same as in 1 and 2
3.

env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc       4026.26k    17522.26k    56889.00k   135259.14k   218065.68k

openssl speed -elapsed -evp aes-256-cbc
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc       4689.60k    17543.15k    58354.77k   136928.60k   217352.87k

In 3 and 4 we can see that there is no difference because aesni.ko is loaded and therefore AES-NI on SOC is not used.
We also see that 2, AES-NI on SOC gives best result.

garyd9

I hope you understand that I believe you, but I wanted to make sure that I understood you. Obviously, what you describe sounds like it's working backwards.

Enable h/w AES support and it's NOT used, but if you disable h/w AES support, then it IS used. Backwards, right?

In the linux world, such a kernel module would probably provide canned routines that make use of AES instructions. Afterall, it wouldn't make much sense to have an entire kernel module just to enable the instructions. So, the only way I can make sense of your experience is that, perhaps, the openVPN code has more efficient use of the AES instructions when compared to the canned routines in the kernel module (for the usage that openVPN makes of them) (assuming my guess on what the module does is correct.) Of course, THAT doesn't make sense either, because even poorly written canned routines using AES should perform at a significantly different speed than no use of AES-NI to begin with.

Is the AES-NI kernel module just an emulator and not really h/w support?

I wonder if the same backwards behavior occurs with IPSec.. (I haven't configured VPN yet, so haven't had a chance to play with it.)

Sadly, I'm not familiar with BSD-based kernels whatsoever, so I can't really contribute anything more than startled exclamations and questions based on knowledge of a different platform.

Pippin

@garyd9:

I hope you understand that I believe you, but I wanted to make sure that I understood you. Obviously, what you describe sounds like it's working backwards.
Enable h/w AES support and it's NOT used, but if you disable h/w AES support, then it IS used. Backwards, right?

I think the thing to understand is that there is nothing needed to enable AES-NI support because OpenSSL will detect/support it automatically.
It`s possible that the built-in code in OpenSSL is doing a better job then the kernel module.
An answer I got a while back from a OpenVPN dev:
–Using AES-NI via kernel cryptodev is almost always a bad idea - because
--it is much slower than just using the same AES-NI instructions in openssl
--userland ("same CPU opcodes, less jump-to-kernel-and-back").

--So "just don't do that"..."

the openVPN code has more efficient use of the AES instructions…........
.............
Of course, THAT doesn't make sense either

It`s actually the OpenSSL code, OpenVPN just makes a call and gets a encrypted/decrypted buffer back.
But i think it can make sense, see "same CPU opcodes, less jump-to-kernel-and-back"

Is the AES-NI kernel module just an emulator and not really h/w support?

About deeper/inner workings my knowledge is limited, only just about seeing the big picture and my results.
Most of it is from trying out myself and trying to filter correct from incorrect info from reading.

I wonder if the same backwards behavior occurs with IPSec..

Never used IPsec, it doesn`t see userland i think? That could possibly mean aesni.ko needs to be loaded?
If IPsec requires aesni.ko module and one also uses OpenVPN then i can imagine that OpenVPN performance is, to some degree, degraded, crypto-wise.
I would think so because when aesni.ko is loaded the OpenSSL built-in engine is not used.

One can repeat the above test and see what works best…

Paint

Im sorry Pippin, I know you are trying to help here…. but I want to make sure a future pfSense user is not confused by this thread.

This is correct as of pfSense 2.3.2:

OpenVPN will not show a large benefit from AES-NI until the next version of OpenVPN
AES-NI should be enabled via the pfSense GUI so that the kernel module is loaded.
IPsec VPNs will show a speed improvement with AES-NI enabled,if your processor supports it.
OpenVPN will not be slower if AES-NI is enabled.
OpenSSL will still be able to use AES-NI with the kernel modules loaded - its not an either or situation.

garyd9

@Paint:

OpenVPN will not be slower if AES-NI is enabled.

Based on the data above, that's a false statement. His tests show openVPN performing SLOWER when then AES-NI module is loaded (enabled in pfSense) when compared to it NOT being loaded.

@Paint:

IPsec VPNs will show a speed improvement with AES-NI enabled,if your processor supports it.

Is it? Has that statement been verified recently? Sure, it seems reasonable to assume IPSec would be faster with AES-NI enabled, but it ALSO seems reasonable to assume that enabling AES-NI would help, not hinder, openVPN (which Pippen shows is false.)

mauroman33

I did the test suggested by Pippin
These are the results:

System/ Advanced/ Miscellaneous
Cryptographic Hardware: AES-NI CPU-based Acceleration

openssl speed -elapsed -evp aes-256-cbc
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 4872.05k 18312.96k 59575.30k 138123.61k 219373.57k

System/ Advanced/ Miscellaneous
Cryptographic Hardware: None

openssl speed -elapsed -evp aes-256-cbc
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 155467.39k 211837.63k 244339.11k 254131.88k 256329.17k

Pippin

Thanks @mauroman33, that seems to confirm it once more.

@Paint:

Im sorry Pippin, I know you are trying to help here…. but I want to make sure a future pfSense user is not confused by this thread.

Glad to help and Im trying to take away confusion, for myself too, I not understand/know it all ;) If you have data showing otherwise, test method described and better explanations, maybe clear up the mystic. To me its a complex thing to understand, especially having no background (at all) in IT whatsoever.

This is correct as of pfSense 2.3.2:

1. OpenVPN will not show a large benefit from AES-NI until the next version of OpenVPN

2. AES-NI should be enabled via the pfSense GUI so that the kernel module is loaded.

3. IPsec VPNs will show a speed improvement with AES-NI enabled,if your processor supports it.

4. OpenVPN will not be slower if AES-NI is enabled.

5. OpenSSL will still be able to use AES-NI with the kernel modules loaded - its not an either or situation.

First part is not true, see my results.
The way i understand it the culprit is hashing not supported on hardware crypto, having bigger impact on performance.
Second part yes, it will improve even more when OpenVPN 2.4 is ready due to support for AES-GCM which has the hashing included, so to speak.
AES-GCM is not debated here though and would not be correct to compare to AES-CBC for the latter test not includes the hashing.
To test including hashing one could do: openssl speed -evp aes-256-cbc-hmac-sha1

But just do test 1 and 2 exactly as above, one will see the improvement.

Statement in itself is correct to get the aesni.ko module loaded but only for in kernel crypto which is less performing compared to using AES-NI on the SOC.
Understand the difference between kernel and userland…..which I don`t fully :)

I`ll stay away from IPSec, no experience.

I assume you mean when AES-NI module (aesni.ko) is loaded?
On a system under load, it`s better to use AES-NI on SOC (hardware acceleration), looks kind of logical to me.
Meaning not selecting anything in WebUI.

Yes, it will.
But the question is: Is crypto performance better using aesni.ko or handled by OpenSSLs built-in code for hardware crypto device support. My results show, better use OpenSSLs built-in code in case AES-NI on SOC is supported.
Also think about a system under load using CPU power that cannot be used for crypto, then better use crypto hardware.

Stewart

@Pippin and @mauroman33

Have a look at the results I found while testing on an APU1D and an APU2C4, especially the heatmap in the attachment and particularly the APU2C4.
https://forum.pfsense.org/index.php?topic=106444.msg646667#msg646667

Enabling the aes-ni in the GUI has tremendous impact, usually negatively, on the new unit. For example:


openssl speed -elapsed -evp aes-128-cbc
Without aes-ni:   116,857.16 	 167,172.30 	 205,183.44 	 216,286.74 	 219,179.69 
With aes-ni:            1,455.86 	     5,778.35 	   21,179.49 	   64,385.85 	 158,815.65 
openssl speed -elapsed -evp aes-256-cbc
Without aes-ni:   96,810.10 	 129,034.06 	 150,190.10 	 156,638.07 	 158,143.28 
With aes-ni:          1,404.00 	 5,528.13 	 19,735.86 	 55,687.85 	 119,758.85

I guess I'm only adding to the confusion. I would expect the encryption to work better with aes-ni loaded, but it definitely doesn't appear to.

mauroman33

Also the following test seems to be affected by that setting.

System/ Advanced/ Miscellaneous
Cryptographic Hardware: AES-NI CPU-based Acceleration

openvpn –genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc
3200/27.41=116.74 Mbps OpenVPN performance (estimate)

System/ Advanced/ Miscellaneous
Cryptographic Hardware: None

openvpn –genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc
3200/26.94=118.78 Mbps OpenVPN performance (estimate)

Pippin

@Stewart

I guess I'm only adding to the confusion. I would expect the encryption to work better with aes-ni loaded, but it definitely doesn't appear to.

Yes, maybe add to confusion but you seem to confirm it again.

The way I understand it/picture it in my head, your result could be expected.
When loading the module which, for what I understand means crypto in kernel, then it boils down to what the CPU is capable of.

If you are willing, you could do as described in Reply: #47 and post the four results.

garyd9

I'm really trying to understand what the end result of all this is: Is AES-NI h/w "broken" in freebsd (and therefore pfsense), or do these results only impact openssl/openVPN?

MikeV7896

@garyd9:

I'm really trying to understand what the end result of all this is: Is AES-NI h/w "broken" in freebsd (and therefore pfsense), or do these results only impact openssl/openVPN?

I think these results only impact OpenSSL/OpenVPN, since it will automatically detect when AES-NI is available, while other crypto capabilities (IPSEC?) may not do so. So other things may need the kernel module to be loaded for full benefit.

Pippin

@virgiliomi:

I think these results only impact OpenSSL/OpenVPN, since it will automatically detect when AES-NI is available

Yes, not only think but I'm pretty sure.

while other crypto capabilities (IPSEC?) may not do so. So other things may need the kernel module to be loaded for full benefit.

Yes, indeed it`s probably because IPSec is L2, but there is also a but :) in case one uses IPsec and OpenVPN simultaneously.
As I wrote before, OpenVPN/OpenSSL wil not use hardware crypto device if the module is loaded.
Therefore it will perform less.

Since I not use IPsec, never did, I will stick to my findings and use settings as described.

Edit: Forgot the simultaneous part

Stewart

@Pippin:

@Stewart

I guess I'm only adding to the confusion. I would expect the encryption to work better with aes-ni loaded, but it definitely doesn't appear to.

Yes, maybe add to confusion but you seem to confirm it again.

The way I understand it/picture it in my head, your result could be expected.
When loading the module which, for what I understand means crypto in kernel, then it boils down to what the CPU is capable of.

If you are willing, you could do as described in Reply: #47 and post the four results.

@Pippin

I can run those tests on Monday but it's my understanding that setting the aes-ni in the GUI is really just loading/unloading aesni.ko.

switchman

Here is the result of my system which does not support aes-ni.

Intel(R) Celeron(R) CPU 1037U @ 1.80GHz
2 CPUs: 1 package(s) x 2 core(s)

env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc

You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 8821510 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 2379023 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 606261 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 334626 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 42234 aes-256-cbc's in 3.01s
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 47048.05k 50752.49k 51734.27k 114219.01k 115027.43k

openssl speed -elapsed -evp aes-256-cbc

You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 22248563 aes-256-cbc's in 3.01s
Doing aes-256-cbc for 3s on 64 size blocks: 5986655 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 1515738 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 382211 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 47915 aes-256-cbc's in 3.00s
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 118350.80k 127715.31k 129342.98k 130461.35k 130839.89k

Pippin

Added:
CPU spend on userland
CPU spend on kernel/system
aes-256-cbc-hmac-sha1
8 threads -multi 8

Command for top in second SSH window:


top -s 1 -aSCHIP

No module loaded:
1. Run without hardware crypto support


env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc -multi 8
evp              97659.74k   117140.11k   123383.16k   307494.50k   339199.79k

CPU user 99-100% on all cores
CPU kernel 0-0,3% on all cores

No module loaded:
2. Run with hardware crypto support


openssl speed -elapsed -evp aes-256-cbc -multi 8
evp             618920.09k   882027.93k   1003271.00k  1041262.20k  1150214.31k

CPU user 98-100% on all cores
CPU kernel 0-0,3% on all cores

^^^ Compare 1 and 2. Improvement using hardware crypto support

No module loaded:
3. Run with hardware crypto support and hmac-sha1


openssl speed -elapsed -evp aes-256-cbc-hmac-sha1 -multi 8
evp             214362.25k   408669.56k   440537.10k   468988.33k   505701.81k

CPU user 98-100% on all cores
CPU kernel 0-0,3% on all cores

^^^ Compare 2 and 3. Decrease when including SHA1 HMAC.
Compare 1 and 3. Improvement, that would be more if I could test 1 with
aes-256-cbc-hmac-sha1, but it doesn`t let me.


kldload aesni.ko

Module loaded.
4. Run with module loaded


openssl speed -elapsed -evp aes-256-cbc -multi 8
evp              15317.98k    62119.05k   211655.98k   516341.52k   917814.69k

CPU user 1-10% on all cores
CPU kernel 89-99% on all cores

Module loaded.
5. Run with module loaded and hmac-sha1


openssl speed -elapsed -evp aes-256-cbc-hmac-sha1 -multi 8
evp             244592.48k   407765.54k   444189.26k   458533.81k   501885.04k

CPU user 98-100% on all cores
CPU kernel 0-0,3% on all cores

Compare 2 and 4. Decrease when using module.
Compare 3 and 5. About the same.
Compare 4 and 5. Decrease when including SHA1 HMAC.

See the difference between CPU in 4 and 5.
Why?

Stewart

@Stewart:

@Pippin:

@Stewart

I guess I'm only adding to the confusion. I would expect the encryption to work better with aes-ni loaded, but it definitely doesn't appear to.

Yes, maybe add to confusion but you seem to confirm it again.

The way I understand it/picture it in my head, your result could be expected.
When loading the module which, for what I understand means crypto in kernel, then it boils down to what the CPU is capable of.

If you are willing, you could do as described in Reply: #47 and post the four results.

@Pippin

I can run those tests on Monday but it's my understanding that setting the aes-ni in the GUI is really just loading/unloading aesni.ko.


kldload aesni.ko
openssl speed -elapsed -evp aes-256-cbc -multi 4
evp               5616.59k    21923.05k    78318.57k   221039.27k   460087.30k

kldunload aesni.ko
openssl speed -elapsed -evp aes-256-cbc -multi 4
evp             383593.98k   500903.62k   577343.57k   599402.27k   597577.82k

Those are reproducible.

Pippin

Yeah, reproducible here too.
The difference I pretty noticeable.

But difficult thing to dissect. :)

Stewart

However, as confusing as it is, I think we can reliably state that AES-128 (at over 567MB/s) and aes-256 (Over 442MB/s) will both perform well (by the benchmarks that I ran) regardless if the aes-ni box is checked in the GUI. I don't think any of us are using the APU2 boxes in environments where we are pushing more than 400MB/s encrypted traffic. And if they can do greater than 400MB/s encrypted then they can certainly do faster than that on pure NAT. While my iperf numbers are consistently low no matter what I try (gave up on it), speed tests show I have no problem cracking 200MB/s with Squid+SquidGuard+AV+Snort running and that shows a peak of 33% utilization in the pfSense GUI.

Can they to GB Line speed with just NAT? Probably. I can't really test it. We can safely say, however, that the APU2D4 can certainly do at least 500MB/s UTM which places it within arms reach of the SonicWall TZ400 for 2/3 less and no annual fees. Aside from expandability and convenience factors it's better than all Cisco ISA models and many Cisco ASA models. (Although I fear saying that since I'm sure some CCNA will find this thread and jump in to talk about how nothing can touch a Cisco, ever!)

Do I wish it could do full GB no matter what we throw at it? Absolutely! But since the whole kit is $200 I'm not sure there's room to complain. Does this need to be fixed/clarified? Certainly! Something needs to be addressed here but as long as we know the limitations, it's just something to work around for now. Any other thoughts?