Pfsense hardware for home

Pippin

I hope you understand that I believe you, but I wanted to make sure that I understood you. Obviously, what you describe sounds like it's working backwards.
Enable h/w AES support and it's NOT used, but if you disable h/w AES support, then it IS used. Backwards, right?

I think the thing to understand is that there is nothing needed to enable AES-NI support because OpenSSL will detect/support it automatically.
It`s possible that the built-in code in OpenSSL is doing a better job then the kernel module.
An answer I got a while back from a OpenVPN dev:
–Using AES-NI via kernel cryptodev is almost always a bad idea - because
--it is much slower than just using the same AES-NI instructions in openssl
--userland ("same CPU opcodes, less jump-to-kernel-and-back").

--So "just don't do that"..."

the openVPN code has more efficient use of the AES instructions…........
.............
Of course, THAT doesn't make sense either

It`s actually the OpenSSL code, OpenVPN just makes a call and gets a encrypted/decrypted buffer back.
But i think it can make sense, see "same CPU opcodes, less jump-to-kernel-and-back"

Is the AES-NI kernel module just an emulator and not really h/w support?

About deeper/inner workings my knowledge is limited, only just about seeing the big picture and my results.
Most of it is from trying out myself and trying to filter correct from incorrect info from reading.

I wonder if the same backwards behavior occurs with IPSec..

Never used IPsec, it doesn`t see userland i think? That could possibly mean aesni.ko needs to be loaded?
If IPsec requires aesni.ko module and one also uses OpenVPN then i can imagine that OpenVPN performance is, to some degree, degraded, crypto-wise.
I would think so because when aesni.ko is loaded the OpenSSL built-in engine is not used.

One can repeat the above test and see what works best…

Paint

Im sorry Pippin, I know you are trying to help here…. but I want to make sure a future pfSense user is not confused by this thread.

This is correct as of pfSense 2.3.2:

OpenVPN will not show a large benefit from AES-NI until the next version of OpenVPN
AES-NI should be enabled via the pfSense GUI so that the kernel module is loaded.
IPsec VPNs will show a speed improvement with AES-NI enabled,if your processor supports it.
OpenVPN will not be slower if AES-NI is enabled.
OpenSSL will still be able to use AES-NI with the kernel modules loaded - its not an either or situation.

garyd9

@Paint:

OpenVPN will not be slower if AES-NI is enabled.

Based on the data above, that's a false statement. His tests show openVPN performing SLOWER when then AES-NI module is loaded (enabled in pfSense) when compared to it NOT being loaded.

@Paint:

IPsec VPNs will show a speed improvement with AES-NI enabled,if your processor supports it.

Is it? Has that statement been verified recently? Sure, it seems reasonable to assume IPSec would be faster with AES-NI enabled, but it ALSO seems reasonable to assume that enabling AES-NI would help, not hinder, openVPN (which Pippen shows is false.)

mauroman33

I did the test suggested by Pippin
These are the results:

System/ Advanced/ Miscellaneous
Cryptographic Hardware: AES-NI CPU-based Acceleration

openssl speed -elapsed -evp aes-256-cbc
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 4872.05k 18312.96k 59575.30k 138123.61k 219373.57k

System/ Advanced/ Miscellaneous
Cryptographic Hardware: None

openssl speed -elapsed -evp aes-256-cbc
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 155467.39k 211837.63k 244339.11k 254131.88k 256329.17k

Pippin

Thanks @mauroman33, that seems to confirm it once more.

@Paint:

Im sorry Pippin, I know you are trying to help here…. but I want to make sure a future pfSense user is not confused by this thread.

Glad to help and Im trying to take away confusion, for myself too, I not understand/know it all ;) If you have data showing otherwise, test method described and better explanations, maybe clear up the mystic. To me its a complex thing to understand, especially having no background (at all) in IT whatsoever.

This is correct as of pfSense 2.3.2:

1. OpenVPN will not show a large benefit from AES-NI until the next version of OpenVPN

2. AES-NI should be enabled via the pfSense GUI so that the kernel module is loaded.

3. IPsec VPNs will show a speed improvement with AES-NI enabled,if your processor supports it.

4. OpenVPN will not be slower if AES-NI is enabled.

5. OpenSSL will still be able to use AES-NI with the kernel modules loaded - its not an either or situation.

First part is not true, see my results.
The way i understand it the culprit is hashing not supported on hardware crypto, having bigger impact on performance.
Second part yes, it will improve even more when OpenVPN 2.4 is ready due to support for AES-GCM which has the hashing included, so to speak.
AES-GCM is not debated here though and would not be correct to compare to AES-CBC for the latter test not includes the hashing.
To test including hashing one could do: openssl speed -evp aes-256-cbc-hmac-sha1

But just do test 1 and 2 exactly as above, one will see the improvement.

Statement in itself is correct to get the aesni.ko module loaded but only for in kernel crypto which is less performing compared to using AES-NI on the SOC.
Understand the difference between kernel and userland…..which I don`t fully :)

I`ll stay away from IPSec, no experience.

I assume you mean when AES-NI module (aesni.ko) is loaded?
On a system under load, it`s better to use AES-NI on SOC (hardware acceleration), looks kind of logical to me.
Meaning not selecting anything in WebUI.

Yes, it will.
But the question is: Is crypto performance better using aesni.ko or handled by OpenSSLs built-in code for hardware crypto device support. My results show, better use OpenSSLs built-in code in case AES-NI on SOC is supported.
Also think about a system under load using CPU power that cannot be used for crypto, then better use crypto hardware.

Stewart

@Pippin and @mauroman33

Have a look at the results I found while testing on an APU1D and an APU2C4, especially the heatmap in the attachment and particularly the APU2C4.
https://forum.pfsense.org/index.php?topic=106444.msg646667#msg646667

Enabling the aes-ni in the GUI has tremendous impact, usually negatively, on the new unit. For example:


openssl speed -elapsed -evp aes-128-cbc
Without aes-ni:   116,857.16 	 167,172.30 	 205,183.44 	 216,286.74 	 219,179.69 
With aes-ni:            1,455.86 	     5,778.35 	   21,179.49 	   64,385.85 	 158,815.65 
openssl speed -elapsed -evp aes-256-cbc
Without aes-ni:   96,810.10 	 129,034.06 	 150,190.10 	 156,638.07 	 158,143.28 
With aes-ni:          1,404.00 	 5,528.13 	 19,735.86 	 55,687.85 	 119,758.85

I guess I'm only adding to the confusion. I would expect the encryption to work better with aes-ni loaded, but it definitely doesn't appear to.

mauroman33

Also the following test seems to be affected by that setting.

System/ Advanced/ Miscellaneous
Cryptographic Hardware: AES-NI CPU-based Acceleration

openvpn –genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc
3200/27.41=116.74 Mbps OpenVPN performance (estimate)

System/ Advanced/ Miscellaneous
Cryptographic Hardware: None

openvpn –genkey --secret /tmp/secret
time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc
3200/26.94=118.78 Mbps OpenVPN performance (estimate)

Pippin

@Stewart

I guess I'm only adding to the confusion. I would expect the encryption to work better with aes-ni loaded, but it definitely doesn't appear to.

Yes, maybe add to confusion but you seem to confirm it again.

The way I understand it/picture it in my head, your result could be expected.
When loading the module which, for what I understand means crypto in kernel, then it boils down to what the CPU is capable of.

If you are willing, you could do as described in Reply: #47 and post the four results.

garyd9

I'm really trying to understand what the end result of all this is: Is AES-NI h/w "broken" in freebsd (and therefore pfsense), or do these results only impact openssl/openVPN?

MikeV7896

@garyd9:

I'm really trying to understand what the end result of all this is: Is AES-NI h/w "broken" in freebsd (and therefore pfsense), or do these results only impact openssl/openVPN?

I think these results only impact OpenSSL/OpenVPN, since it will automatically detect when AES-NI is available, while other crypto capabilities (IPSEC?) may not do so. So other things may need the kernel module to be loaded for full benefit.

Pippin

@virgiliomi:

I think these results only impact OpenSSL/OpenVPN, since it will automatically detect when AES-NI is available

Yes, not only think but I'm pretty sure.

while other crypto capabilities (IPSEC?) may not do so. So other things may need the kernel module to be loaded for full benefit.

Yes, indeed it`s probably because IPSec is L2, but there is also a but :) in case one uses IPsec and OpenVPN simultaneously.
As I wrote before, OpenVPN/OpenSSL wil not use hardware crypto device if the module is loaded.
Therefore it will perform less.

Since I not use IPsec, never did, I will stick to my findings and use settings as described.

Edit: Forgot the simultaneous part

Stewart

@Pippin:

@Stewart

I guess I'm only adding to the confusion. I would expect the encryption to work better with aes-ni loaded, but it definitely doesn't appear to.

Yes, maybe add to confusion but you seem to confirm it again.

The way I understand it/picture it in my head, your result could be expected.
When loading the module which, for what I understand means crypto in kernel, then it boils down to what the CPU is capable of.

If you are willing, you could do as described in Reply: #47 and post the four results.

@Pippin

I can run those tests on Monday but it's my understanding that setting the aes-ni in the GUI is really just loading/unloading aesni.ko.

switchman

Here is the result of my system which does not support aes-ni.

Intel(R) Celeron(R) CPU 1037U @ 1.80GHz
2 CPUs: 1 package(s) x 2 core(s)

env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc

You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 8821510 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 2379023 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 606261 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 334626 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 42234 aes-256-cbc's in 3.01s
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(8x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 47048.05k 50752.49k 51734.27k 114219.01k 115027.43k

openssl speed -elapsed -evp aes-256-cbc

You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 22248563 aes-256-cbc's in 3.01s
Doing aes-256-cbc for 3s on 64 size blocks: 5986655 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 1515738 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 382211 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 47915 aes-256-cbc's in 3.00s
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 118350.80k 127715.31k 129342.98k 130461.35k 130839.89k

Pippin

Added:
CPU spend on userland
CPU spend on kernel/system
aes-256-cbc-hmac-sha1
8 threads -multi 8

Command for top in second SSH window:


top -s 1 -aSCHIP

No module loaded:
1. Run without hardware crypto support


env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc -multi 8
evp              97659.74k   117140.11k   123383.16k   307494.50k   339199.79k

CPU user 99-100% on all cores
CPU kernel 0-0,3% on all cores

No module loaded:
2. Run with hardware crypto support


openssl speed -elapsed -evp aes-256-cbc -multi 8
evp             618920.09k   882027.93k   1003271.00k  1041262.20k  1150214.31k

CPU user 98-100% on all cores
CPU kernel 0-0,3% on all cores

^^^ Compare 1 and 2. Improvement using hardware crypto support

No module loaded:
3. Run with hardware crypto support and hmac-sha1


openssl speed -elapsed -evp aes-256-cbc-hmac-sha1 -multi 8
evp             214362.25k   408669.56k   440537.10k   468988.33k   505701.81k

CPU user 98-100% on all cores
CPU kernel 0-0,3% on all cores

^^^ Compare 2 and 3. Decrease when including SHA1 HMAC.
Compare 1 and 3. Improvement, that would be more if I could test 1 with
aes-256-cbc-hmac-sha1, but it doesn`t let me.


kldload aesni.ko

Module loaded.
4. Run with module loaded


openssl speed -elapsed -evp aes-256-cbc -multi 8
evp              15317.98k    62119.05k   211655.98k   516341.52k   917814.69k

CPU user 1-10% on all cores
CPU kernel 89-99% on all cores

Module loaded.
5. Run with module loaded and hmac-sha1


openssl speed -elapsed -evp aes-256-cbc-hmac-sha1 -multi 8
evp             244592.48k   407765.54k   444189.26k   458533.81k   501885.04k

CPU user 98-100% on all cores
CPU kernel 0-0,3% on all cores

Compare 2 and 4. Decrease when using module.
Compare 3 and 5. About the same.
Compare 4 and 5. Decrease when including SHA1 HMAC.

See the difference between CPU in 4 and 5.
Why?

Stewart

@Stewart:

@Pippin:

@Stewart

I guess I'm only adding to the confusion. I would expect the encryption to work better with aes-ni loaded, but it definitely doesn't appear to.

Yes, maybe add to confusion but you seem to confirm it again.

The way I understand it/picture it in my head, your result could be expected.
When loading the module which, for what I understand means crypto in kernel, then it boils down to what the CPU is capable of.

If you are willing, you could do as described in Reply: #47 and post the four results.

@Pippin

I can run those tests on Monday but it's my understanding that setting the aes-ni in the GUI is really just loading/unloading aesni.ko.


kldload aesni.ko
openssl speed -elapsed -evp aes-256-cbc -multi 4
evp               5616.59k    21923.05k    78318.57k   221039.27k   460087.30k

kldunload aesni.ko
openssl speed -elapsed -evp aes-256-cbc -multi 4
evp             383593.98k   500903.62k   577343.57k   599402.27k   597577.82k

Those are reproducible.

Pippin

Yeah, reproducible here too.
The difference I pretty noticeable.

But difficult thing to dissect. :)

Stewart

However, as confusing as it is, I think we can reliably state that AES-128 (at over 567MB/s) and aes-256 (Over 442MB/s) will both perform well (by the benchmarks that I ran) regardless if the aes-ni box is checked in the GUI. I don't think any of us are using the APU2 boxes in environments where we are pushing more than 400MB/s encrypted traffic. And if they can do greater than 400MB/s encrypted then they can certainly do faster than that on pure NAT. While my iperf numbers are consistently low no matter what I try (gave up on it), speed tests show I have no problem cracking 200MB/s with Squid+SquidGuard+AV+Snort running and that shows a peak of 33% utilization in the pfSense GUI.

Can they to GB Line speed with just NAT? Probably. I can't really test it. We can safely say, however, that the APU2D4 can certainly do at least 500MB/s UTM which places it within arms reach of the SonicWall TZ400 for 2/3 less and no annual fees. Aside from expandability and convenience factors it's better than all Cisco ISA models and many Cisco ASA models. (Although I fear saying that since I'm sure some CCNA will find this thread and jump in to talk about how nothing can touch a Cisco, ever!)

Do I wish it could do full GB no matter what we throw at it? Absolutely! But since the whole kit is $200 I'm not sure there's room to complain. Does this need to be fixed/clarified? Certainly! Something needs to be addressed here but as long as we know the limitations, it's just something to work around for now. Any other thoughts?

Pippin

@Stewart:

However, as confusing as it is, I think we can reliably state that AES-128 (at over 567MB/s) and aes-256 (Over 442MB/s) will both perform well (by the benchmarks that I ran) regardless if the aes-ni box is checked in the GUI.

It depends I would think.
On a system under load where the load is taking CPU cycles it could be better to use the hardware crypto support (not loading module).

400MB/s
200MB/s

Think you mean Mbit/s…

Any other thoughts?

Saw a ticket last week about they will change something about the crypto module, maybe they try to address something, I don`t know.

Stewart

@Pippin:

@Stewart:

However, as confusing as it is, I think we can reliably state that AES-128 (at over 567MB/s) and aes-256 (Over 442MB/s) will both perform well (by the benchmarks that I ran) regardless if the aes-ni box is checked in the GUI.

It depends I would think.
On a system under load where the load is taking CPU cycles it could be better to use the hardware crypto support (not loading module).

400MB/s
200MB/s

Think you mean Mbit/s…

Any other thoughts?

Saw a ticket last week about they will change something about the crypto module, maybe they try to address something, I don`t know.

1. Yes. My tests show anywhere from 99% loss to 4% gain when enabling the module in the GUI. It's definitely better to just leave it off. It is perplexing, though, how the APU2C lags behind the APU1D in some of the tests (particularly in small sizes). As for load, it's just easier to compare raw numbers against other products since that's what they are using. When you see a product listed at 300Mb/s VPN throughput you know it's max theoretical with no other load. I would like to know if the real world usage with the aes-ni disabled in the GUI is more akin to the command "openssl speed -elapsed aes-256-cbc -multi " or "openssl speed -elapsed -evp aes-256-cbc -multi".

2. You are correct in that it should have been Mb/s. MB/s would be rather nice! :)

Is there any way to test IPSEC performance? I'm under the impression that these tests aren't the same. What I really care about are IPSEC speeds since those are the site-to-site tunnels that would most likely be symmetrical fiber. Anything else would be limited by the ISP upload anyway.

chercheur

@user09:

Intel Pentium Processor N3700

X11SBA-LN4F Supermicro

8 GB S0-DDR3

Kingston SV300S37A/60G SSDNow V300 interne SSD-Festplatte 60GB

Hello user09,
Did you make your choice ?