AES-NI performance

iorx

Hi!

For fun or reference :). A Hyper-v hosted pfsense on a hp microserver gen 8 with a Xeon 1265Lv2.


[2.3.2-RELEASE][n23]/root: openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 1084848 aes-256-cbc's in 0.45s
Doing aes-256-cbc for 3s on 64 size blocks: 1345250 aes-256-cbc's in 0.24s
Doing aes-256-cbc for 3s on 256 size blocks: 709374 aes-256-cbc's in 0.23s
Doing aes-256-cbc for 3s on 1024 size blocks: 472042 aes-256-cbc's in 0.19s
Doing aes-256-cbc for 3s on 8192 size blocks: 110932 aes-256-cbc's in 0.03s
OpenSSL 1.0.1s-freebsd  1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc      38978.40k   355493.16k   774825.57k  2577978.71k 29080158.21k

AR15USR

This is from a connection over wifi, not sure if that makes a difference.

On a SuperMicro 2758:

openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 982926 aes-256-cbc's in 0.38s
Doing aes-256-cbc for 3s on 64 size blocks: 921181 aes-256-cbc's in 0.27s
Doing aes-256-cbc for 3s on 256 size blocks: 761431 aes-256-cbc's in 0.33s
Doing aes-256-cbc for 3s on 1024 size blocks: 448646 aes-256-cbc's in 0.19s
Doing aes-256-cbc for 3s on 8192 size blocks: 92805 aes-256-cbc's in 0.04s
OpenSSL 1.0.1s-freebsd  1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) 
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc      41938.18k   215608.99k   594061.21k  2450205.35k 19462619.14k

With the aesni turned off in the advanced settings:

openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 955806 aes-256-cbc's in 0.29s
Doing aes-256-cbc for 3s on 64 size blocks: 909612 aes-256-cbc's in 0.26s
Doing aes-256-cbc for 3s on 256 size blocks: 758911 aes-256-cbc's in 0.30s
Doing aes-256-cbc for 3s on 1024 size blocks: 446740 aes-256-cbc's in 0.13s
Doing aes-256-cbc for 3s on 8192 size blocks: 92400 aes-256-cbc's in 0.04s
OpenSSL 1.0.1s-freebsd  1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) 
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc      52905.15k   225804.29k   654420.94k  3659694.08k 19377684.48k

highwire

@aesguy:

highwire - 91090845.70k is the fastest we've seen on this forum!

That's a ZOTAC ZBox ID92 right? Quite a bit more pricey than Raspberry Pi 3 but also better AES performance.

Yes, an ID92. Quite pricey (I bought mine on sale, but still). I bought it for a HTPC but abandoned that plan and re purposed it. It is very much overkill for this application (even running a VPN server) as my connection is only 100mbps/10mbps.

W4RH34D

I was thinking of firing up the 6 core xeon but I just don't really care for epeen stuff anymore. I mean if someone needs to see it I'll do it, no time for "just for grins" these days.

Koenig

I have a chinese "mini-computer" (gen 5 i5)

I did 2 test and got a very varying result:

Try one:
Doing aes-256-cbc for 3s on 16 size blocks: 1704782 aes-256-cbc's in 0.28s
Doing aes-256-cbc for 3s on 64 size blocks: 1762586 aes-256-cbc's in 0.31s
Doing aes-256-cbc for 3s on 256 size blocks: 1417931 aes-256-cbc's in 0.32s
Doing aes-256-cbc for 3s on 1024 size blocks: 811284 aes-256-cbc's in 0.13s
Doing aes-256-cbc for 3s on 8192 size blocks: 163126 aes-256-cbc's in 0.05s
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 96983.15k 360977.61k 1133238.12k 6646038.53k 24435715.51k

Try two:
Doing aes-256-cbc for 3s on 16 size blocks: 1727740 aes-256-cbc's in 0.41s
Doing aes-256-cbc for 3s on 64 size blocks: 1742973 aes-256-cbc's in 0.38s
Doing aes-256-cbc for 3s on 256 size blocks: 1414059 aes-256-cbc's in 0.29s
Doing aes-256-cbc for 3s on 1024 size blocks: 815243 aes-256-cbc's in 0.13s
Doing aes-256-cbc for 3s on 8192 size blocks: 163008 aes-256-cbc's in 0.01s
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 68046.38k 291396.63k 1252321.22k 6285619.44k 170926276.61k

aesguy

Thanks all for the results, keep them coming!

Here are the results so far:

170926276.61k		gen 5 i5
91090845.70k	Zotac ZBOX ID92	Core i5 4570T
29080158.21k	hp microserver gen 8	Xeon 1265Lv2
24435715.51k		gen 5 i5
24345837.57k	Lanner FW-7525D	Quad-core Atom C2558 @ 2.40GHz
19462619.14k		SuperMicro 2758
18390712.32k	AM1	Athlon 5370
14241549.52k	pfSense SG-2440	Dual-core Atom C2358 @ 1.74GHz
7123763.20k	Raspberry Pi 3	ARMv7l

aesguy

AR15USR,

There doesn't seem any difference in your tests. Can you try running without the "-evp" option?

openssl speed aes-256-cbc

aesguy

Koenig,

Can you provide the make and model of your "gen 5 i5"?

aesguy

Here's an updated list of results:

170926276.61k		gen 5 i5	
91090845.70k	Zotac ZBOX ID92	Core i5 4570T	
42008576.00k	Gigabyte GA-N3150N-D3V board	Celeron N3150 with AES-NI	https://forum.pfsense.org/index.php?topic=108119.0
29080158.21k	hp microserver gen 8	Xeon 1265Lv2	
27986842.97k	Gigabyte GA-N3150N-D3V	Celeron N3150 with AES-NI	https://forum.pfsense.org/index.php?topic=105114.msg601520#msg601520
24435715.51k		gen 5 i5	
24345837.57k	Lanner FW-7525D	Quad-core Atom C2558 @ 2.40GHz	
19462619.14k		SuperMicro 2758	
18390712.32k	AM1	Athlon 5370	
14241549.52k	pfSense SG-2440	Dual-core Atom C2358 @ 1.74GHz	
7123763.20k	Raspberry Pi 3	ARMv7l	
405686.95k	Intel i7-4510U + 2x Intel 82574 + 2x Intel i350 Mini-ITX Build		https://forum.pfsense.org/index.php?topic=115627.msg646395#msg646395
230708.57k	ci323 nano u	Celeron N3150 with AES-NI w/ -engine cryptodev	https://forum.pfsense.org/index.php?topic=115673.msg656602#msg656602
217617.75k	RCC-VE 2440	Intel Atom C2358	https://forum.pfsense.org/index.php?topic=91974.0
124788.74k	ALIX.APU2B4/APU2C4	1 GHz Quad Core AMD GX-412TC	http://wiki.ipfire.org/en/hardware/pcengines/apu2b4
34204.33k	ALIX.APU1C/APU1D	1 GHz Dual Core AMD G-T40E	http://wiki.ipfire.org/en/hardware/pcengines/apu1c

Koenig

@aesguy:

Koenig,

Can you provide the make and model of your "gen 5 i5"?

There's no brand or model on it…

Something like this: https://www.aliexpress.com/item/Fanless-PC-Intel-NUC-Core-i7-5500u-i5-5257u-Iris-6100-Barebone-Mini-PC-Windows-2HDMI/32755490163.html?spm=2114.01010208.3.100.Dtd346&ws_ab_test=searchweb0_0,searchweb201602_2_10091_10090_10088_10089,searchweb201603_1&btsid=6d47dcd0-df75-47e8-84cf-86813f160f8e

Some more results:

aes-256-cbc 99810.65k 375805.41k 1454872.58k 4844784.55k 28507460.95k

aes-256-cbc 62518.77k 350371.84k 1217122.52k 5055197.38k 34182738.74k

aes-256-cbc 76404.78k 341786.43k 1224697.10k 4425564.16k 34284240.90k

aes-256-cbc 91091.47k 242748.12k 1191453.72k 5068092.37k 85483061.25k

aes-256-cbc 100148.30k 299186.69k 1330803.04k 6668591.10k 86076555.26k

aes-256-cbc 105877.45k 377916.58k 1538361.48k 6694084.61k 57179897.86k

aes-256-cbc 84355.12k 320069.81k 1420017.17k 6647087.10k 57598978.73k

aes-256-cbc 106102.67k 260300.35k 1792681.83k 9638188.87k 34206646.27k

All from the same machine.

Derelict

14241549.52k pfSense SG-2440 Dual-core Atom C2358 @ 1.74GHz
217617.75k RCC-VE 2440 Intel Atom C2358 https://forum.pfsense.org/index.php?topic=91974.0

Obviously something off there.

bytesizedalex

First my system details -

System: Netgate SG-4860
Version: 2.3.2-RELEASE-p1 (amd64) built on Fri Sep 30 14:36:56 CDT 2016 FreeBSD 10.3-RELEASE-p9
CPU Type: Intel(R) Atom(TM) CPU C2558 @ 2.40GHz 4 CPUs: 1 package(s) x 4 core(s)
Hardware crypto: AES-CBC,AES-XTS,AES-GCM,AES-ICM

Results (system pretty active so possibility for skewed results) -

[2.3.2-RELEASE][admin@pfSense.localdomain]/root: openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 984814 aes-256-cbc's in 0.35s
Doing aes-256-cbc for 3s on 64 size blocks: 920037 aes-256-cbc's in 0.30s
Doing aes-256-cbc for 3s on 256 size blocks: 759776 aes-256-cbc's in 0.26s
Doing aes-256-cbc for 3s on 1024 size blocks: 452100 aes-256-cbc's in 0.15s
Doing aes-256-cbc for 3s on 8192 size blocks: 92821 aes-256-cbc's in 0.03s
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 44819.98k 193254.95k 754434.54k 3118823.75k 24332468.22k

AR15USR

@aesguy:

AR15USR,

There doesn't seem any difference in your tests. Can you try running without the "-evp" option?
openssl speed aes-256-cbc

/root: openssl speed aes-256-cbc
Doing aes-256 cbc for 3s on 16 size blocks: 5517180 aes-256 cbc's in 3.01s
Doing aes-256 cbc for 3s on 64 size blocks: 1544753 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 256 size blocks: 399657 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 1024 size blocks: 258521 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 8192 size blocks: 32712 aes-256 cbc's in 2.99s
OpenSSL 1.0.1s-freebsd  1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) 
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256 cbc      29348.53k    32954.73k    34104.06k    88241.83k    89558.79k

For comparison:

/root: openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 957210 aes-256-cbc's in 0.39s
Doing aes-256-cbc for 3s on 64 size blocks: 893869 aes-256-cbc's in 0.24s
Doing aes-256-cbc for 3s on 256 size blocks: 751299 aes-256-cbc's in 0.27s
Doing aes-256-cbc for 3s on 1024 size blocks: 450002 aes-256-cbc's in 0.10s
Doing aes-256-cbc for 3s on 8192 size blocks: 92472 aes-256-cbc's in 0.02s
OpenSSL 1.0.1s-freebsd  1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) 
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc      39207.32k   236212.09k   724075.46k  4537127.86k 32321306.62k

damir

I have no idea what it means, and how good or bad output is, as i do not understand this, but, i thought lets try on my box :)

any good?

[2.3.2-RELEASE][admin@pfSense]/root: openssl speed aes-256-cbc
Doing aes-256 cbc for 3s on 16 size blocks: 12830479 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 64 size blocks: 3389641 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 256 size blocks: 
858407 aes-256 cbc's in 3.00s
Doing aes-256 cbc for 3s on 1024 size blocks: 217919 aes-256 cbc's in 3.03s
Doing aes-256 cbc for 3s on 8192 size blocks: 27176 aes-256 cbc's in 3.02s
OpenSSL 1.0.1s-freebsd  1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) 
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256 cbc      68429.22k    72312.34k    73250.73k    73616.18k    73633.34k


[2.3.2-RELEASE][admin@pfSense]/root:  openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 77185949 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 20190084 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 5139740 aes-256-cbc's in 3.02s
Doing aes-256-cbc for 3s on 1024 size blocks: 1286608 aes-256-cbc's in 3.02s
Doing aes-256-cbc for 3s on 8192 size blocks: 160088 aes-256-cbc's in 3.00s
OpenSSL 1.0.1s-freebsd  1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx) 
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc     411658.39k   430721.79k   435191.22k   436886.75k   437146.97k

tnx

Derelict

~~Means you don't have AES-NI or it is disabled or ?

The first 3 secs indicates clock time. The second time interval indicates CPU time. Note that on the accelerated systems they are performing operations on more data in < 1/10 the CPU time.~~ Don't listen to that guy.

Engineer

SuperMicro with Intel N3700. Not bad for a 6W CPU (System pulls 11 Watts from the wall).

$ openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 991459 aes-256-cbc's in 0.25s
Doing aes-256-cbc for 3s on 64 size blocks: 971848 aes-256-cbc's in 0.26s
Doing aes-256-cbc for 3s on 256 size blocks: 785303 aes-256-cbc's in 0.28s
Doing aes-256-cbc for 3s on 1024 size blocks: 393543 aes-256-cbc's in 0.16s
Doing aes-256-cbc for 3s on 8192 size blocks: 92318 aes-256-cbc's in 0.02s
OpenSSL 1.0.1l-freebsd 15 Jan 2015
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 63453.38k 241253.90k 714800.24k 2579123.40k 32267479.72k

aesguy

@Koenig: thanks, I've labelled it as Unknown(China)

@Derelict: I'm taking an indiscriminate method and keeping all data points provided. It might be due to the OS version, random timing, etc.

@bytesizedalex: thanks and added to the list

@AR15USR: as I suspected - OpenSSL -evp determines itself whether AES-NI is present and uses it - doesn't matter what you set in pfsense.

@NEK4TE: can you provide what your box & CPU are?

@Engineer: which Supermicro box is it?

aesguy

Updated results list:

170926276.61k	unknown (China)	gen 5 i5	
91090845.70k	Zotac ZBOX ID92	Core i5 4570T	
42008576.00k	Gigabyte GA-N3150N-D3V board	Celeron N3150 with AES-NI	https://forum.pfsense.org/index.php?topic=108119.0
32321306.62k	SuperMicro 2758		
32267479.72k	Supermicro	Intel N3700	
29080158.21k	hp microserver gen 8	Xeon 1265Lv2	
27986842.97k	Gigabyte GA-N3150N-D3V	Celeron N3150 with AES-NI	https://forum.pfsense.org/index.php?topic=105114.msg601520#msg601520
24435715.51k	unknown (China)	gen 5 i5	
24345837.57k	Lanner FW-7525D	Quad-core Atom C2558 @ 2.40GHz	
24332468.22k	Netgate SG-4860  	Intel(R) Atom(TM) CPU C2558 @ 2.40GHz 4 CPUs	
19462619.14k	SuperMicro 2758		
18390712.32k	AM1	Athlon 5370	
14241549.52k	pfSense SG-2440	Dual-core Atom C2358 @ 1.74GHz	
7123763.20k	Raspberry Pi 3	ARMv7l	
405686.95k	Intel i7-4510U + 2x Intel 82574 + 2x Intel i350 Mini-ITX Build		https://forum.pfsense.org/index.php?topic=115627.msg646395#msg646395
230708.57k	ci323 nano u	Celeron N3150 with AES-NI w/ -engine cryptodev	https://forum.pfsense.org/index.php?topic=115673.msg656602#msg656602
217617.75k	RCC-VE 2440	Intel Atom C2358	https://forum.pfsense.org/index.php?topic=91974.0
124788.74k	ALIX.APU2B4/APU2C4	1 GHz Quad Core AMD GX-412TC	http://wiki.ipfire.org/en/hardware/pcengines/apu2b4
34204.33k	ALIX.APU1C/APU1D	1 GHz Dual Core AMD G-T40E	http://wiki.ipfire.org/en/hardware/pcengines/apu1c

biggsy

iorx,

Interested to see you're running the same processor in a Microserver Gen 8 as I do.

Mine is running ESXi 6.0 though and produces slightly different numbers:

[2.3.2-RELEASE] /root: openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 1767436 aes-256-cbc's in 0.38s
Doing aes-256-cbc for 3s on 64 size blocks: 1616969 aes-256-cbc's in 0.35s
Doing aes-256-cbc for 3s on 256 size blocks: 1308617 aes-256-cbc's in 0.27s
Doing aes-256-cbc for 3s on 1024 size blocks: 723750 aes-256-cbc's in 0.13s
Doing aes-256-cbc for 3s on 8192 size blocks: 143766 aes-256-cbc's in 0.01s
OpenSSL 1.0.1s-freebsd  1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc      75410.60k   294360.22k  1225164.62k  5580197.65k 150749577.22k

@iorx:

Hi!

For fun or reference :). A Hyper-v hosted pfsense on a hp microserver gen 8 with a Xeon 1265Lv2.


[2.3.2-RELEASE][n23]/root: openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 1084848 aes-256-cbc's in 0.45s
Doing aes-256-cbc for 3s on 64 size blocks: 1345250 aes-256-cbc's in 0.24s
Doing aes-256-cbc for 3s on 256 size blocks: 709374 aes-256-cbc's in 0.23s
Doing aes-256-cbc for 3s on 1024 size blocks: 472042 aes-256-cbc's in 0.19s
Doing aes-256-cbc for 3s on 8192 size blocks: 110932 aes-256-cbc's in 0.03s
OpenSSL 1.0.1s-freebsd  1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc      38978.40k   355493.16k   774825.57k  2577978.71k 29080158.21k

Engineer

@aesguy:

@Engineer: which Supermicro box is it?

SuperMicro Board: X11SBA-LN4F with Intel N3700.

Running 2.2.5 and whatever FreeBSD version comes with it but not sure if there have been improvements in the newer versions or not.

Just re-ran the test with nobody using the Internet (wife and two kids on Facebook, snapchat, youtube, etc. really change the results) and got this….

$ openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 951002 aes-256-cbc's in 0.28s
Doing aes-256-cbc for 3s on 64 size blocks: 961593 aes-256-cbc's in 0.26s
Doing aes-256-cbc for 3s on 256 size blocks: 770095 aes-256-cbc's in 0.23s
Doing aes-256-cbc for 3s on 1024 size blocks: 454015 aes-256-cbc's in 0.14s
Doing aes-256-cbc for 3s on 8192 size blocks: 92419 aes-256-cbc's in 0.02s
OpenSSL 1.0.1l-freebsd 15 Jan 2015
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 54101.45k 238708.18k 870154.24k 3306036.34k 48454172.67k

Makes more sense compared to the N3150 in the chart now.