AES-NI performance

RMB

Lanner FW-7525D (Quad-core Atom C2558 @ 2.40GHz)
Shell Output - openssl speed -evp aes-256-cbc -elapsed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 988744 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 926802 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 762164 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 455059 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 93341 aes-256-cbc's in 3.00s
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 5273.30k 19771.78k 65037.99k 155326.81k 254883.16k

Shell Output - openssl speed -evp aes-256-gcm -elapsed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-gcm for 3s on 16 size blocks: 20826334 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 64 size blocks: 8843173 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 256 size blocks: 2794049 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 1024 size blocks: 754329 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 8192 size blocks: 96056 aes-256-gcm's in 3.00s
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-gcm 111073.78k 188654.36k 238425.51k 257477.63k 262296.92k

PfSense SG-2440 (Dual-core Atom C2358 @ 1.74GHz)
Shell Output - openssl speed -evp aes-256-cbc -elapsed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 727986 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 680875 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 557737 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 327133 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 67983 aes-256-cbc's in 3.01s
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 3882.59k 14525.33k 47593.56k 111661.40k 185156.73k

Shell Output - openssl speed -evp aes-256-gcm -elapsed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-gcm for 3s on 16 size blocks: 14925214 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 64 size blocks: 6436982 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 256 size blocks: 2026331 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 1024 size blocks: 549702 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 8192 size blocks: 70004 aes-256-gcm's in 3.00s
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-gcm 79601.14k 137322.28k 172913.58k 187631.62k 191157.59k

aesguy

VAMike,

I am interested in top performance possible for given hardware. "-elapsed" is not useful in that regarding because it measures the typical performance on that box given everything in place. In your case, not only are your tests including things like other applications and processes running (and the swapping in and out of all those processes millions of times per second) but additionally you are running in a VM where typically the operating system itself is given limited access to the underlying hardware resources! Of course your elapsed time ("-elapsed") is going to show drastically slower speeds - because the overhead of the operating system swapping and context switching all those applications and processes AND other operating system instances are all coming into play!! You are not only swapping out all the processes millions of times per second, but in your case, your operating system running in a VM itself is causing overhead! Naturally if your system running other applications AND operating systems then your "-elapsed" is going to show radically different results.

So we're talking apples to oranges. I am interested in the BEST performance that a hardware CAN achieve - not the performance of a particular box as configured and loaded with all sorts of junk - and openssl without "-elapsed" displays that better than with "-elapsed". If you care about maximizing AES performance, then dump all the process and applications running in the background, tune the OS (such as for example highly minimize context switching), build the application for the hardware - and now we're getting closer to achieving maximum performance possible - and this is what openssl without "-elapsed" can give us an idea of today before engaging in investing in a hardware platform. Heck, if one wanted to, you could go further and dump the OS altogether and write an application that boots and runs natively on the hardware itself but in many cases that would be non-trivial costs for that last mile of performance - but it's doable.

You seem to be more interested in performance of A GIVEN SYSTEM built for general purpose - which by the way is perfectly valid and what most pfsense users are after, but it's just not what I'm after in trying to get a good idea of the top performing hardware. Indeed, as you suggest by your tests you are seeing a difference of ~20fold - a big difference and hence why AES-NI offers a big gain in performance - if you can harness it properly.

regarding unloading aesni, try instead invoking openssl with and without "-evp". Like I showed you, the actual OpenSSL source (when invoked using -evp) invokes the actual AES-NI CPU instructions. Then consult the version of openssl you are using and your hardware platform against the source code to see whether the AES-NI instructions are being invoked directly. Here is a link: https://github.com/openssl/openssl/blob/master/crypto/aes/asm/

VAMike

@aesguy:

I am interested in top performance possible for given hardware. "-elapsed" is not useful in that regarding because it measures the typical performance on that box given everything in place.

Well, that's not what you're measuring in your results. You're measuring the time the application spends talking to the kernel to ask for encryption services, and not counting at all the time spent doing encryption. If that's really what you want that's great, but it's a number with no utility whatsoever.

In your case, not only are your tests including things like other applications and processes running (and the swapping in and out of all those processes millions of times per second) but additionally you are running in a VM where typically the operating system itself is given limited access to the underlying hardware resources! Of course your elapsed time ("-elapsed") is going to show drastically slower speeds - because the overhead of the operating system swapping and context switching all those applications and processes AND other operating system instances are all coming into play!! You are not only swapping out all the processes millions of times per second, but in your case, your operating system running in a VM itself is causing overhead! Naturally if your system running other applications AND operating systems then your "-elapsed" is going to show radically different results.

You're simply confused here. The difference isn't system load, it's whether you're actually measuring the time spent doing crypto or not. If you were not using cryptodev then elapsed would be telling you what you think it is–as it is in the cases I showed where the aesni module is unloaded or in the GCM case. Those are valid uses of the defaults, and do show actual cpu time and are worth considering. To put this a different way, -elapsed is usually a less accurate measure, but in the cryptodev case it's the only way to get a measurement that's anywhere close to reality. Unloading the cryptodev aesni module and dropping -elapsed is a better solution (unless you're specifically trying to measure the performance with the aesni module.) Again, if you look at the diagnostic output and see something like "in 0.x seconds", that's bogus because it's not even close to the time spent. (And if you see that output you pretty much know it's a system with cryptodev loaded, it doesn't happen otherwise.) If you see something like "in 2.83s" then you're seeing the benefit of basing the calculation on cpu time, because that's an indication that openssl didn't get a full 3 seconds of cpu time and it's more accurate to base the bandwidth number on the amount of time it actually got. But that cpu time is going to be something pretty close to 3 seconds in any case where basing the calculation on cpu time is remotely valid.

You seem to be more interested in performance of A GIVEN SYSTEM built for general purpose

No, I'm interested in reality. The cryptodev numbers do not reflect reality. Again, look at the number of computations actually being performed, and try to understand what that means. You are looking at bogus numbers because the time used in the calculation simply does not reflect the time the cpu spends doing crypto (because it's being done in kernel space rather than user space). By ignoring the time the CPU spends doing the crypto you aren't getting a better understanding of the performance characteristics of the hardware, you're simply making an incorrect calculation.

which by the way is perfectly valid and what most pfsense users are after, but it's just not what I'm after in trying to get a good idea of the top performing hardware. Indeed, as you suggest by your tests you are seeing a difference of ~20fold - a big difference and hence why AES-NI offers a big gain in performance - if you can harness it properly.

The other thing you seem to not understand is that openssl uses the AES-NI instructions without using cryptodev. If you unload the aesni module on your hardware and run with and without -evp you'll see a significant difference, one that's real rather than one that's an interpretation error, and that difference is the use of the AES-NI instructions. Or focus entirely on the GCM results, which don't get screwed up by cryptodev.

aesguy

You're measuring the time the application spends talking to the kernel to ask for encryption services

OpenSSL isn't "talking to the kernel" per se - but rather the kernel gives the process running openssl small slices of time with the CPU and memory etc and feeds them openssl's instructions. And more importantly, openssl -evp on many architectures supports the native instructions for the specific architecture in use. On those supported architectures, it is NOT "asking for encryption services" - it knows the low-level CPU AES-NI instructions appropriate for that CPU and calls those. This is the 3rd time I'll provide the link to the source code, these are the ARMv8 assembly instructions: https://github.com/openssl/openssl/blob/master/crypto/aes/asm/aesv8-armx.pl

Out of that, you'll see for example the AES CBC encrypt instructions:

$code.=<<___;
	subs	$len,$len,#16
	mov	$step,#16
	b.lo	.Lcbc_abort
	cclr	$step,eq
	cmp	$enc,#0			// en- or decrypting?
	ldr	$rounds,[$key,#240]
	and	$len,$len,#-16
	vld1.8	{$ivec},[$ivp]
	vld1.8	{$dat},[$inp],$step
	vld1.32	{q8-q9},[$key]		// load key schedule...
	sub	$rounds,$rounds,#6
	add	$key_,$key,x5,lsl#4	// pointer to last 7 round keys
	sub	$rounds,$rounds,#2
	vld1.32	{q10-q11},[$key_],#32
	vld1.32	{q12-q13},[$key_],#32
	vld1.32	{q14-q15},[$key_],#32
	vld1.32	{$rndlast},[$key_]
	add	$key_,$key,#32
	mov	$cnt,$rounds
	b.eq	.Lcbc_dec
	cmp	$rounds,#2
	veor	$dat,$dat,$ivec
	veor	$rndzero_n_last,q8,$rndlast
	b.eq	.Lcbc_enc128
	vld1.32	{$in0-$in1},[$key_]
	add	$key_,$key,#16
	add	$key4,$key,#16*4
	add	$key5,$key,#16*5
	aese	$dat,q8
	aesmc	$dat,$dat
	add	$key6,$key,#16*6
	add	$key7,$key,#16*7
	b	.Lenter_cbc_enc

As you can see, these call the ARM AES-NI instruction set (including AESE). There is no "talking to the kernel" other than the kernel swaps the process in and out. Perhaps you are familiar with cryptodev and so believe everything needs to go through that - but it doesn't, look at openssl itself and when -evp is used. The above instructions are where openssl ends up when it detects ARMv8 architecture and then calls ARM CPU instructions directly.

The difference isn't system load, it's whether you're actually measuring the time spent doing crypto or not.

I'm not talking about system load - I'm talking about the necessary overhead that the OS has to do to handle multiple processes from running multiple applications. You don't seem to understand that there is overhead - not only are you measuring with other things running but your tests were also performed on a VM. Hardware resources are fixed and things like VM's are really just software implementations to mimic being able to handle multiple images - but you can't multiply the CPU, the bus, the network, the RAM, etc… Like it or not, but your measurements are being affected by other things running on your system.

If you were not using cryptodev then elapsed would be telling you what you think it is-

Again, on architectures that openssl supports the native AES-NI instructions, -evp does not involve cryptodev. You can dig up what happens on your architecture: https://github.com/openssl/openssl/tree/master/crypto/aes/asm

Perhaps what is confusing you in all this is that there is the cryptodev and the openssl implementations. They are both different, and likely do not support the same architectures.

To put this a different way, -elapsed is usually a less accurate measure, but in the cryptodev case it's the only way to get a measurement that's anywhere close to reality.

In your case, "reality" is the specific system as configured that is being measured. That's what I said last time - but that's different from the maximum top performance which is what I'm trying to gauge.

Unloading the cryptodev aesni module and dropping -elapsed is a better solution (unless you're specifically trying to measure the performance with the aesni module.)

Yes getting closer! That's what we're trying to measure - the top performance possible for a specific hardware. Even if that's not going to happen because say other applications need to run (as is the case in many deployed pfsense installations that run multiple functions).

You are looking at bogus numbers because the time used in the calculation simply does not reflect the time the cpu spends doing crypto (because it's being done in kernel space rather than user space).

Again, I believe you believe cryptodev is involved. And on certain architectures, openssl implements the native AES-NI instruction set. On Intel, those are AESENC, AESENCLAST, etc… - there's 7 of them, but on ARM they're AESE, etc... Completely different CPU's, completely different instruction sets. Really nothing to do with kernel - the kernel is there to provide operating system functionality, but in supported CPUs there's no "encryption services".

The other thing you seem to not understand is that openssl uses the AES-NI instructions without using cryptodev.

Wrong. Look at the openssl source code. On supported architectures, the AES-NI instructions are invoked directly.

VAMike

@aesguy:

You're measuring the time the application spends talking to the kernel to ask for encryption services

OpenSSL isn't "talking to the kernel" per se - but rather the kernel gives the process running openssl small slices of time with the CPU and memory etc and feeds them openssl's instructions.

No, you're quite simply wrong. openssl on pfsense opens /dev/crypto and offloads the AES CBC routines if a kernel module supporting AES CBC is loaded. That's why the performance changes so dramatically when you load and unload that module. That's why the performance is so bad on small blocks, it has to send each block up through the kernel and back. I even walked through step by step how to demonstrate that.

And more importantly, openssl -evp on many architectures supports the native instructions for the specific architecture in use.

Yes, except that the implementation on pfsense will ignore the built-in routines in the presence of a /dev/crypto that reports AES capability. "openssl engine -t -c" will show whether openssl has detected such.

If you were not using cryptodev then elapsed would be telling you what you think it is-

Again, on architectures that openssl supports the native AES-NI instructions, -evp does not involve cryptodev.

If that were true, then unloading the module would have no effect, right? But it does. "truss openssl speed -evp aes-128-cbc |& grep /dev/crypto". You can also take a look at the number of ioctls with and without the cryptodev stuff to see openssl passing the blocks off to the kernel one by one. (grep ioctl instead of grep /dev/crypto and be prepared for a lot of output) https://github.com/pfsense/FreeBSD-src/blob/devel/crypto/openssl/crypto/engine/eng_cryptodev.c is the code implementing cryptodev support, including support for evp mode (look for the "EVP" functions).

aesguy

Unfortunately we've strayed far from the overall goal - to roughly measure top performance possible of various AES-NI implementations. You are missing the point that I want to roughly measure this. I don't care about -elapsed simply because it adds in various other factors which distorts or does not give good idea of the maximum performance possible. For example:

openssl on pfsense opens /dev/crypto and offloads the AES CBC routines if a kernel module supporting AES CBC is loaded. That's why the performance changes so dramatically when you load and unload that module. That's why the performance is so bad on small blocks, it has to send each block up through the kernel and back.

I'm not interested in various factors due to a kernel implementation or design of cryptodev. The fact that data is being copied back and forth from userspace to kernel affects performance in non-trivial amounts. Who says everything has to be the cryptodev-way? And in that respect, openssl without "-elapsed" offers a much better indication of maximum possible performance for a particular piece of hardware.

aesguy

VAMike,

You've provided several AES-128-CBC results but any chance you can provide the results for AES-256-CBC (along with box info)?

openssl speed -evp aes-256-cbc

aesguy

RMB, thanks for the info but unfortunately run with "-elapsed", can you rerun without "-elapsed"?

openssl speed -evp aes-256-cbc

aesguy

Updated table for "openssl speed -evp aes-256-cbc":

8192BYTES	BOX	CPU	USERNAME	LINK
170926276.61k	unknown (China)	gen 5 i5	Koenig	
150749577.22k	Microserver Gen 8	ESXi 6.0	biggsy	
91090845.70k	Zotac ZBOX ID92	Core i5 4570T	highwire	
48454172.67k	SuperMicro Board: X11SBA-LN4F	Intel N3700	Engineer	
48351936.51k	SuperMicro 2758	Intel(R) Atom(TM) CPU C2758 @ 2.40GHz 8 CPUs	AR15USR	
42008576.00k	Gigabyte GA-N3150N-D3V board	Celeron N3150 with AES-NI		https://forum.pfsense.org/index.php?topic=108119.0
32321306.62k	SuperMicro 2758	Intel(R) Atom(TM) CPU C2758 @ 2.40GHz 8 CPUs	AR15USR	
32267479.72k	Supermicro	Intel N3700	Engineer	
29080158.21k	hp microserver gen 8	Xeon 1265Lv2	iorx	
27986842.97k	Gigabyte GA-N3150N-D3V	Celeron N3150 with AES-NI		https://forum.pfsense.org/index.php?topic=105114.msg601520#msg601520
24435715.51k	unknown (China)	gen 5 i5	Koenig	
24345837.57k	Lanner FW-7525D	Quad-core Atom C2558 @ 2.40GHz	RMB	
24332468.22k	Netgate SG-4860  	Intel(R) Atom(TM) CPU C2558 @ 2.40GHz 4 CPUs	bytesizedalex	
19462619.14k	SuperMicro 2758	Intel(R) Atom(TM) CPU C2758 @ 2.40GHz 8 CPUs	AR15USR	
18390712.32k	AM1	Athlon 5370	W4RH34D	
14241549.52k	pfSense SG-2440	Dual-core Atom C2358 @ 1.74GHz	RMB	
7123763.20k	Raspberry Pi 3	ARMv7l		
405686.95k	Intel i7-4510U + 2x Intel 82574 + 2x Intel i350 Mini-ITX Build			https://forum.pfsense.org/index.php?topic=115627.msg646395#msg646395
230708.57k	ci323 nano u	Celeron N3150 with AES-NI w/ -engine cryptodev		https://forum.pfsense.org/index.php?topic=115673.msg656602#msg656602
217617.75k	RCC-VE 2440	Intel Atom C2358		https://forum.pfsense.org/index.php?topic=91974.0
124788.74k	ALIX.APU2B4/APU2C4	1 GHz Quad Core AMD GX-412TC		http://wiki.ipfire.org/en/hardware/pcengines/apu2b4
34204.33k	ALIX.APU1C/APU1D	1 GHz Dual Core AMD G-T40E		http://wiki.ipfire.org/en/hardware/pcengines/apu1c

gjaltemba

Platform pfSense ESXI 6.0 VM
Version 2.3.2-RELEASE-p1 (amd64)
built on Tue Sep 27 12:13:07 CDT 2016
FreeBSD 10.3-RELEASE-p9

CPU Type Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
2 CPUs: 1 package(s) x 2 core(s)

Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM

Shell Output - openssl speed -evp aes-256-cbc

Doing aes-256-cbc for 3s on 16 size blocks: 1019263 aes-256-cbc's in 0.31s
Doing aes-256-cbc for 3s on 64 size blocks: 959341 aes-256-cbc's in 0.29s
Doing aes-256-cbc for 3s on 256 size blocks: 779985 aes-256-cbc's in 0.22s
Doing aes-256-cbc for 3s on 1024 size blocks: 437868 aes-256-cbc's in 0.18s
Doing aes-256-cbc for 3s on 8192 size blocks: 88484 aes-256-cbc's in 0.01s
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 52186.27k 212403.28k 912805.30k 2495314.54k 92782198.78k

VAMike

@aesguy:

Unfortunately we've strayed far from the overall goal - to roughly measure top performance possible of various AES-NI implementations. You are missing the point that I want to roughly measure this.

You're not roughly measuring this, you're measuring the wrong thing (basically you're measuring how many syscalls/second you can execute, which is completely unrelated to how fast you can process crypto). The hardware in question IS NOT CAPABLE OF THE SPEEDS YOU ARE DISCUSSING. Using cryptodev without -elapsed is simply wrong. If you take cryptodev out of the equation then you can drop -elapsed and get meaningful results.

You should really run this by an openssl developer if you won't believe me. When they stop laughing at the idea of a raspberry pi getting 7GByte/s AES 256 CBC (the hardware is capable of something around 40Mbyte/s) they'll tell you what I explained above.

You're clearly not going to actually consider this, but I hope whoever else reads this and sees the fantastical speed test results understands that they're bogus.

Side note: anyone who's getting these crazy results (the 0.whatever seconds) would be well served by turning off aesni.ko in their config unless they're primarily interested in ipsec performance–it's actually slowing down openvpn and anything else that uses openssl. Pippin did a nice writeup a couple of months ago walking through it: https://forum.pfsense.org/index.php?topic=115627.msg646775#msg646775
What pfsense should really be doing is making aesni.ko and cryptodev.ko two separate items. For good kernel ipsec performance you want aesni.ko (that will implement AES-NI in the kernel) without cryptodev.ko (which makes openssl stop using its internal routines in favor of the less efficient kernel syscall implementation). The only case where cryptodev actually makes sense is if you're using off-cpu acceleration like the old hifn or padlock stuff or quickassist. (But in that case you'll still want -elapsed to measure real throughput--the rule is, always use -elapsed with /dev/crypto.)

aesguy

When they stop laughing at the idea of a raspberry pi getting 7GByte/s AES 256 CBC (the hardware is capable of something around 40Mbyte/s) they'll tell you what I explained above.

Who ever said anything about 7GB/s?! We're comparing the relative performance of different AES-NI hardware implementations.

VAMike

@aesguy:

Who ever said anything about 7GB/s?! We're comparing the relative performance of different AES-NI hardware implementations.

No, you aren't. You're comparing their context switching rates, which has nothing to do with their crypto processing rates. You can't rationalize this into something positive. And the icing on the cake is that when people post the right numbers you turn them down because they don't fit your misconceptions.

Edit to add: actually, it's worse than that–given two cpus that are otherwise equal, this methodology will actually penalize the one with the more efficient crypto implementation (because it will spend relatively less time doing crypto in kernel space where the time isn't counted and more time in user space doing context switches, which are the only time counted).

aesguy

VAMike, the keyword is "relative" - and in this context refers to comparing results from different hardware.

VAMike

@aesguy:

VAMike, the keyword is "relative" - and in this context refers to comparing results from different hardware.

Again, the words put together don't make any sense. Why would you reject solid data in favor of bogus data, it's not like it's any harder to gather. If you collected the real numbers, then you'd have an actual "relative" comparison rather than an "irrelevant" comparison. Is it really that difficult to just admit you were wrong and move on?

aesguy

Again, the words put together don't make any sense.

Of course they don't make any sense to you - because you're either not reading or not trying to understand what I'm saying.

Guest

@Engineer

SuperMicro Board: X11SBA-LN4F with Intel N3700.

Why you are not using the IPSec VPN together with AES-GCM with that CPU?

@aesguy

…a big difference and hence why AES-NI offers a big gain in performance - if you can harness it properly.

Another member of this forum was posting in on reddit that he got a real throughput of nearly ~500 MBit/s
together with IPSec VPN over AES-GCM, based on a pfSense SG-4860 on a 1 GBit/s Internet connection.

@VAMike
I consider to the circumstance that a real life VPN connection is better then all the testing runs on a bare
hardware machine. What I can get out from a device is not able to test on that device alone and only over
an OpenSSL test, OpenSSL is multi core using and the OpenVPN part isn´t tight now using that.

Also talking over crypto cards such the soekris vpn14x1 is today a little bit outdated, with an viewing eyes
on the todays Internet speed. But in the past getting instead of ~14 MBit/s without it and then ~42 MBit/s
using it (vpn1411) was really impressive for me on a net5501 or an Alix Board. It was nearly the 3x speed!

If I am using site-2-side VPN I only use IPSec with AES-GCM.

VAMike

@BlueKobold:

I consider to the circumstance that a real life VPN connection is better then all the testing runs on a bare
hardware machine. What I can get out from a device is not able to test on that device alone and only over
an OpenSSL test

Certainly it makes sense to optimize for the actual application. That said, running the openssl speed routine will give a ceiling for your performance. If you need to get N and openssl speed (the real results, not the meaningless /dev/crypto ones without -elapsed) says your hardware delivers N/2, no amount of tweaking is going to get you the results you need. The results (again, the real ones) are also useful to compare hardware: you may find (this is really a thing) that at a given price point three different systems have order of magnitude differences in their crypto processing rate–that's valuable information that's definitely worth knowing if crypto processing is a factor in choosing a solution. Actual VPN throughput would be a better basis for comparison, but that's much more configuration dependent and hard to communicate as a single repeatable value that you can ask someone for.

OpenSSL is multi core using and the OpenVPN part isn´t tight now using that.

The openssl speed routine is single threaded. If you add the -multi N parameter with a new enough version it will launch N single threaded processes and combine the results.

Also talking over crypto cards such the soekris vpn14x1 is today a little bit outdated, with an viewing eyes
on the todays Internet speed. But in the past getting instead of ~14 MBit/s without it and then ~42 MBit/s
using it (vpn1411) was really impressive for me on a net5501 or an Alix Board. It was nearly the 3x speed!
If I am using site-2-side VPN I only use IPSec with AES-GCM.

If 42Mbit/s is acceptable for you, then you're golden. Almost anything modern will run rings around that, though, without the vpn card. You're right that AES GCM is generally a winner. People sometimes compare it to AES CBC and get disappointed, but the proper comparison is to AES CBC + SHA HMAC (because GCM includes MAC) and that changes things, especially on the lower end where GCM isn't optimized as well as it is on the better architectures so the comparison between GCM and CBC without HMAC looks worse:

(GX-412TC / APU2 @1GHZ)
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-128-cbc 129634.49k 180885.10k 218398.55k 230835.20k 233439.23k
aes-128-cbc-hmac-sha1 37836.21k 57600.58k 64128.26k 76066.47k 81273.41k
aes-128-gcm 66775.78k 171264.79k 256270.08k 293397.23k 304955.39k

(silvermont @2.4GHz)
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-128-cbc 212028.47k 324202.84k 387726.68k 407513.09k 412789.42k
aes-128-cbc-hmac-sha1 82160.17k 127661.48k 144141.78k 152486.91k 155320.32k
aes-128-gcm 127695.16k 218440.90k 280572.06k 304679.94k 310804.48k

(sandy bridge @2.5GHz)
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-128-cbc 466082.23k 481016.45k 527165.95k 575894.53k 572252.16k
aes-128-cbc-hmac-sha1 177098.15k 224449.86k 325795.96k 379470.51k 400328.52k
aes-128-gcm 237922.04k 572122.82k 761623.45k 835320.15k 919997.10k

(haswell @2.6GHz)
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-128-cbc 617610.13k 704447.23k 724869.63k 723632.81k 718821.97k
aes-128-cbc-hmac-sha1 214205.59k 341683.78k 514962.58k 617723.90k 656337.58k
aes-128-gcm 422036.43k 1069918.31k 1470884.44k 1609671.68k 1635520.47k

(skylake @3.7GHz)
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
aes-128-cbc 919971.09k 1366752.68k 1394404.61k 1400528.21k 1400209.41k 1400182.10k
aes-128-cbc-hmac-sha1 335511.72k 513717.82k 675188.48k 804944.90k 874438.66k 882191.02k
aes-128-gcm 643566.37k 1481056.34k 2880229.72k 4531479.55k 5638567.25k 5748069.72k

You can see the silvermont GCM is actually slower than CBC, but ~2x CBC+HMAC. And for skylake the difference between CBC & GCM is huge and CBC+HMAC is just blown away. You can also see where the relatively inefficient PCLMULQDQ implementation hurts silvermont, to the point that an APU2 running at half the speed is actually competitive (real world VPN performance won't be nearly as close because the overall platform is much slower, and the small block results highlight the difference in a case where the crypto instructions have less room to run). And you can see the really impressive improvements intel has made over the past few years from sandy bridge to haswell to skylake. N.b., I didn't make any attempt to quiesce the systems or do real multi-trial benchmarking, but the numbers should be within about 20% or so across platforms and pretty consistent within a platform–certainly good enough for the discussion. It's also worth noting those are fairly recent versions of openssl, and older versions don't implement the CBC+HMAC EVP mode (so don't try to compare apples and oranges).

MoonKnight

Here is my test. pfsense spec is in the signature :)

[2.3.2-RELEASE][root@pfsense.local]/root: openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 1697468 aes-256-cbc's in 0.23s
Doing aes-256-cbc for 3s on 64 size blocks: 1735785 aes-256-cbc's in 0.27s
Doing aes-256-cbc for 3s on 256 size blocks: 1514519 aes-256-cbc's in 0.28s
Doing aes-256-cbc for 3s on 1024 size blocks: 1025506 aes-256-cbc's in 0.22s
Doing aes-256-cbc for 3s on 8192 size blocks: 253309 aes-256-cbc's in 0.05s
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 115880.48k 418222.08k 1378548.85k 4800540.09k 37944819.71k

VAMike

@CiscoX:

Here is my test. pfsense spec is in the signature :)

[2.3.2-RELEASE][root@pfsense.local]/root: openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 1697468 aes-256-cbc's in 0.23s
Doing aes-256-cbc for 3s on 64 size blocks: 1735785 aes-256-cbc's in 0.27s
Doing aes-256-cbc for 3s on 256 size blocks: 1514519 aes-256-cbc's in 0.28s
Doing aes-256-cbc for 3s on 1024 size blocks: 1025506 aes-256-cbc's in 0.22s
Doing aes-256-cbc for 3s on 8192 size blocks: 253309 aes-256-cbc's in 0.05s
OpenSSL 1.0.1s-freebsd 1 Mar 2016
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 115880.48k 418222.08k 1378548.85k 4800540.09k 37944819.71k

So the real number is 691702, which is actually a bit low for a skylake @2.7GHz.