OpenSSL not loading full SafeXcel capabilities.

JonathanLee

@stephenw10 said in OpenSSL not loading full SafeXcel capabilities.:

cryptodev

Can you use cryptodev with ssl_engine in squid at the same time?

JonathanLee

@JonathanLee I take that back only doing tests with cbc increment counters

stephenw10

armv8crypto is more like aesni for ARM CPUs. It's not safeXcel.

When you run openssl with an engine selected it will use software for algorithms not supported by the 'engine'.

The cryptodev framework can only be used by kernel space code. devcrypto existed to allow it to be used by userspace but is much slower and, as I understand it, not maintained which is why it stopped working.

You should use -elapsed with openssl speed to see the actual time taken and not just CPU time. Othewise, when an actual hardware engine is used, you see very small times.

If you're using DCO in OpenVPN you should not use devcrypto. DCO will use whatever is registered by the cryptodev framework and that should be safexcel.

I will say there's a bunch of confusion here. Still. As I understand it the devcrypto part hasn't worked as I expected since FreeBSD11, so 2.4.X.
Let me re-dig into it....

JonathanLee

@stephenw10 said in OpenSSL not loading full SafeXcel capabilities.:

cryptodev

Thanks for the reply,

I cant do any speed test with that engine.

Shell Output - openssl speed -evp aes-256-gcm -engine cryptodev
invalid engine "cryptodev"
111773477724160:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/engines/cryptodev.so): Cannot open "/usr/lib/engines/cryptodev.so"
111773477724160:error:25070067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_lib.c:162:
111773477724160:error:260B6084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_dyn.c:434:
111773477724160:error:2606A074:engine routines:ENGINE_by_id:no such engine:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_list.c:421:id=cryptodev
111773477724160:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(libcryptodev.so): Shared object "libcryptodev.so" not found, required by "openssl"
111773477724160:error:25070067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_lib.c:162:
111773477724160:error:260B6084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_dyn.c:434:
Doing aes-256-gcm for 3s on 16 size blocks: 7561396 aes-256-gcm's in 2.24s
Doing aes-256-gcm for 3s on 64 size blocks: 7055718 aes-256-gcm's in 2.93s
Doing aes-256-gcm for 3s on 256 size blocks: 3388454 aes-256-gcm's in 2.89s
Doing aes-256-gcm for 3s on 1024 size blocks: 1151120 aes-256-gcm's in 2.99s
Doing aes-256-gcm for 3s on 8192 size blocks: 160921 aes-256-gcm's in 3.06s
Doing aes-256-gcm for 3s on 16384 size blocks: 78897 aes-256-gcm's in 2.99s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-gcm      53957.28k   154134.51k   300088.81k   393941.52k   430453.82k   432007.84k

How would you access it with openssl, I have DOC enabled but it does not recognize this as a useable engine?

Side note, In 23.05.01 devcrypto does work and look at the noticeable differences:
Check out 64bytes, 1024bytes, and 8192bytes.

Please can you also research what is the normal block that use is in use.

The 16384 bytes lags behind safexcel but the 64bytes,1024bytes and 8129bytes have a positive difference.

Now running this test does reflect a use of the safexcel chip so it is accessed seen here also the counters increment during the test.

Screenshot 2025-03-05 at 11.20.34.png

Chart of differences:
Screenshot 2025-03-05 at 11.18.47.png
without safexcel the advantages favor 16 bytes, 256bytes, and 16384bytes tests.

No devcrypto:

Doing aes-256-cbc for 3s on 16 size blocks: 184854 aes-256-cbc's in 0.13s
Doing aes-256-cbc for 3s on 64 size blocks: 191301 aes-256-cbc's in 0.19s
Doing aes-256-cbc for 3s on 256 size blocks: 176461 aes-256-cbc's in 0.10s
Doing aes-256-cbc for 3s on 1024 size blocks: 150608 aes-256-cbc's in 0.20s
Doing aes-256-cbc for 3s on 8192 size blocks: 69531 aes-256-cbc's in 0.09s
Doing aes-256-cbc for 3s on 16384 size blocks: 42587 aes-256-cbc's in 0.02s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-cbc      23661.31k    65297.41k   444790.31k   759249.68k  6628048.90k 44655706.11k

With devcrypto and aes-cbc as (are-gcm is missing from list for some reason)

Shell Output - openssl speed -evp aes-256-cbc -engine devcrypto
engine "devcrypto" set.
Doing aes-256-cbc for 3s on 16 size blocks: 188255 aes-256-cbc's in 0.14s
Doing aes-256-cbc for 3s on 64 size blocks: 186644 aes-256-cbc's in 0.16s
Doing aes-256-cbc for 3s on 256 size blocks: 179483 aes-256-cbc's in 0.13s
Doing aes-256-cbc for 3s on 1024 size blocks: 157366 aes-256-cbc's in 0.18s
Doing aes-256-cbc for 3s on 8192 size blocks: 70007 aes-256-cbc's in 0.05s
Doing aes-256-cbc for 3s on 16384 size blocks: 42670 aes-256-cbc's in 0.03s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-cbc      21419.24k    72808.94k   345958.76k   896794.62k 12234610.01k 22371368.96k

Without:

aes-256-cbc      23661.31k    65297.41k   444790.31k   759249.68k  6628048.90k 44655706.11k

With:

aes-256-cbc      21419.24k    72808.94k   345958.76k   896794.62k 12234610.01k 22371368.96k

stephenw10

Both those tests are using some crypto off-loading. You can see because without using the -elapsed switch the time recorded shows CPU time during the 3s of the test and it's far lower than 3s. That also means the resulting Bps numbers are meaningless because the time taken is wrong.

That's interesting though because I don't see it in 25.03-beta. What version are you seeing those numbers in? 23.05.1?

JonathanLee

@stephenw10 23.05.01

JonathanLee

@stephenw10 does 25.03 reflect increments on use of the chip in your system?

stephenw10

You mean in the interrupts shown by vmstat? They do, but not for aes-gcm. Which is what I'm digging into.

JonathanLee

@stephenw10 thank you. Yes on the vmstat.I was thinking about adapting the OpenSSL.conf file to add the SafeXcel but I don’t know how, I am sure it is just adding the chip. I seem to remember I could change openvpn to bsd/safexcel on the hardware offload a couple months ago.

Side Note: There is also a way to use offloading with the directive ssl_engine on my favorite Squid package I have learned, that is amazing.

Per Squid support..

If your Squid is built for libssl 3.0 or later, you may be able to configure /etc/ssl/openssl.cnf default provider to be the one you want. Such that Squid does not have to do anything for it to work.

I expect all the details relating to how devcrypto does its thing to be configured in /etc/ssl/openssl.cnf.

You may find this discussion from the OpenSSL community helpful:
https://github.com/openssl/openssl/issues/10701

(FTR; the

HTH
Amos
____________________*

ssl_engine devcrypto

Again same thing it accepts the directive but the vmstat counters do not increase.

Another note:

I also use .eli in fstab

# Device		Mountpoint	FStype	Options		Dump	Pass#
/dev/msdosfs/EFISYS	/boot/efi	msdosfs	rw,noatime,noauto	0	0
/dev/msdosfs/DTBFAT0	/boot/msdos	msdosfs	rw,noatime,noauto	0	0
/dev/gpt/swapUSB.eli		none	swap	sw		0	0

This .eli use with FreeBSD encrypts the swap and it does work when looking at logs.

Screenshot 2025-03-07 at 06.22.39.png

Weird right this does increase vmstat when it is used. Again that would not block out everything because I also disabled it to test and it still did not normalize the system.

Mar 5 20:23:52	kernel		safexcel0: <SafeXcel EIP-97 crypto accelerator> mem 0x90000-0xaffff irq 18,19,20,21,22,23 on simplebus1
Mar 5 20:23:52	kernel		warning: increase kern.maxswzone or reduce amount of swap.
Mar 5 20:23:52	kernel		warning: total configured swap (3831807 pages) exceeds maximum recommended amount (3383256 pages).
Mar 5 20:23:52	kernel		GEOM_ELI: Crypto: accelerated software
Mar 5 20:23:52	kernel		GEOM_ELI: Encryption: AES-XTS 128
Mar 5 20:23:52	kernel		GEOM_ELI: Device gpt/swapUSB.eli created.
Mar 5 20:23:52	kernel		nda0: 244198MB (500118192 512 byte sectors)
Mar 5 20:23:52	kernel		nda0: nvme version 1.3 x1 (max x2) lanes PCIe Gen2 (max Gen3) link
Mar 5 20:23:52	kernel		nda0: Serial Number BTTE9510086T256D-1
Mar 5 20:23:52	kernel		nda0: <INTEL HBRPEKNX0101AH HPS1 BTTE9510086T256D-1>
Mar 5 20:23:52	kernel		nda0 at nvme0 bus 0 scbus1 target 0 lun 1
Mar 5 20:23:52	kernel		ada0: 122104MB (250069680 512 byte sectors)
Mar 5 20:23:52	kernel		ada0: Command Queueing enabled
Mar 5 20:23:52	kernel		ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
Mar 5 20:23:52	kernel		ada0: Serial Number 0014307000651
Mar 5 20:23:52	kernel		ada0: <NT-128 2242 SN14941> ACS-4 ATA SATA 3.x device

JonathanLee

@stephenw10 any update on this? Maybe a change in openssl.conf ? I don’t know I am wondering where to add that cipher, as it is supported.

stephenw10

It's unlikely to work with the userspace devcrypto device. And even if it did you should not use it because it's so slow as to be worthless. Especially since armv8crypto was added.

But it should work for in-kernel crypto functions and appears not to be. Still digging...

JonathanLee

@stephenw10 thanks for checking.

JonathanLee

@stephenw10 Does use of .eli in the fstab file cause this? It should be able to do both right?

stephenw10

I would expect that to be kernel level crypto, yes. But I've never checked specifically.

JonathanLee

@stephenw10 how you doing today?

Any updates on that kernel crypto functions?

stephenw10

Not yet. I was away this weekend.

JonathanLee

@stephenw10 I think it has to do with fstab use or .eli for swap, but even if I turned off .eli it still does not work.

JonathanLee

@stephenw10 Any word, if you need a copy of my config that is no problem.

JonathanLee

@stephenw10 I just wanted to follow up on this. I was able to get SafeXcel to increment with use of setting Squid proxy to use the sslengine as devcrypto. Don’t know if that helps

stephenw10

Hmm, interesting so you see interrupts there but not when calling it via openssl-speed?