OpenSSL not loading full SafeXcel capabilities.

JonathanLee

Yes thank you for clarifying,

It shows

Shell Output - /usr/bin/openssl engine -t -c
(devcrypto) /dev/crypto engine
 [AES-128-CBC, AES-192-CBC, AES-256-CBC]

This is on 23.05 so it is still using the version that should work with AES-256-GCM right? I can see counters generate but it acts as if it does not use it.

Screenshot 2025-03-04 at 16.57.35.png
OpenVPN sees /dev/crypto

Screenshot 2025-03-04 at 16.58.38.png
(counter do not increment with vpn use I used it alot today)

Screenshot 2025-03-04 at 16.59.26.png
(It is loaded)

Screenshot 2025-03-04 at 16.59.55.png
(GUI reflects it is active)

Screenshot 2025-03-04 at 17.01.06.png
(boot logs show it is supported)

Screenshot 2025-03-04 at 17.01.41.png
(swap is encrypted with .eli and reflects it is loaded in system logs)

If i disable eli and reboot same thing. All of the sudden it no longer sees /safexcel stuff. Did I mix up something, did a patch kill the keys?

I have DOC enabled also.

Screenshot 2025-03-04 at 17.03.30.png

I am a full computer science student, I just do not know what I did that would cause that.

JonathanLee

@stephenw10 said in OpenSSL not loading full SafeXcel capabilities.:

OpenSSL can no longer use devcrypto as a user space engine to access i

Should I go back to 22.01 to show it works?

JonathanLee

I noticed
safexcel_load="YES"
is missing from /boot/loader.conf

I do not know if it will fix it I will have to try

Updated: Same result on reboot

JonathanLee

Shell Output - openssl speed -evp aes-256-gcm -engine cryptodev
invalid engine "cryptodev"
63265584361472:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/engines/cryptodev.so): Cannot open "/usr/lib/engines/cryptodev.so"
63265584361472:error:25070067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_lib.c:162:
63265584361472:error:260B6084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_dyn.c:434:
63265584361472:error:2606A074:engine routines:ENGINE_by_id:no such engine:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_list.c:421:id=cryptodev
63265584361472:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(libcryptodev.so): Shared object "libcryptodev.so" not found, required by "openssl"
63265584361472:error:25070067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_lib.c:162:
63265584361472:error:260B6084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_dyn.c:434:
Doing aes-256-gcm for 3s on 16 size blocks: 9644470 aes-256-gcm's in 2.85s
Doing aes-256-gcm for 3s on 64 size blocks: 7296158 aes-256-gcm's in 2.98s
Doing aes-256-gcm for 3s on 256 size blocks: 3508073 aes-256-gcm's in 2.98s
Doing aes-256-gcm for 3s on 1024 size blocks: 1096076 aes-256-gcm's in 2.86s
Doing aes-256-gcm for 3s on 8192 size blocks: 159707 aes-256-gcm's in 3.04s
Doing aes-256-gcm for 3s on 16384 size blocks: 33755 aes-256-gcm's in 1.28s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-gcm      54114.72k   156876.97k   300922.87k   392526.98k   430501.10k   431642.47k

JonathanLee

@stephenw10 Check this out it does support it...

Shell Output - openssl speed -evp aes-256-gcm -engine devcrypto
engine "devcrypto" set.
Doing aes-256-gcm for 3s on 16 size blocks: 9609060 aes-256-gcm's in 2.83s
Doing aes-256-gcm for 3s on 64 size blocks: 6636379 aes-256-gcm's in 2.72s
Doing aes-256-gcm for 3s on 256 size blocks: 3487196 aes-256-gcm's in 2.96s
Doing aes-256-gcm for 3s on 1024 size blocks: 1146424 aes-256-gcm's in 2.98s
Doing aes-256-gcm for 3s on 8192 size blocks: 142475 aes-256-gcm's in 2.73s
Doing aes-256-gcm for 3s on 16384 size blocks: 80084 aes-256-gcm's in 3.04s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-gcm      54362.86k   156221.89k   301499.84k   394393.93k   426845.33k   431743.76k

Why is the GUI not allowing it??

Again this is faster

hell Output - openssl speed -evp aes-256-cbc -engine devcrypto
engine "devcrypto" set.
Doing aes-256-cbc for 3s on 16 size blocks: 186704 aes-256-cbc's in 0.13s
Doing aes-256-cbc for 3s on 64 size blocks: 185430 aes-256-cbc's in 0.16s
Doing aes-256-cbc for 3s on 256 size blocks: 178314 aes-256-cbc's in 0.15s
Doing aes-256-cbc for 3s on 1024 size blocks: 155657 aes-256-cbc's in 0.11s
Doing aes-256-cbc for 3s on 8192 size blocks: 69417 aes-256-cbc's in 0.04s
Doing aes-256-cbc for 3s on 16384 size blocks: 37161 aes-256-cbc's in 0.04s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-cbc      23898.11k    75952.13k   307525.96k  1457305.31k 14557800.04k 15586453.09k

JonathanLee

@stephenw10 said in OpenSSL not loading full SafeXcel capabilities.:

cryptodev

Can you use cryptodev with ssl_engine in squid at the same time?

JonathanLee

@JonathanLee I take that back only doing tests with cbc increment counters

stephenw10

armv8crypto is more like aesni for ARM CPUs. It's not safeXcel.

When you run openssl with an engine selected it will use software for algorithms not supported by the 'engine'.

The cryptodev framework can only be used by kernel space code. devcrypto existed to allow it to be used by userspace but is much slower and, as I understand it, not maintained which is why it stopped working.

You should use -elapsed with openssl speed to see the actual time taken and not just CPU time. Othewise, when an actual hardware engine is used, you see very small times.

If you're using DCO in OpenVPN you should not use devcrypto. DCO will use whatever is registered by the cryptodev framework and that should be safexcel.

I will say there's a bunch of confusion here. Still. As I understand it the devcrypto part hasn't worked as I expected since FreeBSD11, so 2.4.X.
Let me re-dig into it....

JonathanLee

@stephenw10 said in OpenSSL not loading full SafeXcel capabilities.:

cryptodev

Thanks for the reply,

I cant do any speed test with that engine.

Shell Output - openssl speed -evp aes-256-gcm -engine cryptodev
invalid engine "cryptodev"
111773477724160:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/engines/cryptodev.so): Cannot open "/usr/lib/engines/cryptodev.so"
111773477724160:error:25070067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_lib.c:162:
111773477724160:error:260B6084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_dyn.c:434:
111773477724160:error:2606A074:engine routines:ENGINE_by_id:no such engine:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_list.c:421:id=cryptodev
111773477724160:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(libcryptodev.so): Shared object "libcryptodev.so" not found, required by "openssl"
111773477724160:error:25070067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_lib.c:162:
111773477724160:error:260B6084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_dyn.c:434:
Doing aes-256-gcm for 3s on 16 size blocks: 7561396 aes-256-gcm's in 2.24s
Doing aes-256-gcm for 3s on 64 size blocks: 7055718 aes-256-gcm's in 2.93s
Doing aes-256-gcm for 3s on 256 size blocks: 3388454 aes-256-gcm's in 2.89s
Doing aes-256-gcm for 3s on 1024 size blocks: 1151120 aes-256-gcm's in 2.99s
Doing aes-256-gcm for 3s on 8192 size blocks: 160921 aes-256-gcm's in 3.06s
Doing aes-256-gcm for 3s on 16384 size blocks: 78897 aes-256-gcm's in 2.99s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-gcm      53957.28k   154134.51k   300088.81k   393941.52k   430453.82k   432007.84k

How would you access it with openssl, I have DOC enabled but it does not recognize this as a useable engine?

Side note, In 23.05.01 devcrypto does work and look at the noticeable differences:
Check out 64bytes, 1024bytes, and 8192bytes.

Please can you also research what is the normal block that use is in use.

The 16384 bytes lags behind safexcel but the 64bytes,1024bytes and 8129bytes have a positive difference.

Now running this test does reflect a use of the safexcel chip so it is accessed seen here also the counters increment during the test.

Screenshot 2025-03-05 at 11.20.34.png

Chart of differences:
Screenshot 2025-03-05 at 11.18.47.png
without safexcel the advantages favor 16 bytes, 256bytes, and 16384bytes tests.

No devcrypto:

Doing aes-256-cbc for 3s on 16 size blocks: 184854 aes-256-cbc's in 0.13s
Doing aes-256-cbc for 3s on 64 size blocks: 191301 aes-256-cbc's in 0.19s
Doing aes-256-cbc for 3s on 256 size blocks: 176461 aes-256-cbc's in 0.10s
Doing aes-256-cbc for 3s on 1024 size blocks: 150608 aes-256-cbc's in 0.20s
Doing aes-256-cbc for 3s on 8192 size blocks: 69531 aes-256-cbc's in 0.09s
Doing aes-256-cbc for 3s on 16384 size blocks: 42587 aes-256-cbc's in 0.02s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-cbc      23661.31k    65297.41k   444790.31k   759249.68k  6628048.90k 44655706.11k

With devcrypto and aes-cbc as (are-gcm is missing from list for some reason)

Shell Output - openssl speed -evp aes-256-cbc -engine devcrypto
engine "devcrypto" set.
Doing aes-256-cbc for 3s on 16 size blocks: 188255 aes-256-cbc's in 0.14s
Doing aes-256-cbc for 3s on 64 size blocks: 186644 aes-256-cbc's in 0.16s
Doing aes-256-cbc for 3s on 256 size blocks: 179483 aes-256-cbc's in 0.13s
Doing aes-256-cbc for 3s on 1024 size blocks: 157366 aes-256-cbc's in 0.18s
Doing aes-256-cbc for 3s on 8192 size blocks: 70007 aes-256-cbc's in 0.05s
Doing aes-256-cbc for 3s on 16384 size blocks: 42670 aes-256-cbc's in 0.03s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-cbc      21419.24k    72808.94k   345958.76k   896794.62k 12234610.01k 22371368.96k

Without:

aes-256-cbc      23661.31k    65297.41k   444790.31k   759249.68k  6628048.90k 44655706.11k

With:

aes-256-cbc      21419.24k    72808.94k   345958.76k   896794.62k 12234610.01k 22371368.96k

stephenw10

Both those tests are using some crypto off-loading. You can see because without using the -elapsed switch the time recorded shows CPU time during the 3s of the test and it's far lower than 3s. That also means the resulting Bps numbers are meaningless because the time taken is wrong.

That's interesting though because I don't see it in 25.03-beta. What version are you seeing those numbers in? 23.05.1?

JonathanLee

@stephenw10 23.05.01

JonathanLee

@stephenw10 does 25.03 reflect increments on use of the chip in your system?

stephenw10

You mean in the interrupts shown by vmstat? They do, but not for aes-gcm. Which is what I'm digging into.

JonathanLee

@stephenw10 thank you. Yes on the vmstat.I was thinking about adapting the OpenSSL.conf file to add the SafeXcel but I don’t know how, I am sure it is just adding the chip. I seem to remember I could change openvpn to bsd/safexcel on the hardware offload a couple months ago.

Side Note: There is also a way to use offloading with the directive ssl_engine on my favorite Squid package I have learned, that is amazing.

Per Squid support..

If your Squid is built for libssl 3.0 or later, you may be able to configure /etc/ssl/openssl.cnf default provider to be the one you want. Such that Squid does not have to do anything for it to work.

I expect all the details relating to how devcrypto does its thing to be configured in /etc/ssl/openssl.cnf.

You may find this discussion from the OpenSSL community helpful:
https://github.com/openssl/openssl/issues/10701

(FTR; the

HTH
Amos
____________________*

ssl_engine devcrypto

Again same thing it accepts the directive but the vmstat counters do not increase.

Another note:

I also use .eli in fstab

# Device		Mountpoint	FStype	Options		Dump	Pass#
/dev/msdosfs/EFISYS	/boot/efi	msdosfs	rw,noatime,noauto	0	0
/dev/msdosfs/DTBFAT0	/boot/msdos	msdosfs	rw,noatime,noauto	0	0
/dev/gpt/swapUSB.eli		none	swap	sw		0	0

This .eli use with FreeBSD encrypts the swap and it does work when looking at logs.

Screenshot 2025-03-07 at 06.22.39.png

Weird right this does increase vmstat when it is used. Again that would not block out everything because I also disabled it to test and it still did not normalize the system.

Mar 5 20:23:52	kernel		safexcel0: <SafeXcel EIP-97 crypto accelerator> mem 0x90000-0xaffff irq 18,19,20,21,22,23 on simplebus1
Mar 5 20:23:52	kernel		warning: increase kern.maxswzone or reduce amount of swap.
Mar 5 20:23:52	kernel		warning: total configured swap (3831807 pages) exceeds maximum recommended amount (3383256 pages).
Mar 5 20:23:52	kernel		GEOM_ELI: Crypto: accelerated software
Mar 5 20:23:52	kernel		GEOM_ELI: Encryption: AES-XTS 128
Mar 5 20:23:52	kernel		GEOM_ELI: Device gpt/swapUSB.eli created.
Mar 5 20:23:52	kernel		nda0: 244198MB (500118192 512 byte sectors)
Mar 5 20:23:52	kernel		nda0: nvme version 1.3 x1 (max x2) lanes PCIe Gen2 (max Gen3) link
Mar 5 20:23:52	kernel		nda0: Serial Number BTTE9510086T256D-1
Mar 5 20:23:52	kernel		nda0: <INTEL HBRPEKNX0101AH HPS1 BTTE9510086T256D-1>
Mar 5 20:23:52	kernel		nda0 at nvme0 bus 0 scbus1 target 0 lun 1
Mar 5 20:23:52	kernel		ada0: 122104MB (250069680 512 byte sectors)
Mar 5 20:23:52	kernel		ada0: Command Queueing enabled
Mar 5 20:23:52	kernel		ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
Mar 5 20:23:52	kernel		ada0: Serial Number 0014307000651
Mar 5 20:23:52	kernel		ada0: <NT-128 2242 SN14941> ACS-4 ATA SATA 3.x device

JonathanLee

@stephenw10 any update on this? Maybe a change in openssl.conf ? I don’t know I am wondering where to add that cipher, as it is supported.

stephenw10

It's unlikely to work with the userspace devcrypto device. And even if it did you should not use it because it's so slow as to be worthless. Especially since armv8crypto was added.

But it should work for in-kernel crypto functions and appears not to be. Still digging...

JonathanLee

@stephenw10 thanks for checking.

JonathanLee

@stephenw10 Does use of .eli in the fstab file cause this? It should be able to do both right?

stephenw10

I would expect that to be kernel level crypto, yes. But I've never checked specifically.

JonathanLee

@stephenw10 how you doing today?

Any updates on that kernel crypto functions?