OpenSSL not loading full SafeXcel capabilities.
-
armv8crypto is more like aesni for ARM CPUs. It's not safeXcel.
When you run openssl with an engine selected it will use software for algorithms not supported by the 'engine'.
The cryptodev framework can only be used by kernel space code. devcrypto existed to allow it to be used by userspace but is much slower and, as I understand it, not maintained which is why it stopped working.
You should use
-elapsedwith openssl speed to see the actual time taken and not just CPU time. Othewise, when an actual hardware engine is used, you see very small times.If you're using DCO in OpenVPN you should not use devcrypto. DCO will use whatever is registered by the cryptodev framework and that should be safexcel.
I will say there's a bunch of confusion here. Still. As I understand it the devcrypto part hasn't worked as I expected since FreeBSD11, so 2.4.X.
Let me re-dig into it.... -
@stephenw10 said in OpenSSL not loading full SafeXcel capabilities.:
cryptodev
Thanks for the reply,
I cant do any speed test with that engine.
Shell Output - openssl speed -evp aes-256-gcm -engine cryptodev invalid engine "cryptodev" 111773477724160:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/engines/cryptodev.so): Cannot open "/usr/lib/engines/cryptodev.so" 111773477724160:error:25070067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_lib.c:162: 111773477724160:error:260B6084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_dyn.c:434: 111773477724160:error:2606A074:engine routines:ENGINE_by_id:no such engine:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_list.c:421:id=cryptodev 111773477724160:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(libcryptodev.so): Shared object "libcryptodev.so" not found, required by "openssl" 111773477724160:error:25070067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_lib.c:162: 111773477724160:error:260B6084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_dyn.c:434: Doing aes-256-gcm for 3s on 16 size blocks: 7561396 aes-256-gcm's in 2.24s Doing aes-256-gcm for 3s on 64 size blocks: 7055718 aes-256-gcm's in 2.93s Doing aes-256-gcm for 3s on 256 size blocks: 3388454 aes-256-gcm's in 2.89s Doing aes-256-gcm for 3s on 1024 size blocks: 1151120 aes-256-gcm's in 2.99s Doing aes-256-gcm for 3s on 8192 size blocks: 160921 aes-256-gcm's in 3.06s Doing aes-256-gcm for 3s on 16384 size blocks: 78897 aes-256-gcm's in 2.99s OpenSSL 1.1.1t-freebsd 7 Feb 2023 built on: reproducible build, date unspecified options:bn(64,64) rc4(int) des(int) aes(partial) idea(int) blowfish(ptr) compiler: clang The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes aes-256-gcm 53957.28k 154134.51k 300088.81k 393941.52k 430453.82k 432007.84kHow would you access it with openssl, I have DOC enabled but it does not recognize this as a useable engine?
Side note, In 23.05.01 devcrypto does work and look at the noticeable differences:
Check out 64bytes, 1024bytes, and 8192bytes.Please can you also research what is the normal block that use is in use.
The 16384 bytes lags behind safexcel but the 64bytes,1024bytes and 8129bytes have a positive difference.
Now running this test does reflect a use of the safexcel chip so it is accessed seen here also the counters increment during the test.
Chart of differences:
without safexcel the advantages favor 16 bytes, 256bytes, and 16384bytes tests.No devcrypto:
Doing aes-256-cbc for 3s on 16 size blocks: 184854 aes-256-cbc's in 0.13s Doing aes-256-cbc for 3s on 64 size blocks: 191301 aes-256-cbc's in 0.19s Doing aes-256-cbc for 3s on 256 size blocks: 176461 aes-256-cbc's in 0.10s Doing aes-256-cbc for 3s on 1024 size blocks: 150608 aes-256-cbc's in 0.20s Doing aes-256-cbc for 3s on 8192 size blocks: 69531 aes-256-cbc's in 0.09s Doing aes-256-cbc for 3s on 16384 size blocks: 42587 aes-256-cbc's in 0.02s OpenSSL 1.1.1t-freebsd 7 Feb 2023 built on: reproducible build, date unspecified options:bn(64,64) rc4(int) des(int) aes(partial) idea(int) blowfish(ptr) compiler: clang The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes aes-256-cbc 23661.31k 65297.41k 444790.31k 759249.68k 6628048.90k 44655706.11kWith devcrypto and aes-cbc as (are-gcm is missing from list for some reason)
Shell Output - openssl speed -evp aes-256-cbc -engine devcrypto engine "devcrypto" set. Doing aes-256-cbc for 3s on 16 size blocks: 188255 aes-256-cbc's in 0.14s Doing aes-256-cbc for 3s on 64 size blocks: 186644 aes-256-cbc's in 0.16s Doing aes-256-cbc for 3s on 256 size blocks: 179483 aes-256-cbc's in 0.13s Doing aes-256-cbc for 3s on 1024 size blocks: 157366 aes-256-cbc's in 0.18s Doing aes-256-cbc for 3s on 8192 size blocks: 70007 aes-256-cbc's in 0.05s Doing aes-256-cbc for 3s on 16384 size blocks: 42670 aes-256-cbc's in 0.03s OpenSSL 1.1.1t-freebsd 7 Feb 2023 built on: reproducible build, date unspecified options:bn(64,64) rc4(int) des(int) aes(partial) idea(int) blowfish(ptr) compiler: clang The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes aes-256-cbc 21419.24k 72808.94k 345958.76k 896794.62k 12234610.01k 22371368.96kWithout:
aes-256-cbc 23661.31k 65297.41k 444790.31k 759249.68k 6628048.90k 44655706.11kWith:
aes-256-cbc 21419.24k 72808.94k 345958.76k 896794.62k 12234610.01k 22371368.96k -
Both those tests are using some crypto off-loading. You can see because without using the
-elapsedswitch the time recorded shows CPU time during the 3s of the test and it's far lower than 3s. That also means the resulting Bps numbers are meaningless because the time taken is wrong.That's interesting though because I don't see it in 25.03-beta. What version are you seeing those numbers in? 23.05.1?
-
@stephenw10 23.05.01
-
@stephenw10 does 25.03 reflect increments on use of the chip in your system?
-
You mean in the interrupts shown by vmstat? They do, but not for aes-gcm. Which is what I'm digging into.
-
@stephenw10 thank you. Yes on the vmstat.I was thinking about adapting the OpenSSL.conf file to add the SafeXcel but I don’t know how, I am sure it is just adding the chip. I seem to remember I could change openvpn to bsd/safexcel on the hardware offload a couple months ago.
Side Note: There is also a way to use offloading with the directive ssl_engine on my favorite Squid package I have learned, that is amazing.
Per Squid support..
- If your Squid is built for libssl 3.0 or later, you may be able to configure /etc/ssl/openssl.cnf default provider to be the one you want. Such that Squid does not have to do anything for it to work.
I expect all the details relating to how devcrypto does its thing to be configured in /etc/ssl/openssl.cnf.
You may find this discussion from the OpenSSL community helpful:
https://github.com/openssl/openssl/issues/10701(FTR; the
HTH
Amos
____________________*ssl_engine devcryptoAgain same thing it accepts the directive but the vmstat counters do not increase.
Another note:
I also use .eli in fstab
# Device Mountpoint FStype Options Dump Pass# /dev/msdosfs/EFISYS /boot/efi msdosfs rw,noatime,noauto 0 0 /dev/msdosfs/DTBFAT0 /boot/msdos msdosfs rw,noatime,noauto 0 0 /dev/gpt/swapUSB.eli none swap sw 0 0This .eli use with FreeBSD encrypts the swap and it does work when looking at logs.
Weird right this does increase vmstat when it is used. Again that would not block out everything because I also disabled it to test and it still did not normalize the system.
Mar 5 20:23:52 kernel safexcel0: <SafeXcel EIP-97 crypto accelerator> mem 0x90000-0xaffff irq 18,19,20,21,22,23 on simplebus1 Mar 5 20:23:52 kernel warning: increase kern.maxswzone or reduce amount of swap. Mar 5 20:23:52 kernel warning: total configured swap (3831807 pages) exceeds maximum recommended amount (3383256 pages). Mar 5 20:23:52 kernel GEOM_ELI: Crypto: accelerated software Mar 5 20:23:52 kernel GEOM_ELI: Encryption: AES-XTS 128 Mar 5 20:23:52 kernel GEOM_ELI: Device gpt/swapUSB.eli created. Mar 5 20:23:52 kernel nda0: 244198MB (500118192 512 byte sectors) Mar 5 20:23:52 kernel nda0: nvme version 1.3 x1 (max x2) lanes PCIe Gen2 (max Gen3) link Mar 5 20:23:52 kernel nda0: Serial Number BTTE9510086T256D-1 Mar 5 20:23:52 kernel nda0: <INTEL HBRPEKNX0101AH HPS1 BTTE9510086T256D-1> Mar 5 20:23:52 kernel nda0 at nvme0 bus 0 scbus1 target 0 lun 1 Mar 5 20:23:52 kernel ada0: 122104MB (250069680 512 byte sectors) Mar 5 20:23:52 kernel ada0: Command Queueing enabled Mar 5 20:23:52 kernel ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes) Mar 5 20:23:52 kernel ada0: Serial Number 0014307000651 Mar 5 20:23:52 kernel ada0: <NT-128 2242 SN14941> ACS-4 ATA SATA 3.x device -
@stephenw10 any update on this? Maybe a change in openssl.conf ? I don’t know I am wondering where to add that cipher, as it is supported.
-
It's unlikely to work with the userspace devcrypto device. And even if it did you should not use it because it's so slow as to be worthless. Especially since armv8crypto was added.
But it should work for in-kernel crypto functions and appears not to be. Still digging...
-
@stephenw10 thanks for checking.
-
@stephenw10 Does use of .eli in the fstab file cause this? It should be able to do both right?
-
I would expect that to be kernel level crypto, yes. But I've never checked specifically.
-
@stephenw10 how you doing today?
Any updates on that kernel crypto functions?
-
Not yet. I was away this weekend.
-
@stephenw10 I think it has to do with fstab use or .eli for swap, but even if I turned off .eli it still does not work.
-
@stephenw10 Any word, if you need a copy of my config that is no problem.
-
@stephenw10 I just wanted to follow up on this. I was able to get SafeXcel to increment with use of setting Squid proxy to use the sslengine as devcrypto. Don’t know if that helps
-
Hmm, interesting so you see interrupts there but not when calling it via openssl-speed?
-
@stephenw10 yes, I can see interrupts when using squid’s ssl engine directive when doing ssl intercept, but when the OpenVPN use it will not increment. I keep thinking it is because I use .Eli in the fstab file for the swap encryption, but if that was the case why does it increment when I use .Eli and squid’s ssl engine directive? Weird right ? And it does improve performance with the certificate stuff.
Squid custom option.
ssl_engine devcrypto -
@JonathanLee said in OpenSSL not loading full SafeXcel capabilities.:
I can see interrupts when using squid’s ssl engine directive when doing ssl intercept, but when the OpenVPN use it will not increment.
Seems normal and understandable to me.
The OpenVPN app connects only to the OpenVPN server, and the connection is created if authentication worked out fine.
I don't see the MITM (pfSense) doing that : emulating and OpenVPN server authentication so it can intercept.And its a waste of time trying to decrypt a OpenVPN stream, OpenVPN can't be 'MITMed', not with the hardware that exist in 2025.
Maybe the quantum pfSense version in the future ?
