OpenSSL not loading full SafeXcel capabilities.

JonathanLee

@Gertjan is that specific to devcrypto?

I guess my question is why does devcrypto not see it anymore it should support aes-gcm.
I have it SafeXcel enabled and marked active and the other one disabled.

JonathanLee

@Gertjan

Shell Output - /usr/bin/openssl ciphers ALL
TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-CCM8:ECDHE-ECDSA-AES256-CCM:DHE-RSA-AES256-CCM8:DHE-RSA-AES256-CCM:ECDHE-ECDSA-ARIA256-GCM-SHA384:ECDHE-ARIA256-GCM-SHA384:DHE-DSS-ARIA256-GCM-SHA384:DHE-RSA-ARIA256-GCM-SHA384:ADH-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-CCM8:ECDHE-ECDSA-AES128-CCM:DHE-RSA-AES128-CCM8:DHE-RSA-AES128-CCM:ECDHE-ECDSA-ARIA128-GCM-SHA256:ECDHE-ARIA128-GCM-SHA256:DHE-DSS-ARIA128-GCM-SHA256:DHE-RSA-ARIA128-GCM-SHA256:ADH-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:ECDHE-ECDSA-CAMELLIA256-SHA384:ECDHE-RSA-CAMELLIA256-SHA384:DHE-RSA-CAMELLIA256-SHA256:DHE-DSS-CAMELLIA256-SHA256:ADH-AES256-SHA256:ADH-CAMELLIA256-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:ECDHE-ECDSA-CAMELLIA128-SHA256:ECDHE-RSA-CAMELLIA128-SHA256:DHE-RSA-CAMELLIA128-SHA256:DHE-DSS-CAMELLIA128-SHA256:ADH-AES128-SHA256:ADH-CAMELLIA128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-CAMELLIA256-SHA:DHE-DSS-CAMELLIA256-SHA:AECDH-AES256-SHA:ADH-AES256-SHA:ADH-CAMELLIA256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DHE-RSA-SEED-SHA:DHE-DSS-SEED-SHA:DHE-RSA-CAMELLIA128-SHA:DHE-DSS-CAMELLIA128-SHA:AECDH-AES128-SHA:ADH-AES128-SHA:ADH-SEED-SHA:ADH-CAMELLIA128-SHA:RSA-PSK-AES256-GCM-SHA384:DHE-PSK-AES256-GCM-SHA384:RSA-PSK-CHACHA20-POLY1305:DHE-PSK-CHACHA20-POLY1305:ECDHE-PSK-CHACHA20-POLY1305:DHE-PSK-AES256-CCM8:DHE-PSK-AES256-CCM:RSA-PSK-ARIA256-GCM-SHA384:DHE-PSK-ARIA256-GCM-SHA384:AES256-GCM-SHA384:AES256-CCM8:AES256-CCM:ARIA256-GCM-SHA384:PSK-AES256-GCM-SHA384:PSK-CHACHA20-POLY1305:PSK-AES256-CCM8:PSK-AES256-CCM:PSK-ARIA256-GCM-SHA384:RSA-PSK-AES128-GCM-SHA256:DHE-PSK-AES128-GCM-SHA256:DHE-PSK-AES128-CCM8:DHE-PSK-AES128-CCM:RSA-PSK-ARIA128-GCM-SHA256:DHE-PSK-ARIA128-GCM-SHA256:AES128-GCM-SHA256:AES128-CCM8:AES128-CCM:ARIA128-GCM-SHA256:PSK-AES128-GCM-SHA256:PSK-AES128-CCM8:PSK-AES128-CCM:PSK-ARIA128-GCM-SHA256:AES256-SHA256:CAMELLIA256-SHA256:AES128-SHA256:CAMELLIA128-SHA256:ECDHE-PSK-AES256-CBC-SHA384:ECDHE-PSK-AES256-CBC-SHA:SRP-DSS-AES-256-CBC-SHA:SRP-RSA-AES-256-CBC-SHA:SRP-AES-256-CBC-SHA:RSA-PSK-AES256-CBC-SHA384:DHE-PSK-AES256-CBC-SHA384:RSA-PSK-AES256-CBC-SHA:DHE-PSK-AES256-CBC-SHA:ECDHE-PSK-CAMELLIA256-SHA384:RSA-PSK-CAMELLIA256-SHA384:DHE-PSK-CAMELLIA256-SHA384:AES256-SHA:CAMELLIA256-SHA:PSK-AES256-CBC-SHA384:PSK-AES256-CBC-SHA:PSK-CAMELLIA256-SHA384:ECDHE-PSK-AES128-CBC-SHA256:ECDHE-PSK-AES128-CBC-SHA:SRP-DSS-AES-128-CBC-SHA:SRP-RSA-AES-128-CBC-SHA:SRP-AES-128-CBC-SHA:RSA-PSK-AES128-CBC-SHA256:DHE-PSK-AES128-CBC-SHA256:RSA-PSK-AES128-CBC-SHA:DHE-PSK-AES128-CBC-SHA:ECDHE-PSK-CAMELLIA128-SHA256:RSA-PSK-CAMELLIA128-SHA256:DHE-PSK-CAMELLIA128-SHA256:AES128-SHA:SEED-SHA:CAMELLIA128-SHA:IDEA-CBC-SHA:PSK-AES128-CBC-SHA256:PSK-AES128-CBC-SHA:PSK-CAMELLIA128-SHA256

stephenw10

As I understand it the problem is not that safexcel doesn't support it. The issue is that OpenSSL can no longer use devcrypto as a user space engine to access it like that.

But you should still be able to use it via the cryptodev framework in kernel space. So for IPSec, Wireguard or OpenVPN with DCO.

JonathanLee

@stephenw10

Yes thank you for clarifying,

It shows

Shell Output - /usr/bin/openssl engine -t -c
(devcrypto) /dev/crypto engine
 [AES-128-CBC, AES-192-CBC, AES-256-CBC]

This is on 23.05 so it is still using the version that should work with AES-256-GCM right? I can see counters generate but it acts as if it does not use it.

Screenshot 2025-03-04 at 16.57.35.png
OpenVPN sees /dev/crypto

Screenshot 2025-03-04 at 16.58.38.png
(counter do not increment with vpn use I used it alot today)

Screenshot 2025-03-04 at 16.59.26.png
(It is loaded)

Screenshot 2025-03-04 at 16.59.55.png
(GUI reflects it is active)

Screenshot 2025-03-04 at 17.01.06.png
(boot logs show it is supported)

Screenshot 2025-03-04 at 17.01.41.png
(swap is encrypted with .eli and reflects it is loaded in system logs)

If i disable eli and reboot same thing. All of the sudden it no longer sees /safexcel stuff. Did I mix up something, did a patch kill the keys?

I have DOC enabled also.

Screenshot 2025-03-04 at 17.03.30.png

I am a full computer science student, I just do not know what I did that would cause that.

JonathanLee

@stephenw10 said in OpenSSL not loading full SafeXcel capabilities.:

OpenSSL can no longer use devcrypto as a user space engine to access i

Should I go back to 22.01 to show it works?

JonathanLee

I noticed
safexcel_load="YES"
is missing from /boot/loader.conf

I do not know if it will fix it I will have to try

Updated: Same result on reboot

JonathanLee

Shell Output - openssl speed -evp aes-256-gcm -engine cryptodev
invalid engine "cryptodev"
63265584361472:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/engines/cryptodev.so): Cannot open "/usr/lib/engines/cryptodev.so"
63265584361472:error:25070067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_lib.c:162:
63265584361472:error:260B6084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_dyn.c:434:
63265584361472:error:2606A074:engine routines:ENGINE_by_id:no such engine:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_list.c:421:id=cryptodev
63265584361472:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(libcryptodev.so): Shared object "libcryptodev.so" not found, required by "openssl"
63265584361472:error:25070067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_lib.c:162:
63265584361472:error:260B6084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_dyn.c:434:
Doing aes-256-gcm for 3s on 16 size blocks: 9644470 aes-256-gcm's in 2.85s
Doing aes-256-gcm for 3s on 64 size blocks: 7296158 aes-256-gcm's in 2.98s
Doing aes-256-gcm for 3s on 256 size blocks: 3508073 aes-256-gcm's in 2.98s
Doing aes-256-gcm for 3s on 1024 size blocks: 1096076 aes-256-gcm's in 2.86s
Doing aes-256-gcm for 3s on 8192 size blocks: 159707 aes-256-gcm's in 3.04s
Doing aes-256-gcm for 3s on 16384 size blocks: 33755 aes-256-gcm's in 1.28s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-gcm      54114.72k   156876.97k   300922.87k   392526.98k   430501.10k   431642.47k

JonathanLee

@stephenw10 Check this out it does support it...

Shell Output - openssl speed -evp aes-256-gcm -engine devcrypto
engine "devcrypto" set.
Doing aes-256-gcm for 3s on 16 size blocks: 9609060 aes-256-gcm's in 2.83s
Doing aes-256-gcm for 3s on 64 size blocks: 6636379 aes-256-gcm's in 2.72s
Doing aes-256-gcm for 3s on 256 size blocks: 3487196 aes-256-gcm's in 2.96s
Doing aes-256-gcm for 3s on 1024 size blocks: 1146424 aes-256-gcm's in 2.98s
Doing aes-256-gcm for 3s on 8192 size blocks: 142475 aes-256-gcm's in 2.73s
Doing aes-256-gcm for 3s on 16384 size blocks: 80084 aes-256-gcm's in 3.04s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-gcm      54362.86k   156221.89k   301499.84k   394393.93k   426845.33k   431743.76k

Why is the GUI not allowing it??

Again this is faster

hell Output - openssl speed -evp aes-256-cbc -engine devcrypto
engine "devcrypto" set.
Doing aes-256-cbc for 3s on 16 size blocks: 186704 aes-256-cbc's in 0.13s
Doing aes-256-cbc for 3s on 64 size blocks: 185430 aes-256-cbc's in 0.16s
Doing aes-256-cbc for 3s on 256 size blocks: 178314 aes-256-cbc's in 0.15s
Doing aes-256-cbc for 3s on 1024 size blocks: 155657 aes-256-cbc's in 0.11s
Doing aes-256-cbc for 3s on 8192 size blocks: 69417 aes-256-cbc's in 0.04s
Doing aes-256-cbc for 3s on 16384 size blocks: 37161 aes-256-cbc's in 0.04s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-cbc      23898.11k    75952.13k   307525.96k  1457305.31k 14557800.04k 15586453.09k

JonathanLee

@stephenw10 said in OpenSSL not loading full SafeXcel capabilities.:

cryptodev

Can you use cryptodev with ssl_engine in squid at the same time?

JonathanLee

@JonathanLee I take that back only doing tests with cbc increment counters

stephenw10

armv8crypto is more like aesni for ARM CPUs. It's not safeXcel.

When you run openssl with an engine selected it will use software for algorithms not supported by the 'engine'.

The cryptodev framework can only be used by kernel space code. devcrypto existed to allow it to be used by userspace but is much slower and, as I understand it, not maintained which is why it stopped working.

You should use -elapsed with openssl speed to see the actual time taken and not just CPU time. Othewise, when an actual hardware engine is used, you see very small times.

If you're using DCO in OpenVPN you should not use devcrypto. DCO will use whatever is registered by the cryptodev framework and that should be safexcel.

I will say there's a bunch of confusion here. Still. As I understand it the devcrypto part hasn't worked as I expected since FreeBSD11, so 2.4.X.
Let me re-dig into it....

JonathanLee

@stephenw10 said in OpenSSL not loading full SafeXcel capabilities.:

cryptodev

Thanks for the reply,

I cant do any speed test with that engine.

Shell Output - openssl speed -evp aes-256-gcm -engine cryptodev
invalid engine "cryptodev"
111773477724160:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/engines/cryptodev.so): Cannot open "/usr/lib/engines/cryptodev.so"
111773477724160:error:25070067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_lib.c:162:
111773477724160:error:260B6084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_dyn.c:434:
111773477724160:error:2606A074:engine routines:ENGINE_by_id:no such engine:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_list.c:421:id=cryptodev
111773477724160:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(libcryptodev.so): Shared object "libcryptodev.so" not found, required by "openssl"
111773477724160:error:25070067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_lib.c:162:
111773477724160:error:260B6084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_dyn.c:434:
Doing aes-256-gcm for 3s on 16 size blocks: 7561396 aes-256-gcm's in 2.24s
Doing aes-256-gcm for 3s on 64 size blocks: 7055718 aes-256-gcm's in 2.93s
Doing aes-256-gcm for 3s on 256 size blocks: 3388454 aes-256-gcm's in 2.89s
Doing aes-256-gcm for 3s on 1024 size blocks: 1151120 aes-256-gcm's in 2.99s
Doing aes-256-gcm for 3s on 8192 size blocks: 160921 aes-256-gcm's in 3.06s
Doing aes-256-gcm for 3s on 16384 size blocks: 78897 aes-256-gcm's in 2.99s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-gcm      53957.28k   154134.51k   300088.81k   393941.52k   430453.82k   432007.84k

How would you access it with openssl, I have DOC enabled but it does not recognize this as a useable engine?

Side note, In 23.05.01 devcrypto does work and look at the noticeable differences:
Check out 64bytes, 1024bytes, and 8192bytes.

Please can you also research what is the normal block that use is in use.

The 16384 bytes lags behind safexcel but the 64bytes,1024bytes and 8129bytes have a positive difference.

Now running this test does reflect a use of the safexcel chip so it is accessed seen here also the counters increment during the test.

Screenshot 2025-03-05 at 11.20.34.png

Chart of differences:
Screenshot 2025-03-05 at 11.18.47.png
without safexcel the advantages favor 16 bytes, 256bytes, and 16384bytes tests.

No devcrypto:

Doing aes-256-cbc for 3s on 16 size blocks: 184854 aes-256-cbc's in 0.13s
Doing aes-256-cbc for 3s on 64 size blocks: 191301 aes-256-cbc's in 0.19s
Doing aes-256-cbc for 3s on 256 size blocks: 176461 aes-256-cbc's in 0.10s
Doing aes-256-cbc for 3s on 1024 size blocks: 150608 aes-256-cbc's in 0.20s
Doing aes-256-cbc for 3s on 8192 size blocks: 69531 aes-256-cbc's in 0.09s
Doing aes-256-cbc for 3s on 16384 size blocks: 42587 aes-256-cbc's in 0.02s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-cbc      23661.31k    65297.41k   444790.31k   759249.68k  6628048.90k 44655706.11k

With devcrypto and aes-cbc as (are-gcm is missing from list for some reason)

Shell Output - openssl speed -evp aes-256-cbc -engine devcrypto
engine "devcrypto" set.
Doing aes-256-cbc for 3s on 16 size blocks: 188255 aes-256-cbc's in 0.14s
Doing aes-256-cbc for 3s on 64 size blocks: 186644 aes-256-cbc's in 0.16s
Doing aes-256-cbc for 3s on 256 size blocks: 179483 aes-256-cbc's in 0.13s
Doing aes-256-cbc for 3s on 1024 size blocks: 157366 aes-256-cbc's in 0.18s
Doing aes-256-cbc for 3s on 8192 size blocks: 70007 aes-256-cbc's in 0.05s
Doing aes-256-cbc for 3s on 16384 size blocks: 42670 aes-256-cbc's in 0.03s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-cbc      21419.24k    72808.94k   345958.76k   896794.62k 12234610.01k 22371368.96k

Without:

aes-256-cbc      23661.31k    65297.41k   444790.31k   759249.68k  6628048.90k 44655706.11k

With:

aes-256-cbc      21419.24k    72808.94k   345958.76k   896794.62k 12234610.01k 22371368.96k

stephenw10

Both those tests are using some crypto off-loading. You can see because without using the -elapsed switch the time recorded shows CPU time during the 3s of the test and it's far lower than 3s. That also means the resulting Bps numbers are meaningless because the time taken is wrong.

That's interesting though because I don't see it in 25.03-beta. What version are you seeing those numbers in? 23.05.1?

JonathanLee

@stephenw10 23.05.01

JonathanLee

@stephenw10 does 25.03 reflect increments on use of the chip in your system?

stephenw10

You mean in the interrupts shown by vmstat? They do, but not for aes-gcm. Which is what I'm digging into.