23.09d - Is QAT Broken?

jaltman

@RobbieTT what do "openssl engine" and "openssl engine -t -c -v qatengine" report on your device when running 23.05.1?

jimp

The redmine has not been "shelved" it's waiting on more information because without more information there is nothing we can do.

dev.qat_ocf.0.enable is to tie QAT into the opencrypto framework (cryptodev) which isn't really a change in how it operates but the newer base OS displays it differently.

Another difference in your case is that your WAN is PPPoE which may be using encryption of its own. It's possible that is the difference. There may be some difference in how PPPoE is doing encryption which is now not going through QAT when it was before.

Anything you're trying is also going over your WAN which would be causing PPPoE traffic.

So that's the next thing I would look at is the PPP connection logs from both to see if there is anything different there.

I still wouldn't normally expect that to be using QAT since mpd is a userspace daemon, but some parts of what happens for PPP do end up in the kernel so it's not 100% clear either way at the moment.

RobbieTT

@jaltman said in 23.09d - Is QAT Broken?:

openssl engine -t -c -v qatengine

[23.05.1-RELEASE][admin@Router-8.redacted.me]/root: openssl engine
(devcrypto) /dev/crypto engine
(rdrand) Intel RDRAND engine
(dynamic) Dynamic engine loading support
[23.05.1-RELEASE][admin@Router-8.redacted.me]/root:

[23.05.1-RELEASE][admin@Router-8.redacted.me]/root: openssl engine -t -c -v qatengine
14607459921920:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/engines/qatengine.so): Cannot open "/usr/lib/engines/qatengine.so"
14607459921920:error:25070067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_lib.c:162:
14607459921920:error:260B6084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_dyn.c:434:
14607459921920:error:2606A074:engine routines:ENGINE_by_id:no such engine:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_list.c:421:id=qatengine
[23.05.1-RELEASE][admin@Router-8.redacted.me]/root:

️

RobbieTT

@jimp said in 23.09d - Is QAT Broken?:

Another difference in your case is that your WAN is PPPoE which may be using encryption of its own. It's possible that is the difference. There may be some difference in how PPPoE is doing encryption which is now not going through QAT when it was before.

Anything you're trying is also going over your WAN which would be causing PPPoE traffic.

So that's the next thing I would look at is the PPP connection logs from both to see if there is anything different there.

Nothing of note in the PPPoE logs, just the regular handshakes.

The QAT interrupt rates are not varied by the WAN (PPPoE) handling traffic with no observable change between near-idle and almost saturated.

️

jimp

And you didn't do anything different with the installation, like maybe turn on filesystem encryption in the installer?

Any differences in sysctl tunables? loader.conf.local contents? Any packages that might be doing something with encryption? Any difference in kernel modules (not just qat modules)?

There has to be something more fundamentally different at play here. I can't replicate your experience here at all on 23.05.1 with a 4100.

For me, both before and after, it gets used with kernel encryption for IPsec and OpenVPN DCO and that's it. That's all that it should work with the way it's been implemented in the current driver on FreeBSD.

I even tried forcing on userspace QAT but the driver failed saying that only works on 4xxx QAT devices, so again, that isn't something that would have possibly been functional now or before.

RobbieTT

@jimp said in 23.09d - Is QAT Broken?:

And you didn't do anything different with the installation, like maybe turn on filesystem encryption in the installer?

Any differences in sysctl tunables? loader.conf.local contents? Any packages that might be doing something with encryption? Any difference in kernel modules (not just qat modules)?

irq175: qat0:b1                     3344          0
irq176: qat0:b2                     3684          0

No changes to the kernel and nothing via loader.conf.local. I do have 1 small change via sysctl tuneables, as you would expect:

net.isr.dispatch=deferred

Along with PPPoE I also use IPv6 and use FQ_CoDel. Installed packages and services actually active are:

2023-10-02 at 18.44.20.png

2023-10-02 at 18.44.33.png

I don't think these are significant but mentioned for completness.

️

NRgia

@RobbieTT said in 23.09d - Is QAT Broken?:

@jaltman said in 23.09d - Is QAT Broken?:

openssl engine -t -c -v qatengine

[23.05.1-RELEASE][admin@Router-8.redacted.me]/root: openssl engine
(devcrypto) /dev/crypto engine
(rdrand) Intel RDRAND engine
(dynamic) Dynamic engine loading support
[23.05.1-RELEASE][admin@Router-8.redacted.me]/root:

[23.05.1-RELEASE][admin@Router-8.redacted.me]/root: openssl engine -t -c -v qatengine
14607459921920:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/engines/qatengine.so): Cannot open "/usr/lib/engines/qatengine.so"
14607459921920:error:25070067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/dso/dso_lib.c:162:
14607459921920:error:260B6084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_dyn.c:434:
14607459921920:error:2606A074:engine routines:ENGINE_by_id:no such engine:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05_1-main/sources/FreeBSD-src-plus-RELENG_23_05_1/crypto/openssl/crypto/engine/eng_list.c:421:id=qatengine
[23.05.1-RELEASE][admin@Router-8.redacted.me]/root:

️

I'm getting the same error as you on a C3xxx board runnig pfSense+ 23.05.1. Is it normal, should we ignore this?

RobbieTT

@NRgia

Not much different on today's 23.09d. I don't know if it is significant error or not:

[23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root: openssl engine
(rdrand) Intel RDRAND engine
(dynamic) Dynamic engine loading support
[23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root: 

[23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root: openssl engine -t -c -v qatengine
0020E1AF5B420000:error:12800067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/engines-3/qatengine.so): Cannot open "/usr/lib/engines-3/qatengine.so"
0020E1AF5B420000:error:12800067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/dso/dso_lib.c:152:
0020E1AF5B420000:error:13000084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/engine/eng_dyn.c:442:
0020E1AF5B420000:error:13000074:engine routines:ENGINE_by_id:no such engine:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/engine/eng_list.c:430:id=qatengine
[23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root:

️

NRgia

@RobbieTT If on a development snapshot I can understand to happen, why I am also getting this on 23.05.1 which is a production version...I know you don't have the answers, but maybe someone will help with a reply.

jaltman

@NRgia said in 23.09d - Is QAT Broken?:

I'm getting the same error as you on a C3xxx board runnig pfSense+ 23.05.1. Is it normal, should we ignore this?

As far as I can tell the openssl qatengine is not built for pfSense and has never been. Therefore it is normal that these commands will fail.
If the openssl qatengine were present then nginx, curl, sshd, and ssh and other userspace tools would be able to leverage the QAT hardware.

@jimp has said that QAT is only expected to be enabled for kernel functions. The kernel QAT support does not rely upon the openssl qatengine.

These failures can be ignored.

NRgia

@jaltman Thanks, I feel releaved now.

RobbieTT

@jaltman said in 23.09d - Is QAT Broken?:

@jimp has said that QAT is only expected to be enabled for kernel functions. The kernel QAT support does not rely upon the openssl qatengine.

We still have the open question as to why pre 23.09d the QAT interrupts increment concurrently with on-device TLS/SSL type tasks.

I would be surprised if Netgate deliberately neutered the QAT support in the latest freeBSD offerings, given their close ties with upstream. They push QAT as a feature for pfSense+ so it would make more sense to unleash it wherever possible.

️

jimp

That kind of FUD is completely uncalled for. Netgate didn't reduce the functionality of the driver at all, we have put significant resources into its development.

eracerxrs

I am encountering a similar problem with QAT not working on a fresh install, but on 23.05.01.

I started a post before I found this one here.

Similar to OP here is my dmesg and vmstat output:

qat0: <Intel 200xx QuickAssist> mem 0xfe600000-0xfe63ffff,0xfe640000-0xfe67ffff irq 16 at device 0.0 on pci2
qat0: qat_dev0 started 6 acceleration engines
qat0: FW version: 4.18.0
qat0: Excessive clock measure delay
qat_ocf0: <QAT engine>
[23.05.1-RELEASE][admin@pfSense.home.arpa]/root: vmstat -i | grep qat
[23.05.1-RELEASE][admin@pfSense.home.arpa]/root:

I have not configured any VPNs as I was trying to get the dashboard QAT Crypto status to change to YES as a first step.

I'm a QAT newb, so it's possible I have overlooked something simple...

RobbieTT

@jimp said in 23.09d - Is QAT Broken?:

That kind of FUD is completely uncalled for. Netgate didn't reduce the functionality of the driver at all, we have put significant resources into its development.

@jimp I'm not sure if you are aiming at @jaltman (who was merely repeating your words) or myself, when I expressed surprise if Netgate's desire was to limit QAT functionality, especially as you push QAT as a feature. I was expressing doubt that Netgate would do this.

Jim, you have been PA or a bit combative on this issue for no real reason that I can see. You have spun my questions back on me by asking me what traffic, TLS/algorithms do I expect to be accelerated and even misstated my questions as statements, which you then baulk at. Meanwhile the original, simple A vs B question remains unexplained and sidestepped.

I have run all the tests you have asked for, checked all the configurations that you requested, spent hours booting in and out of pfSense versions that have only reinforced the original query. Simply put, what is show as accelerated by QAT in 23.05 is not in 23.09d. This is customer feedback on a technical issue that has arisen. Can we just sidestep the emotive and go back to 'the data is the data'?

For clarity:

I do understand that your instance of 23.05 is not triggering QAT interrupts for the traffic types I have given. We need to understand why those of us above have a different experience to yours.
I understand that you do not think QAT should be active on pfSense with SSH, nginx, curl, TLS/SSL, openSSL etc. This would equate to reduced feature-set from that stated in the Intel / freeBSD QAT documentation. That it appeared to work on 23.05 is in doubt as you have opined that this may be false reporting.
I understand that your current thinking is that the only things that would use QAT on pfSense+ is in the kernel space; more specifically only IPsec and OpenVPN DCO.
I understand that you do not expect pfSense+ to utilise QAT for any daemons or user-space. This reduced QAT functionality on pfSense+ would explain what I and others are observing on 23.09d (albeit not fully explaining the interrupts reported on 23.05).
However, you have also stated that Netgate has not reduced the Intel QAT functionality at all. This appears to be a contradiction to the bullet above.

Perhaps we are divided by a common language and the barrier of the written word but I really am investing time and effort to understand this and have been drawn down into a depth of the system that I don't think I should be as a customer.

So far we have learned that the Intel QAT / freeBSD implementation on the latest pfSense+ appears to be missing all functionality save for those executed through the kernel. This has (apparently) excluded all QAT user space functions including those frameworks directly enabled by Intel (eg OpenSSL, libcrypto etc) or via the QAT API (compression / decompression, SSL, TLS, nginx et al) and the QAT User Space Additional Functions.

I understand that the attempt to force a user space QAT driver on your device produced an error stating it only works on 4xxx QAT devices. The Intel QAT software release for freeBSD (which includes QAT user space support) makes no reference to the 4xxx QAT devices; it states that:

This software release is intended for platforms that contain:
• Intel C62x Chipset
• Intel Atom C3000 processor product family
• Intel QuickAssist Adapter 8960/ Intel QuickAssist Adapter 8970 (formerly known as "Lewis Hill")
• Intel Communications Chipset 8925 to 8955 Series
• Intel Atom P5300 processor product family

Refs: Package Version: QAT.B.3.12.0-00004 - June 2022 & GitHub - Intel - Asynch Mode for NGINX

@jimp I am sure you can see why some of us are confused as to the functionality of QAT in pfSense+ given the apparent (or at least appearance of) technical contradictions. This is not an attack on Netgate devs. We either have full QAT functionality on the C3xxx platforms or we don't. If we don't then this may be due to sound technical reasons, an error or oversight, a bug or just work in progress.

Regards, Rob

️

jaltman

@RobbieTT said in 23.09d - Is QAT Broken?:

I understand that you do not think QAT should be active on pfSense with SSH, nginx, curl, TLS/SSL, openSSL etc. This would equate to reduced feature-set from that stated in the Intel / freeBSD QAT documentation. That it appeared to work on 23.05 is in doubt as you have opined that this may be false reporting.

To be honest, I don't understand why QAT would be active for ssh, sshd, nginx, curl, or anything else linked against openssl's libcrypto when the openssl qatengine is not present on either 23.05.1 or 23.09-dev. There is no driver installed that exposes QAT to userspace nor is there a userspace library to call it. All of the above processes are linked to openssl's libcrypto. In 23.05.1 its an openssl 1.1.x library and in 23.09-dev its an openssl 3.0.x library but in neither case would I expect QAT to be used.

The QAT interrupts we are seeing must be coming from some kernel packet processing. I've tried obtaining a packet capture for the WAN and separately for the LAN while doing various things but there aren't any packets that jump out at me as something that would use QAT. I'm almost wondering if there is something from the WAN that appears to be attempting to establish a tunnel that doesn't exist and perhaps that is triggering the QAT activity with 23.05.1 but in 23.09-dev the trigger in 23.09-dev is correctly filtered out.

I'm not worried that QAT is not being used in 23.09-dev for userspace because I don't think it was being used for userspace in 23.05.1. However, I would like it to be used for userspace in the future. I would also appreciate it if the Netgate pfSense documentation was a bit more specific about when QAT can be used and when it cannot. The text on System->Advanced->Miscellaneous page doesn't explicitly mention QAT.

A cryptographic accelerator module will use hardware support to speed up some cryptographic functions on systems which have the chip. Loading the BSD Crypto Device module will allow access to acceleration devices using drivers built into the kernel, such as Hifn or ubsec chipsets. If the firewall does not contain a crypto chip, this option will have no effect. To unload the selected module, set this option to "none" and then reboot.

RobbieTT

@jaltman said in 23.09d - Is QAT Broken?:

...the openssl qatengine is not present on either 23.05.1 or 23.09-dev. There is no driver installed that exposes QAT to userspace nor is there a userspace library to call it.

So what do you think we are missing? The kernel files on the Intel documents all appear to be in place on pfSense, including the common and API:

/boot/kernel/qat_4xxx_fw.ko
/boot/kernel/qat_dh895xcc_fw.ko
/boot/kernel/qat_hw.ko
/boot/kernel/qat_c2xxxfw.ko
/boot/kernel/qat_c4xxx_fw.ko
/boot/kernel/qat_common.ko
/boot/kernel/qat_api.ko
/boot/kernel/qat_c3xxx_fw.ko
/boot/kernel/qat_c2xxx.ko
/boot/kernel/qat_c62x_fw.ko
/boot/kernel/qat.ko
/boot/kernel/qat_200xx_fw.ko

The QAT engine is there and nothing stands out as missing, at least to my eyes:

qat0: <Intel c3xxx QuickAssist> mem 0x81500000-0x8153ffff,0x81540000-0x8157ffff at device 0.0 on pci1
qat0: qat_dev0 started 6 acceleration engines
qat0: FW version: 4.18.0
qat0: Excessive clock measure delay
qat_ocf0: <QAT engine>
irq175: qat0:b1:353 @cpu0(domain0): 790224
irq176: qat0:b2:355 @cpu0(domain0): 659108
dev.qat_ocf.0.%parent: nexus0
dev.qat_ocf.0.%pnpinfo:
dev.qat_ocf.0.%location:
dev.qat_ocf.0.%driver: qat_ocf
dev.qat_ocf.0.%desc: QAT engine
dev.qat_ocf.%parent:
dev.qat.0.frequency: 685000000
dev.qat.0.cnv_error:
dev.qat.0.fw_counters:
dev.qat.0.mmp_version: 6.0.0
dev.qat.0.hw_version: 17
dev.qat.0.fw_version: 4.18.0
dev.qat.0.heartbeat: 1
dev.qat.0.heartbeat_failed: 0
dev.qat.0.heartbeat_sent: 7
dev.qat.0.dev_cfg: [GENERAL]
dev.qat.0.%parent: pci1
dev.qat.0.%pnpinfo: vendor=0x8086 device=0x19e2 subvendor=0x8086 subdevice=0x19e2 class=0x0b4000
dev.qat.0.%location: slot=0 function=0 dbsf=pci0:1:0:0 handle=\_SB_.PCI0.VRP2.PXSX
dev.qat.0.%driver: qat
dev.qat.0.%desc: Intel c3xxx QuickAssist
dev.qat.%parent:

Is the openssl qatengine supposed to be located somewhere?

️

jaltman

@RobbieTT said in 23.09d - Is QAT Broken?:

Is the openssl qatengine supposed to be located somewhere?

Its the openssl qatengine that is missing. From the following output:

[23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root: openssl engine -t -c -v qatengine
0020E1AF5B420000:error:12800067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/engines-3/qatengine.so): Cannot open "/usr/lib/engines-3/qatengine.so"
0020E1AF5B420000:error:12800067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/dso/dso_lib.c:152:
0020E1AF5B420000:error:13000084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/engine/eng_dyn.c:442:
0020E1AF5B420000:error:13000074:engine routines:ENGINE_by_id:no such engine:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/engine/eng_list.c:430:id=qatengine
[23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root:

the qatengine.so is expected to be in /usr/lib/engines-3 for openssl 3.0.x. But that directory only includes

[23.09-DEVELOPMENT][root@pfsense.bayside.sara-jeff.nyc]/: ls /usr/lib/engines-3
capi.so         devcrypto.so    loader_attic.so padlock.so

I think openssl 3.x has to be built with the QAT engine support enabled to build qatengine.so and then I think there is something that needs to be added to /etc/ssl/openssl.conf to load it.

RobbieTT

@jaltman Thanks for that and on reading some of the Intel guide it looks like there are a few different ways of (fully) enabling QAT on FreeBSD and I cannot find the adf_ctl utility on pfSense to interact with user space QAT.

Reading the Intel guides is not easy as the bulk of the BSD information is in the Linux guide (other OS) with a much thinner document for BSD specifics. pfSense use is more opaque as I cannot find anything substantive as to what QAT functionality they are using from FreeBSD, especially as different packages and plugins can be added or expected to be added to FreeBSD itself to achieve full capability.

I am well-beyond my comfort zone and understanding here but suffice to say that pfSense does not employ all the capabilities QAT can provide or expected to provide. The Intel guides seem to assume that all capabilities would be exposed or used if the hardware is in place.

️

jaltman

@RobbieTT said in 23.09d - Is QAT Broken?:

but suffice to say that pfSense does not employ all the capabilities QAT can provide or expected to provide

As far as I'm concerned QAT for userspace is a feature request. I would like to see it but I can also appreciate it being a low priority for Netgate.
The building block cryptographic algorithms that the QAT hardware provides is fairly inclusive but the OpenSSL QAT Engine only uses them to implement a subset of the algorithms supported by OpenSSL. Unless the userspace application is using one of the implemented algorithms there is no QAT benefit. As an example, the OpenSSL QAT engine would provide no benefit for a Kerberos KDC or anything that uses GSS-API integrity protection and/or privacy modes.

I would expect there to be a benefit for browser connections to the pfSense dashboard as my Firefox 118 to pfSense 23.09-dev connection is TLS 1.3 with TLS_AES_256_GCM_SHA384 which can be optimized using QAT. Likewise there are many cipher and mac algorithms supported by OpenSSH 9.4p1 which could benefit from QAT. The question is how much traffic would a pfSense router typically process that would benefit from QAT?

I don't know the answer to that question. For 23.09 I would simply request that the pfSense documentation regarding the selection of IPSec-MB and the various Cryptographic Hardware options be improved.