23.09d - Is QAT Broken?

RobbieTT

Not much different on today's 23.09d. I don't know if it is significant error or not:

[23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root: openssl engine
(rdrand) Intel RDRAND engine
(dynamic) Dynamic engine loading support
[23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root: 

[23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root: openssl engine -t -c -v qatengine
0020E1AF5B420000:error:12800067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/engines-3/qatengine.so): Cannot open "/usr/lib/engines-3/qatengine.so"
0020E1AF5B420000:error:12800067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/dso/dso_lib.c:152:
0020E1AF5B420000:error:13000084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/engine/eng_dyn.c:442:
0020E1AF5B420000:error:13000074:engine routines:ENGINE_by_id:no such engine:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/engine/eng_list.c:430:id=qatengine
[23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root:

️

NRgia

@RobbieTT If on a development snapshot I can understand to happen, why I am also getting this on 23.05.1 which is a production version...I know you don't have the answers, but maybe someone will help with a reply.

jaltman

@NRgia said in 23.09d - Is QAT Broken?:

I'm getting the same error as you on a C3xxx board runnig pfSense+ 23.05.1. Is it normal, should we ignore this?

As far as I can tell the openssl qatengine is not built for pfSense and has never been. Therefore it is normal that these commands will fail.
If the openssl qatengine were present then nginx, curl, sshd, and ssh and other userspace tools would be able to leverage the QAT hardware.

@jimp has said that QAT is only expected to be enabled for kernel functions. The kernel QAT support does not rely upon the openssl qatengine.

These failures can be ignored.

NRgia

@jaltman Thanks, I feel releaved now.

RobbieTT

@jaltman said in 23.09d - Is QAT Broken?:

@jimp has said that QAT is only expected to be enabled for kernel functions. The kernel QAT support does not rely upon the openssl qatengine.

We still have the open question as to why pre 23.09d the QAT interrupts increment concurrently with on-device TLS/SSL type tasks.

I would be surprised if Netgate deliberately neutered the QAT support in the latest freeBSD offerings, given their close ties with upstream. They push QAT as a feature for pfSense+ so it would make more sense to unleash it wherever possible.

️

jimp

That kind of FUD is completely uncalled for. Netgate didn't reduce the functionality of the driver at all, we have put significant resources into its development.

eracerxrs

I am encountering a similar problem with QAT not working on a fresh install, but on 23.05.01.

I started a post before I found this one here.

Similar to OP here is my dmesg and vmstat output:

qat0: <Intel 200xx QuickAssist> mem 0xfe600000-0xfe63ffff,0xfe640000-0xfe67ffff irq 16 at device 0.0 on pci2
qat0: qat_dev0 started 6 acceleration engines
qat0: FW version: 4.18.0
qat0: Excessive clock measure delay
qat_ocf0: <QAT engine>
[23.05.1-RELEASE][admin@pfSense.home.arpa]/root: vmstat -i | grep qat
[23.05.1-RELEASE][admin@pfSense.home.arpa]/root:

I have not configured any VPNs as I was trying to get the dashboard QAT Crypto status to change to YES as a first step.

I'm a QAT newb, so it's possible I have overlooked something simple...

RobbieTT

@jimp said in 23.09d - Is QAT Broken?:

That kind of FUD is completely uncalled for. Netgate didn't reduce the functionality of the driver at all, we have put significant resources into its development.

@jimp I'm not sure if you are aiming at @jaltman (who was merely repeating your words) or myself, when I expressed surprise if Netgate's desire was to limit QAT functionality, especially as you push QAT as a feature. I was expressing doubt that Netgate would do this.

Jim, you have been PA or a bit combative on this issue for no real reason that I can see. You have spun my questions back on me by asking me what traffic, TLS/algorithms do I expect to be accelerated and even misstated my questions as statements, which you then baulk at. Meanwhile the original, simple A vs B question remains unexplained and sidestepped.

I have run all the tests you have asked for, checked all the configurations that you requested, spent hours booting in and out of pfSense versions that have only reinforced the original query. Simply put, what is show as accelerated by QAT in 23.05 is not in 23.09d. This is customer feedback on a technical issue that has arisen. Can we just sidestep the emotive and go back to 'the data is the data'?

For clarity:

I do understand that your instance of 23.05 is not triggering QAT interrupts for the traffic types I have given. We need to understand why those of us above have a different experience to yours.
I understand that you do not think QAT should be active on pfSense with SSH, nginx, curl, TLS/SSL, openSSL etc. This would equate to reduced feature-set from that stated in the Intel / freeBSD QAT documentation. That it appeared to work on 23.05 is in doubt as you have opined that this may be false reporting.
I understand that your current thinking is that the only things that would use QAT on pfSense+ is in the kernel space; more specifically only IPsec and OpenVPN DCO.
I understand that you do not expect pfSense+ to utilise QAT for any daemons or user-space. This reduced QAT functionality on pfSense+ would explain what I and others are observing on 23.09d (albeit not fully explaining the interrupts reported on 23.05).
However, you have also stated that Netgate has not reduced the Intel QAT functionality at all. This appears to be a contradiction to the bullet above.

Perhaps we are divided by a common language and the barrier of the written word but I really am investing time and effort to understand this and have been drawn down into a depth of the system that I don't think I should be as a customer.

So far we have learned that the Intel QAT / freeBSD implementation on the latest pfSense+ appears to be missing all functionality save for those executed through the kernel. This has (apparently) excluded all QAT user space functions including those frameworks directly enabled by Intel (eg OpenSSL, libcrypto etc) or via the QAT API (compression / decompression, SSL, TLS, nginx et al) and the QAT User Space Additional Functions.

I understand that the attempt to force a user space QAT driver on your device produced an error stating it only works on 4xxx QAT devices. The Intel QAT software release for freeBSD (which includes QAT user space support) makes no reference to the 4xxx QAT devices; it states that:

This software release is intended for platforms that contain:
• Intel C62x Chipset
• Intel Atom C3000 processor product family
• Intel QuickAssist Adapter 8960/ Intel QuickAssist Adapter 8970 (formerly known as "Lewis Hill")
• Intel Communications Chipset 8925 to 8955 Series
• Intel Atom P5300 processor product family

Refs: Package Version: QAT.B.3.12.0-00004 - June 2022 & GitHub - Intel - Asynch Mode for NGINX

@jimp I am sure you can see why some of us are confused as to the functionality of QAT in pfSense+ given the apparent (or at least appearance of) technical contradictions. This is not an attack on Netgate devs. We either have full QAT functionality on the C3xxx platforms or we don't. If we don't then this may be due to sound technical reasons, an error or oversight, a bug or just work in progress.

Regards, Rob

️

jaltman

@RobbieTT said in 23.09d - Is QAT Broken?:

I understand that you do not think QAT should be active on pfSense with SSH, nginx, curl, TLS/SSL, openSSL etc. This would equate to reduced feature-set from that stated in the Intel / freeBSD QAT documentation. That it appeared to work on 23.05 is in doubt as you have opined that this may be false reporting.

To be honest, I don't understand why QAT would be active for ssh, sshd, nginx, curl, or anything else linked against openssl's libcrypto when the openssl qatengine is not present on either 23.05.1 or 23.09-dev. There is no driver installed that exposes QAT to userspace nor is there a userspace library to call it. All of the above processes are linked to openssl's libcrypto. In 23.05.1 its an openssl 1.1.x library and in 23.09-dev its an openssl 3.0.x library but in neither case would I expect QAT to be used.

The QAT interrupts we are seeing must be coming from some kernel packet processing. I've tried obtaining a packet capture for the WAN and separately for the LAN while doing various things but there aren't any packets that jump out at me as something that would use QAT. I'm almost wondering if there is something from the WAN that appears to be attempting to establish a tunnel that doesn't exist and perhaps that is triggering the QAT activity with 23.05.1 but in 23.09-dev the trigger in 23.09-dev is correctly filtered out.

I'm not worried that QAT is not being used in 23.09-dev for userspace because I don't think it was being used for userspace in 23.05.1. However, I would like it to be used for userspace in the future. I would also appreciate it if the Netgate pfSense documentation was a bit more specific about when QAT can be used and when it cannot. The text on System->Advanced->Miscellaneous page doesn't explicitly mention QAT.

A cryptographic accelerator module will use hardware support to speed up some cryptographic functions on systems which have the chip. Loading the BSD Crypto Device module will allow access to acceleration devices using drivers built into the kernel, such as Hifn or ubsec chipsets. If the firewall does not contain a crypto chip, this option will have no effect. To unload the selected module, set this option to "none" and then reboot.

RobbieTT

@jaltman said in 23.09d - Is QAT Broken?:

...the openssl qatengine is not present on either 23.05.1 or 23.09-dev. There is no driver installed that exposes QAT to userspace nor is there a userspace library to call it.

So what do you think we are missing? The kernel files on the Intel documents all appear to be in place on pfSense, including the common and API:

/boot/kernel/qat_4xxx_fw.ko
/boot/kernel/qat_dh895xcc_fw.ko
/boot/kernel/qat_hw.ko
/boot/kernel/qat_c2xxxfw.ko
/boot/kernel/qat_c4xxx_fw.ko
/boot/kernel/qat_common.ko
/boot/kernel/qat_api.ko
/boot/kernel/qat_c3xxx_fw.ko
/boot/kernel/qat_c2xxx.ko
/boot/kernel/qat_c62x_fw.ko
/boot/kernel/qat.ko
/boot/kernel/qat_200xx_fw.ko

The QAT engine is there and nothing stands out as missing, at least to my eyes:

qat0: <Intel c3xxx QuickAssist> mem 0x81500000-0x8153ffff,0x81540000-0x8157ffff at device 0.0 on pci1
qat0: qat_dev0 started 6 acceleration engines
qat0: FW version: 4.18.0
qat0: Excessive clock measure delay
qat_ocf0: <QAT engine>
irq175: qat0:b1:353 @cpu0(domain0): 790224
irq176: qat0:b2:355 @cpu0(domain0): 659108
dev.qat_ocf.0.%parent: nexus0
dev.qat_ocf.0.%pnpinfo:
dev.qat_ocf.0.%location:
dev.qat_ocf.0.%driver: qat_ocf
dev.qat_ocf.0.%desc: QAT engine
dev.qat_ocf.%parent:
dev.qat.0.frequency: 685000000
dev.qat.0.cnv_error:
dev.qat.0.fw_counters:
dev.qat.0.mmp_version: 6.0.0
dev.qat.0.hw_version: 17
dev.qat.0.fw_version: 4.18.0
dev.qat.0.heartbeat: 1
dev.qat.0.heartbeat_failed: 0
dev.qat.0.heartbeat_sent: 7
dev.qat.0.dev_cfg: [GENERAL]
dev.qat.0.%parent: pci1
dev.qat.0.%pnpinfo: vendor=0x8086 device=0x19e2 subvendor=0x8086 subdevice=0x19e2 class=0x0b4000
dev.qat.0.%location: slot=0 function=0 dbsf=pci0:1:0:0 handle=\_SB_.PCI0.VRP2.PXSX
dev.qat.0.%driver: qat
dev.qat.0.%desc: Intel c3xxx QuickAssist
dev.qat.%parent:

Is the openssl qatengine supposed to be located somewhere?

️

jaltman

@RobbieTT said in 23.09d - Is QAT Broken?:

Is the openssl qatengine supposed to be located somewhere?

Its the openssl qatengine that is missing. From the following output:

[23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root: openssl engine -t -c -v qatengine
0020E1AF5B420000:error:12800067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/engines-3/qatengine.so): Cannot open "/usr/lib/engines-3/qatengine.so"
0020E1AF5B420000:error:12800067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/dso/dso_lib.c:152:
0020E1AF5B420000:error:13000084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/engine/eng_dyn.c:442:
0020E1AF5B420000:error:13000074:engine routines:ENGINE_by_id:no such engine:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/engine/eng_list.c:430:id=qatengine
[23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root:

the qatengine.so is expected to be in /usr/lib/engines-3 for openssl 3.0.x. But that directory only includes

[23.09-DEVELOPMENT][root@pfsense.bayside.sara-jeff.nyc]/: ls /usr/lib/engines-3
capi.so         devcrypto.so    loader_attic.so padlock.so

I think openssl 3.x has to be built with the QAT engine support enabled to build qatengine.so and then I think there is something that needs to be added to /etc/ssl/openssl.conf to load it.

RobbieTT

@jaltman Thanks for that and on reading some of the Intel guide it looks like there are a few different ways of (fully) enabling QAT on FreeBSD and I cannot find the adf_ctl utility on pfSense to interact with user space QAT.

Reading the Intel guides is not easy as the bulk of the BSD information is in the Linux guide (other OS) with a much thinner document for BSD specifics. pfSense use is more opaque as I cannot find anything substantive as to what QAT functionality they are using from FreeBSD, especially as different packages and plugins can be added or expected to be added to FreeBSD itself to achieve full capability.

I am well-beyond my comfort zone and understanding here but suffice to say that pfSense does not employ all the capabilities QAT can provide or expected to provide. The Intel guides seem to assume that all capabilities would be exposed or used if the hardware is in place.

️

jaltman

@RobbieTT said in 23.09d - Is QAT Broken?:

but suffice to say that pfSense does not employ all the capabilities QAT can provide or expected to provide

As far as I'm concerned QAT for userspace is a feature request. I would like to see it but I can also appreciate it being a low priority for Netgate.
The building block cryptographic algorithms that the QAT hardware provides is fairly inclusive but the OpenSSL QAT Engine only uses them to implement a subset of the algorithms supported by OpenSSL. Unless the userspace application is using one of the implemented algorithms there is no QAT benefit. As an example, the OpenSSL QAT engine would provide no benefit for a Kerberos KDC or anything that uses GSS-API integrity protection and/or privacy modes.

I would expect there to be a benefit for browser connections to the pfSense dashboard as my Firefox 118 to pfSense 23.09-dev connection is TLS 1.3 with TLS_AES_256_GCM_SHA384 which can be optimized using QAT. Likewise there are many cipher and mac algorithms supported by OpenSSH 9.4p1 which could benefit from QAT. The question is how much traffic would a pfSense router typically process that would benefit from QAT?

I don't know the answer to that question. For 23.09 I would simply request that the pfSense documentation regarding the selection of IPSec-MB and the various Cryptographic Hardware options be improved.

stephenw10

@jaltman said in 23.09d - Is QAT Broken?:

The question is how much traffic would a pfSense router typically process that would benefit from QAT?

It would only benefit traffic to or from the firewall directly. So unless you are using an ssh tunnel to the firewall and passing a lot of traffic through it I doubt you would see any difference with QAT enabled. Though it would still be nice to have.

RobbieTT

@stephenw10
Plus (presumably) any external resources used by any service or package riding on pfSense or indeed by pfSense itself. You can probably add things like DNS-over-TLS as another common use to the list too. The key point being that traffic from/to the firewall itself should use QAT, rather than limiting its use to just external clients using a VPN.

️

stephenw10

Mmm, yes DoT is a good point. That could be significant. Though the actual amount of data is pretty small. It would be interesting to look at that usage. It could be argued that if you have enough DNS traffic to make an impact you should probably be using a dedicated DNS server.

RobbieTT

@stephenw10 said in 23.09d - Is QAT Broken?:

Mmm, yes DoT is a good point. That could be significant.

I don't want over-egg the pudding too much as it's only a factor and really we are talking about lightening the load on a CPU, or in our case a core. I think the individual things, such as DoT, probably only really matter when combined with all the other little things.

Dedicated silicone / accelerators work faster and with less power than pulling things through a core, as well as giving cores more capacity for the stuff they have to do. There is certainly little point leaving QAT idle when it could be put to use; well, in my view. QAT is one of things that attracted me to Netgate / pfSense+.

️

jaltman

@RobbieTT Does anyone know if FreeBSD builds and packages the openssl3 qaengine for FreeBSD 14? If so, perhaps it can be easily pulled into pfSense or turned into a pfSense package that can be optionally installed.

RobbieTT

@jaltman
Unsure as the Intel documentation for BSD seems to top-out at BSD 13.1. With QAT functionality as we know it was only embraced with 13.0 that part of the document set is quite undeveloped (at least on the versions I can find - there may be updated docs hiding somewhere).

There was a significant change in QAT capabilities in freeBSD between 13.1 and 14.0:

freeBSD 13.1

DESCRIPTION
       The  qat	 driver	 implements  crypto(4) support for some	of the crypto-
       graphic acceleration functions of the Intel QuickAssist	(QAT)  device.
       The  qat	driver supports	the QAT	devices	integrated with	Atom C2000 and
       C3000 and Xeon C620 and D-1500 platforms, and  the  Intel  QAT  Adapter
       8950.   Other  platforms	 and adapters not listed here may also be sup-
       ported.	QAT devices are	enumerated through PCIe	and are	 thus  visible
       in pciconf(8) output.

       The  qat	 driver	 can  accelerate  AES in CBC, CTR, XTS (except for the
       C2000) and GCM modes, and can perform authenticated encryption  combin-
       ing  the	 CBC, CTR and XTS modes	with SHA1-HMAC and SHA2-HMAC.  The qat
       driver can also compute SHA1 and	SHA2 digests.  The  implementation  of
       AES-GCM	has a firmware-imposed constraint that the length of any addi-
       tional authenticated data (AAD) must not	exceed 240 bytes.  The	driver
       thus rejects crypto(9) requests that do not satisfy this	constraint.

freeBSD 14.0

DESCRIPTION
       The qat driver supports cryptography and	 compression  acceleration  of
       the Intel (R) QuickAssist Technology (QAT) devices.

       The qat driver is intended for platforms	that contain:
       o   Intel (R) C62x Chipset
       o   Intel (R) Atom C3000	processor product family
       o   Intel  (R)  QuickAssist  Adapter 8960/Intel (R) QuickAssist Adapter
	   8970	(formerly known	as "Lewis Hill")
       o   Intel (R) Communications Chipset 8925 to 8955 Series
       o   Intel (R) Atom P5300	processor product family
       o   Intel (R) QAT 4xxx Series

       The qat driver supports cryptography and	compression  acceleration.   A
       complete	 API  for offloading these operations is exposed in the	kernel
       and may be used by any other entity directly.  For details of usage and
       supported operations and	algorithms refer to the	 following  documenta-
       tion available from 01.org:
       o   Intel (R), QuickAssist Technology API Programmer's Guide.
       o   Intel  (R),	QuickAssist  Technology	 Cryptographic	API  Reference
	   Manual.
       o   Intel (R), QuickAssist Technology Data  Compression	API  Reference
	   Manual.
       o   Intel (R), QuickAssist Technology Performance Optimization Guide.

       In addition to exposing complete	kernel API for offloading cryptography
       and  compression	 operations,  the  qat	driver	also  integrates  with
       crypto(4), allowing offloading supported	cryptography operations	to In-
       tel (R) QuickAssist Technology (QAT) devices.  For details of usage and
       supported operations and	algorithms refer  to  the  documentation  men-
       tioned above and	"SEE ALSO" section.

So it appears that 13.1 was limited to 'some' kernel cryptographics with only 14.0 unleashing full QAT and exposing all of the API for use by other entities (even including compression/decompression, gzip, QATzip etc).

With pfSense+ leaping directly to freeBSD 14.0 the reduced feature set of 13.1+ should not be a factor but as to what is missing from pfSense+ to make use of the more expansive set of BSD 14.0 capabilities is unclear to me. Indeed, it looks like pfSense+ went to the effort of including all the upstream BSD files needed to run the complete set of QAT capabilities.

It's why I wasn't surprised to see QAT apparently working in 23.05.1 and why I assumed an error prevented it working in 23.09d. Now I just don't have a clue as to what is or isn't intended for pfSense+.

️

jaltman

@RobbieTT all of that is discussing the kernel. It says nothing about OpenSSL and without the OpenSSL qatengine there can be no use of QAT for SSL/TLS, SSH or any other application or protocol implemented in user space which relies on libcrypto for cryptographic algorithms.

Until FreeBSD ships the OpenSSL QAT engine I would not expect to see it in pfsense.