23.09d - Is QAT Broken?
-
@RobbieTT said in 23.09d - Is QAT Broken?:
I understand that you do not think QAT should be active on pfSense with SSH, nginx, curl, TLS/SSL, openSSL etc. This would equate to reduced feature-set from that stated in the Intel / freeBSD QAT documentation. That it appeared to work on 23.05 is in doubt as you have opined that this may be false reporting.
To be honest, I don't understand why QAT would be active for ssh, sshd, nginx, curl, or anything else linked against openssl's libcrypto when the openssl qatengine is not present on either 23.05.1 or 23.09-dev. There is no driver installed that exposes QAT to userspace nor is there a userspace library to call it. All of the above processes are linked to openssl's libcrypto. In 23.05.1 its an openssl 1.1.x library and in 23.09-dev its an openssl 3.0.x library but in neither case would I expect QAT to be used.
The QAT interrupts we are seeing must be coming from some kernel packet processing. I've tried obtaining a packet capture for the WAN and separately for the LAN while doing various things but there aren't any packets that jump out at me as something that would use QAT. I'm almost wondering if there is something from the WAN that appears to be attempting to establish a tunnel that doesn't exist and perhaps that is triggering the QAT activity with 23.05.1 but in 23.09-dev the trigger in 23.09-dev is correctly filtered out.
I'm not worried that QAT is not being used in 23.09-dev for userspace because I don't think it was being used for userspace in 23.05.1. However, I would like it to be used for userspace in the future. I would also appreciate it if the Netgate pfSense documentation was a bit more specific about when QAT can be used and when it cannot. The text on System->Advanced->Miscellaneous page doesn't explicitly mention QAT.
A cryptographic accelerator module will use hardware support to speed up some cryptographic functions on systems which have the chip. Loading the BSD Crypto Device module will allow access to acceleration devices using drivers built into the kernel, such as Hifn or ubsec chipsets. If the firewall does not contain a crypto chip, this option will have no effect. To unload the selected module, set this option to "none" and then reboot. -
@jaltman said in 23.09d - Is QAT Broken?:
...the openssl qatengine is not present on either 23.05.1 or 23.09-dev. There is no driver installed that exposes QAT to userspace nor is there a userspace library to call it.
So what do you think we are missing? The kernel files on the Intel documents all appear to be in place on pfSense, including the common and API:
/boot/kernel/qat_4xxx_fw.ko /boot/kernel/qat_dh895xcc_fw.ko /boot/kernel/qat_hw.ko /boot/kernel/qat_c2xxxfw.ko /boot/kernel/qat_c4xxx_fw.ko /boot/kernel/qat_common.ko /boot/kernel/qat_api.ko /boot/kernel/qat_c3xxx_fw.ko /boot/kernel/qat_c2xxx.ko /boot/kernel/qat_c62x_fw.ko /boot/kernel/qat.ko /boot/kernel/qat_200xx_fw.koThe QAT engine is there and nothing stands out as missing, at least to my eyes:
qat0: <Intel c3xxx QuickAssist> mem 0x81500000-0x8153ffff,0x81540000-0x8157ffff at device 0.0 on pci1 qat0: qat_dev0 started 6 acceleration engines qat0: FW version: 4.18.0 qat0: Excessive clock measure delay qat_ocf0: <QAT engine> irq175: qat0:b1:353 @cpu0(domain0): 790224 irq176: qat0:b2:355 @cpu0(domain0): 659108 dev.qat_ocf.0.%parent: nexus0 dev.qat_ocf.0.%pnpinfo: dev.qat_ocf.0.%location: dev.qat_ocf.0.%driver: qat_ocf dev.qat_ocf.0.%desc: QAT engine dev.qat_ocf.%parent: dev.qat.0.frequency: 685000000 dev.qat.0.cnv_error: dev.qat.0.fw_counters: dev.qat.0.mmp_version: 6.0.0 dev.qat.0.hw_version: 17 dev.qat.0.fw_version: 4.18.0 dev.qat.0.heartbeat: 1 dev.qat.0.heartbeat_failed: 0 dev.qat.0.heartbeat_sent: 7 dev.qat.0.dev_cfg: [GENERAL] dev.qat.0.%parent: pci1 dev.qat.0.%pnpinfo: vendor=0x8086 device=0x19e2 subvendor=0x8086 subdevice=0x19e2 class=0x0b4000 dev.qat.0.%location: slot=0 function=0 dbsf=pci0:1:0:0 handle=\_SB_.PCI0.VRP2.PXSX dev.qat.0.%driver: qat dev.qat.0.%desc: Intel c3xxx QuickAssist dev.qat.%parent:Is the
openssl qatenginesupposed to be located somewhere?
️ -
@RobbieTT said in 23.09d - Is QAT Broken?:
Is the openssl qatengine supposed to be located somewhere?
Its the openssl qatengine that is missing. From the following output:
[23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root: openssl engine -t -c -v qatengine 0020E1AF5B420000:error:12800067:DSO support routines:dlfcn_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/engines-3/qatengine.so): Cannot open "/usr/lib/engines-3/qatengine.so" 0020E1AF5B420000:error:12800067:DSO support routines:DSO_load:could not load the shared library:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/dso/dso_lib.c:152: 0020E1AF5B420000:error:13000084:engine routines:dynamic_load:dso not found:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/engine/eng_dyn.c:442: 0020E1AF5B420000:error:13000074:engine routines:ENGINE_by_id:no such engine:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/crypto/openssl/crypto/engine/eng_list.c:430:id=qatengine [23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root:the qatengine.so is expected to be in /usr/lib/engines-3 for openssl 3.0.x. But that directory only includes
[23.09-DEVELOPMENT][root@pfsense.bayside.sara-jeff.nyc]/: ls /usr/lib/engines-3 capi.so devcrypto.so loader_attic.so padlock.soI think openssl 3.x has to be built with the QAT engine support enabled to build qatengine.so and then I think there is something that needs to be added to /etc/ssl/openssl.conf to load it.
-
@jaltman Thanks for that and on reading some of the Intel guide it looks like there are a few different ways of (fully) enabling QAT on FreeBSD and I cannot find the
adf_ctlutility on pfSense to interact with user space QAT.Reading the Intel guides is not easy as the bulk of the BSD information is in the Linux guide (other OS) with a much thinner document for BSD specifics. pfSense use is more opaque as I cannot find anything substantive as to what QAT functionality they are using from FreeBSD, especially as different packages and plugins can be added or expected to be added to FreeBSD itself to achieve full capability.
I am well-beyond my comfort zone and understanding here but suffice to say that pfSense does not employ all the capabilities QAT can provide or expected to provide. The Intel guides seem to assume that all capabilities would be exposed or used if the hardware is in place.
️ -
@RobbieTT said in 23.09d - Is QAT Broken?:
but suffice to say that pfSense does not employ all the capabilities QAT can provide or expected to provide
As far as I'm concerned QAT for userspace is a feature request. I would like to see it but I can also appreciate it being a low priority for Netgate.
The building block cryptographic algorithms that the QAT hardware provides is fairly inclusive but the OpenSSL QAT Engine only uses them to implement a subset of the algorithms supported by OpenSSL. Unless the userspace application is using one of the implemented algorithms there is no QAT benefit. As an example, the OpenSSL QAT engine would provide no benefit for a Kerberos KDC or anything that uses GSS-API integrity protection and/or privacy modes.I would expect there to be a benefit for browser connections to the pfSense dashboard as my Firefox 118 to pfSense 23.09-dev connection is TLS 1.3 with TLS_AES_256_GCM_SHA384 which can be optimized using QAT. Likewise there are many cipher and mac algorithms supported by OpenSSH 9.4p1 which could benefit from QAT. The question is how much traffic would a pfSense router typically process that would benefit from QAT?
I don't know the answer to that question. For 23.09 I would simply request that the pfSense documentation regarding the selection of IPSec-MB and the various Cryptographic Hardware options be improved.
-
R RobbieTT referenced this topic on
-
@jaltman said in 23.09d - Is QAT Broken?:
The question is how much traffic would a pfSense router typically process that would benefit from QAT?
It would only benefit traffic to or from the firewall directly. So unless you are using an ssh tunnel to the firewall and passing a lot of traffic through it I doubt you would see any difference with QAT enabled. Though it would still be nice to have.
-
@stephenw10
Plus (presumably) any external resources used by any service or package riding on pfSense or indeed by pfSense itself. You can probably add things like DNS-over-TLS as another common use to the list too. The key point being that traffic from/to the firewall itself should use QAT, rather than limiting its use to just external clients using a VPN.️ -
Mmm, yes DoT is a good point. That could be significant. Though the actual amount of data is pretty small. It would be interesting to look at that usage. It could be argued that if you have enough DNS traffic to make an impact you should probably be using a dedicated DNS server.
-
@stephenw10 said in 23.09d - Is QAT Broken?:
Mmm, yes DoT is a good point. That could be significant.
I don't want over-egg the pudding too much as it's only a factor and really we are talking about lightening the load on a CPU, or in our case a core. I think the individual things, such as DoT, probably only really matter when combined with all the other little things.
Dedicated silicone / accelerators work faster and with less power than pulling things through a core, as well as giving cores more capacity for the stuff they have to do. There is certainly little point leaving QAT idle when it could be put to use; well, in my view. QAT is one of things that attracted me to Netgate / pfSense+.
️ -
@RobbieTT Does anyone know if FreeBSD builds and packages the openssl3 qaengine for FreeBSD 14? If so, perhaps it can be easily pulled into pfSense or turned into a pfSense package that can be optionally installed.
-
@jaltman
Unsure as the Intel documentation for BSD seems to top-out at BSD 13.1. With QAT functionality as we know it was only embraced with 13.0 that part of the document set is quite undeveloped (at least on the versions I can find - there may be updated docs hiding somewhere).There was a significant change in QAT capabilities in freeBSD between 13.1 and 14.0:
freeBSD 13.1
DESCRIPTION The qat driver implements crypto(4) support for some of the crypto- graphic acceleration functions of the Intel QuickAssist (QAT) device. The qat driver supports the QAT devices integrated with Atom C2000 and C3000 and Xeon C620 and D-1500 platforms, and the Intel QAT Adapter 8950. Other platforms and adapters not listed here may also be sup- ported. QAT devices are enumerated through PCIe and are thus visible in pciconf(8) output. The qat driver can accelerate AES in CBC, CTR, XTS (except for the C2000) and GCM modes, and can perform authenticated encryption combin- ing the CBC, CTR and XTS modes with SHA1-HMAC and SHA2-HMAC. The qat driver can also compute SHA1 and SHA2 digests. The implementation of AES-GCM has a firmware-imposed constraint that the length of any addi- tional authenticated data (AAD) must not exceed 240 bytes. The driver thus rejects crypto(9) requests that do not satisfy this constraint.freeBSD 14.0
DESCRIPTION The qat driver supports cryptography and compression acceleration of the Intel (R) QuickAssist Technology (QAT) devices. The qat driver is intended for platforms that contain: o Intel (R) C62x Chipset o Intel (R) Atom C3000 processor product family o Intel (R) QuickAssist Adapter 8960/Intel (R) QuickAssist Adapter 8970 (formerly known as "Lewis Hill") o Intel (R) Communications Chipset 8925 to 8955 Series o Intel (R) Atom P5300 processor product family o Intel (R) QAT 4xxx Series The qat driver supports cryptography and compression acceleration. A complete API for offloading these operations is exposed in the kernel and may be used by any other entity directly. For details of usage and supported operations and algorithms refer to the following documenta- tion available from 01.org: o Intel (R), QuickAssist Technology API Programmer's Guide. o Intel (R), QuickAssist Technology Cryptographic API Reference Manual. o Intel (R), QuickAssist Technology Data Compression API Reference Manual. o Intel (R), QuickAssist Technology Performance Optimization Guide. In addition to exposing complete kernel API for offloading cryptography and compression operations, the qat driver also integrates with crypto(4), allowing offloading supported cryptography operations to In- tel (R) QuickAssist Technology (QAT) devices. For details of usage and supported operations and algorithms refer to the documentation men- tioned above and "SEE ALSO" section.So it appears that 13.1 was limited to 'some' kernel cryptographics with only 14.0 unleashing full QAT and exposing all of the API for use by other entities (even including compression/decompression, gzip, QATzip etc).
With pfSense+ leaping directly to freeBSD 14.0 the reduced feature set of 13.1+ should not be a factor but as to what is missing from pfSense+ to make use of the more expansive set of BSD 14.0 capabilities is unclear to me. Indeed, it looks like pfSense+ went to the effort of including all the upstream BSD files needed to run the complete set of QAT capabilities.
It's why I wasn't surprised to see QAT apparently working in 23.05.1 and why I assumed an error prevented it working in 23.09d. Now I just don't have a clue as to what is or isn't intended for pfSense+.
️ -
@RobbieTT all of that is discussing the kernel. It says nothing about OpenSSL and without the OpenSSL qatengine there can be no use of QAT for SSL/TLS, SSH or any other application or protocol implemented in user space which relies on libcrypto for cryptographic algorithms.
Until FreeBSD ships the OpenSSL QAT engine I would not expect to see it in pfsense.
-
@jaltman It opens QAT beyond the kernel via the API - indeed, it directly references the API and user space capabilities. I don't know how they could say it more explicitly than in the quote:
A complete API for offloading these operations is exposed in the kernel and may be used by any other entity directly.
They also give examples of user space functions up to and including compression.
I don't doubt that there is something missing with OpenSSL in pfSense+ but I am not sure we can point the finger at freeBSD 14.0 in its non-pfSense guise.
️(If you have tested freeBSD 14.0 separately and found it to be lacking then please accept my apologies and disregard the above.)
https://github.com/intel/QAT_Engine/blob/master/docs/software_requirements.md
https://man.freebsd.org/cgi/man.cgi?query=qat&apropos=0&sektion=0&manpath=FreeBSD+14.0-STABLE&arch=default&format=html -
@RobbieTT What you are quoting from is the features of the driver. Simply because the driver is present does not mean that applications use it. Most of the applications that you care about nginx, apache, sshd, ssh, curl, etc are all linked against OpenSSL's libcrypto. The QAT support is simply unavailable to them unless OpenSSL is built with the options required to use the QAT engine and if the QAT engine is installed and loaded via the openssl.conf file in use by the application.
I've installed FreeBSD-14.0-BETA4-amd64. openssl is not built with QAT support and the qatengine is not packaged. The FreeBSD Ports Search has alternative builds of openssl but none of them include QAT support.
I think we can put this discussion to bed.
-
@jaltman So it just comes down to the version of OpenSSL being used is not built with QAT support?
I ask because openSSL v3.0.10 is specifically called for in the freeBSD QAT requirements and pfSense uses that very same version:
/root: openssl version OpenSSL 3.0.10 1 Aug 2023 (Library: OpenSSL 3.0.10 1 Aug 2023)️ -
@RobbieTT OpenSSL 3.0 is used by FreeBSD but the QAT Engine and its dependencies (ipp-crypto-mb, ipsec-mb, qatlib) are not part of the base OpenSSL 3.0 build.
For example, on Fedora Linux you need to install
intel-ipp-crypto-mb-1.0.8-3.fc37.x86_64 intel-ipsec-mb-1.4.0-1.fc37.x86_64 qatengine-1.4.0-1.fc37.x86_64 qatlib-23.02.0-1.fc37.x86_64 qatlib-service-23.02.0-1.fc37.x86_64only then can the OpenSSL QAT Engine be used
[jaltman@fc36]$ ls /usr/lib64/engines-3/ afalg.so capi.so libpkcs11.so loader_attic.so padlock.so pkcs11.so qatengine.so [jaltman@fc37]$ openssl engine -t -c -v qatengine QAT_SW - Processor unsupported: AVX512F = 0, VAES = 0, VPCLMULQDQ = 0 (qatengine) Reference implementation of QAT crypto engine(qat_hw & qat_sw) v1.4.0 [RSA, AES-128-CBC-HMAC-SHA256, AES-256-CBC-HMAC-SHA256, ChaCha20-Poly1305, id-aes128-GCM, id-aes192-GCM, id-aes256-GCM, SHA3-256, SHA3-384, SHA3-512, TLS1-PRF, X25519, X448, SM2] [ available ] ENABLE_EXTERNAL_POLLING, POLL, SET_INSTANCE_FOR_THREAD, GET_NUM_OP_RETRIES, SET_MAX_RETRY_COUNT, SET_INTERNAL_POLL_INTERVAL, GET_EXTERNAL_POLLING_FD, ENABLE_EVENT_DRIVEN_POLLING_MODE, GET_NUM_CRYPTO_INSTANCES, DISABLE_EVENT_DRIVEN_POLLING_MODE, SET_EPOLL_TIMEOUT, SET_CRYPTO_SMALL_PACKET_OFFLOAD_THRESHOLD, ENABLE_INLINE_POLLING, ENABLE_HEURISTIC_POLLING, GET_NUM_REQUESTS_IN_FLIGHT, INIT_ENGINE, SET_CONFIGURATION_SECTION_NAME, ENABLE_SW_FALLBACK, HEARTBEAT_POLL, DISABLE_QAT_OFFLOAD, HW_ALGO_BITMAP, SW_ALGO_BITMAPAs far as I can tell there is no qatengine.so packaged for OpenSSL 3.0, 3.1 or 3.2 on FreeBSD 14. Hence it cannot be installed and cannot be used.
-
Mmm, as I read it OpenSSL requires the qat engine module to use it in user mode. Interesting that it does use it in 23.05...
-
@stephenw10 following this thread for a while and that’s the general concern here. Why is this behavior different?
-
It's almost certainly because we moved to OpenSSL 3 and there is fallout from that. Most of that has been resolved. Since user mode encryption off-load is generally not supported this was probably just overlooked. I'll see what I can do when I'm home tomorrow.
-
@stephenw10 thank you. Appreciate the quick response
