23.09d - Is QAT Broken?
-
@stephenw10
Plus (presumably) any external resources used by any service or package riding on pfSense or indeed by pfSense itself. You can probably add things like DNS-over-TLS as another common use to the list too. The key point being that traffic from/to the firewall itself should use QAT, rather than limiting its use to just external clients using a VPN.
️ -
Mmm, yes DoT is a good point. That could be significant. Though the actual amount of data is pretty small. It would be interesting to look at that usage. It could be argued that if you have enough DNS traffic to make an impact you should probably be using a dedicated DNS server.
-
@stephenw10 said in 23.09d - Is QAT Broken?:
Mmm, yes DoT is a good point. That could be significant.
I don't want over-egg the pudding too much as it's only a factor and really we are talking about lightening the load on a CPU, or in our case a core. I think the individual things, such as DoT, probably only really matter when combined with all the other little things.
Dedicated silicone / accelerators work faster and with less power than pulling things through a core, as well as giving cores more capacity for the stuff they have to do. There is certainly little point leaving QAT idle when it could be put to use; well, in my view. QAT is one of things that attracted me to Netgate / pfSense+.
️ -
@RobbieTT Does anyone know if FreeBSD builds and packages the openssl3 qaengine for FreeBSD 14? If so, perhaps it can be easily pulled into pfSense or turned into a pfSense package that can be optionally installed.
-
@jaltman
Unsure as the Intel documentation for BSD seems to top-out at BSD 13.1. With QAT functionality as we know it was only embraced with 13.0 that part of the document set is quite undeveloped (at least on the versions I can find - there may be updated docs hiding somewhere).There was a significant change in QAT capabilities in freeBSD between 13.1 and 14.0:
freeBSD 13.1
DESCRIPTION The qat driver implements crypto(4) support for some of the crypto- graphic acceleration functions of the Intel QuickAssist (QAT) device. The qat driver supports the QAT devices integrated with Atom C2000 and C3000 and Xeon C620 and D-1500 platforms, and the Intel QAT Adapter 8950. Other platforms and adapters not listed here may also be sup- ported. QAT devices are enumerated through PCIe and are thus visible in pciconf(8) output. The qat driver can accelerate AES in CBC, CTR, XTS (except for the C2000) and GCM modes, and can perform authenticated encryption combin- ing the CBC, CTR and XTS modes with SHA1-HMAC and SHA2-HMAC. The qat driver can also compute SHA1 and SHA2 digests. The implementation of AES-GCM has a firmware-imposed constraint that the length of any addi- tional authenticated data (AAD) must not exceed 240 bytes. The driver thus rejects crypto(9) requests that do not satisfy this constraint.freeBSD 14.0
DESCRIPTION The qat driver supports cryptography and compression acceleration of the Intel (R) QuickAssist Technology (QAT) devices. The qat driver is intended for platforms that contain: o Intel (R) C62x Chipset o Intel (R) Atom C3000 processor product family o Intel (R) QuickAssist Adapter 8960/Intel (R) QuickAssist Adapter 8970 (formerly known as "Lewis Hill") o Intel (R) Communications Chipset 8925 to 8955 Series o Intel (R) Atom P5300 processor product family o Intel (R) QAT 4xxx Series The qat driver supports cryptography and compression acceleration. A complete API for offloading these operations is exposed in the kernel and may be used by any other entity directly. For details of usage and supported operations and algorithms refer to the following documenta- tion available from 01.org: o Intel (R), QuickAssist Technology API Programmer's Guide. o Intel (R), QuickAssist Technology Cryptographic API Reference Manual. o Intel (R), QuickAssist Technology Data Compression API Reference Manual. o Intel (R), QuickAssist Technology Performance Optimization Guide. In addition to exposing complete kernel API for offloading cryptography and compression operations, the qat driver also integrates with crypto(4), allowing offloading supported cryptography operations to In- tel (R) QuickAssist Technology (QAT) devices. For details of usage and supported operations and algorithms refer to the documentation men- tioned above and "SEE ALSO" section.So it appears that 13.1 was limited to 'some' kernel cryptographics with only 14.0 unleashing full QAT and exposing all of the API for use by other entities (even including compression/decompression, gzip, QATzip etc).
With pfSense+ leaping directly to freeBSD 14.0 the reduced feature set of 13.1+ should not be a factor but as to what is missing from pfSense+ to make use of the more expansive set of BSD 14.0 capabilities is unclear to me. Indeed, it looks like pfSense+ went to the effort of including all the upstream BSD files needed to run the complete set of QAT capabilities.
It's why I wasn't surprised to see QAT apparently working in 23.05.1 and why I assumed an error prevented it working in 23.09d. Now I just don't have a clue as to what is or isn't intended for pfSense+.
️ -
@RobbieTT all of that is discussing the kernel. It says nothing about OpenSSL and without the OpenSSL qatengine there can be no use of QAT for SSL/TLS, SSH or any other application or protocol implemented in user space which relies on libcrypto for cryptographic algorithms.
Until FreeBSD ships the OpenSSL QAT engine I would not expect to see it in pfsense.
-
@jaltman It opens QAT beyond the kernel via the API - indeed, it directly references the API and user space capabilities. I don't know how they could say it more explicitly than in the quote:
A complete API for offloading these operations is exposed in the kernel and may be used by any other entity directly.
They also give examples of user space functions up to and including compression.
I don't doubt that there is something missing with OpenSSL in pfSense+ but I am not sure we can point the finger at freeBSD 14.0 in its non-pfSense guise.
️(If you have tested freeBSD 14.0 separately and found it to be lacking then please accept my apologies and disregard the above.)
https://github.com/intel/QAT_Engine/blob/master/docs/software_requirements.md
https://man.freebsd.org/cgi/man.cgi?query=qat&apropos=0&sektion=0&manpath=FreeBSD+14.0-STABLE&arch=default&format=html -
@RobbieTT What you are quoting from is the features of the driver. Simply because the driver is present does not mean that applications use it. Most of the applications that you care about nginx, apache, sshd, ssh, curl, etc are all linked against OpenSSL's libcrypto. The QAT support is simply unavailable to them unless OpenSSL is built with the options required to use the QAT engine and if the QAT engine is installed and loaded via the openssl.conf file in use by the application.
I've installed FreeBSD-14.0-BETA4-amd64. openssl is not built with QAT support and the qatengine is not packaged. The FreeBSD Ports Search has alternative builds of openssl but none of them include QAT support.
I think we can put this discussion to bed.
-
@jaltman So it just comes down to the version of OpenSSL being used is not built with QAT support?
I ask because openSSL v3.0.10 is specifically called for in the freeBSD QAT requirements and pfSense uses that very same version:
/root: openssl version OpenSSL 3.0.10 1 Aug 2023 (Library: OpenSSL 3.0.10 1 Aug 2023)️ -
@RobbieTT OpenSSL 3.0 is used by FreeBSD but the QAT Engine and its dependencies (ipp-crypto-mb, ipsec-mb, qatlib) are not part of the base OpenSSL 3.0 build.
For example, on Fedora Linux you need to install
intel-ipp-crypto-mb-1.0.8-3.fc37.x86_64 intel-ipsec-mb-1.4.0-1.fc37.x86_64 qatengine-1.4.0-1.fc37.x86_64 qatlib-23.02.0-1.fc37.x86_64 qatlib-service-23.02.0-1.fc37.x86_64only then can the OpenSSL QAT Engine be used
[jaltman@fc36]$ ls /usr/lib64/engines-3/ afalg.so capi.so libpkcs11.so loader_attic.so padlock.so pkcs11.so qatengine.so [jaltman@fc37]$ openssl engine -t -c -v qatengine QAT_SW - Processor unsupported: AVX512F = 0, VAES = 0, VPCLMULQDQ = 0 (qatengine) Reference implementation of QAT crypto engine(qat_hw & qat_sw) v1.4.0 [RSA, AES-128-CBC-HMAC-SHA256, AES-256-CBC-HMAC-SHA256, ChaCha20-Poly1305, id-aes128-GCM, id-aes192-GCM, id-aes256-GCM, SHA3-256, SHA3-384, SHA3-512, TLS1-PRF, X25519, X448, SM2] [ available ] ENABLE_EXTERNAL_POLLING, POLL, SET_INSTANCE_FOR_THREAD, GET_NUM_OP_RETRIES, SET_MAX_RETRY_COUNT, SET_INTERNAL_POLL_INTERVAL, GET_EXTERNAL_POLLING_FD, ENABLE_EVENT_DRIVEN_POLLING_MODE, GET_NUM_CRYPTO_INSTANCES, DISABLE_EVENT_DRIVEN_POLLING_MODE, SET_EPOLL_TIMEOUT, SET_CRYPTO_SMALL_PACKET_OFFLOAD_THRESHOLD, ENABLE_INLINE_POLLING, ENABLE_HEURISTIC_POLLING, GET_NUM_REQUESTS_IN_FLIGHT, INIT_ENGINE, SET_CONFIGURATION_SECTION_NAME, ENABLE_SW_FALLBACK, HEARTBEAT_POLL, DISABLE_QAT_OFFLOAD, HW_ALGO_BITMAP, SW_ALGO_BITMAPAs far as I can tell there is no qatengine.so packaged for OpenSSL 3.0, 3.1 or 3.2 on FreeBSD 14. Hence it cannot be installed and cannot be used.
-
Mmm, as I read it OpenSSL requires the qat engine module to use it in user mode. Interesting that it does use it in 23.05...
-
@stephenw10 following this thread for a while and that’s the general concern here. Why is this behavior different?
-
It's almost certainly because we moved to OpenSSL 3 and there is fallout from that. Most of that has been resolved. Since user mode encryption off-load is generally not supported this was probably just overlooked. I'll see what I can do when I'm home tomorrow.
-
@stephenw10 thank you. Appreciate the quick response
-
I'm still not convinced anyone has accurately demonstrated that it was working on 23.05.1. There isn't any evidence that it was, just what may be coincidental increased in interrupt usage.
And I think people missed the fact that there is support for userspace QAT in the 14 kernel driver but it's only for 4xxx devices. (See my post here: https://forum.netgate.com/post/1128163 )
And the 14 man page:
cfg_mode Override the device mode configuration for kernel space and user space instances. Possible values: "ks", "us", "ks;us". Default value "ks;us".If userspace QAT was working on 23.05.1, anyone could replicate the results being claimed, but so far nobody else has been able to.
Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!
Need help fast? Netgate Global Support!
Do not Chat/PM for help!
-
@stephenw10 said in 23.09d - Is QAT Broken?:
Mmm, as I read it OpenSSL requires the qat engine module to use it in user mode. Interesting that it does use it in 23.05...
Quite a few things have changed with 23.09d. The library of files used by OpenSSL is more expansive, the config files have changed and other new elements (eg Kea) have become users of OpenSSL.
Moving from the QAT-focused OpenSSL 1.1.1t-freebsd to the later OpenSSL 3.0.10 is also a significant delta.
There are other oddities between 23.05 and 23.09d. For example, the openssl engine on 23.05 used:
[23.05.1-RELEASE] /root: openssl engine (devcrypto) /dev/crypto engine (rdrand) Intel RDRAND engine (dynamic) Dynamic engine loading support [23.05.1-RELEASE] /root:With 23.09d the devcrypto line has been removed:
[23.09-DEVELOPMENT] /root: openssl engine (rdrand) Intel RDRAND engine (dynamic) Dynamic engine loading support [23.09-DEVELOPMENT]/root:There also appears to be no
/usr/lib/engines/qatengine.sofile or indeed a qatengine.so anywhere on the system.I have no difficulty replicating the QAT interrupts on 23.05.1. They don't increment by themselves, only when the firewall is doing a relevant task eg TLS/SSL. A simple DoT Dig that is forwarded is enough to increment, as will a curl, package update etc. Not sure I am believed though, for reasons that escape me.
-
@jimp said in 23.09d - Is QAT Broken?:
And I think people missed the fact that there is support for userspace QAT in the 14 kernel driver but it's only for 4xxx devices. (See my post here: https://forum.netgate.com/post/1128163 )
And the 14 man page:
Jim, the 4xxx message could be linked to an errata elsewhere in pfSense as it has been missed from one of the lists. It is included in the actual FW lists though. There was a post on this subject a few days ago which @stephenw10 covered. Of course, being a later QAT generation, it will have key differences to the earlier generations QAT in the C3xxx and probably adds a brace of expanded capabilities.
The man pages you linked to makes no mention of userspace being limited to 4xxx either and it is grouped in the same list as the C3xxx. That does not make it untrue either, just less than clear.
I agree though that 23.09d is limited to kernel space (ks) only but I don't think that is attributed to freeBSD 14.0 alone. That change may have been brought about by pfSense+ and its current configuration.
pfSense 23.05.1 is also on freeBSD 14 and it is flagged to run in the default kernel space + user space (ks;us) mode.
23.05.1:
[23.05.1-RELEASE]/root: sysctl -a | grep "cfg" hw.pci.mcfg: 1 dev.qat.0.dev_cfg: [GENERAL] [23.05.1-RELEASE]/root:23.09d - 'us' mode has been disabled, leaving only 'ks' mode enabled:
[23.09-DEVELOPMENT]/root: sysctl -a | grep "cfg" hw.pci.mcfg: 1 dev.qat.0.dev_cfg: [GENERAL] dev.qat.0.cfg_mode: ks dev.qat.0.cfg_services: sym;dc [23.09-DEVELOPMENT]/root:I really hope someone will check my findings as not being believed feels pretty odd.
️ -
@stephenw10 said in 23.09d - Is QAT Broken?:
Mmm, as I read it OpenSSL requires the qat engine module to use it in user mode. Interesting that it does use it in 23.05...
OpenSSL 1.1.x also requires the QAT Engine in order to support use of QuickAssist. The Intel QAT Engine for OpenSSL was developed against OpenSSL 1.1 on FreeBSD 12.4. However, that release doesn't package or ship the engine.
I have seen no evidence on my 4100 when running 23.05.1 that QAT is being used by userspace. There is a small increase in the qat counters in kernel but I cannot believe that they are result of any userspace cryptographic or compression or signing operations.
-
@RobbieTT said in 23.09d - Is QAT Broken?:
[23.05.1-RELEASE] /root: openssl engine
(devcrypto) /dev/crypto engineIt is possible that QAT on 23.05.1 is triggered for random number generation since /dev/crypto operates in kernel and has access to QAT. Any such usage would not be for encryption, compression or signing of actual network traffic.
-
@RobbieTT said in 23.09d - Is QAT Broken?:
I have no difficulty replicating the QAT interrupts on 23.05.1. They don't increment by themselves, only when the firewall is doing a relevant task eg TLS/SSL. A simple DoT Dig that is forwarded is enough to increment, as will a curl, package update etc. Not sure I am believed though, for reasons that escape me.
I believe that the interrupts occur because I see them as well. I do not believe that it has anything to do with use of QAT to encrypt, sign, or compress network traffic because I understand how QAT plugs into OpenSSL libcrypto and none of nginx, apache, curl, sshd, ssh, kerberos, etc that rely upon libcrypto for encryption, signing and compression primitives would contain any internal code to call into the QAT driver.
libz also has to be built custom in order to make use of QAT.
