23.09d - Is QAT Broken?

RobbieTT

@jaltman It opens QAT beyond the kernel via the API - indeed, it directly references the API and user space capabilities. I don't know how they could say it more explicitly than in the quote:

A complete API for offloading these operations is exposed in the kernel and may be used by any other entity directly.

They also give examples of user space functions up to and including compression.

I don't doubt that there is something missing with OpenSSL in pfSense+ but I am not sure we can point the finger at freeBSD 14.0 in its non-pfSense guise.

️

(If you have tested freeBSD 14.0 separately and found it to be lacking then please accept my apologies and disregard the above.)

https://github.com/intel/QAT_Engine/blob/master/docs/software_requirements.md
https://man.freebsd.org/cgi/man.cgi?query=qat&apropos=0&sektion=0&manpath=FreeBSD+14.0-STABLE&arch=default&format=html

jaltman

@RobbieTT What you are quoting from is the features of the driver. Simply because the driver is present does not mean that applications use it. Most of the applications that you care about nginx, apache, sshd, ssh, curl, etc are all linked against OpenSSL's libcrypto. The QAT support is simply unavailable to them unless OpenSSL is built with the options required to use the QAT engine and if the QAT engine is installed and loaded via the openssl.conf file in use by the application.

I've installed FreeBSD-14.0-BETA4-amd64. openssl is not built with QAT support and the qatengine is not packaged. The FreeBSD Ports Search has alternative builds of openssl but none of them include QAT support.

I think we can put this discussion to bed.

RobbieTT

@jaltman So it just comes down to the version of OpenSSL being used is not built with QAT support?

I ask because openSSL v3.0.10 is specifically called for in the freeBSD QAT requirements and pfSense uses that very same version:

/root: openssl version
OpenSSL 3.0.10 1 Aug 2023 (Library: OpenSSL 3.0.10 1 Aug 2023)

️

jaltman

@RobbieTT OpenSSL 3.0 is used by FreeBSD but the QAT Engine and its dependencies (ipp-crypto-mb, ipsec-mb, qatlib) are not part of the base OpenSSL 3.0 build.

For example, on Fedora Linux you need to install

  intel-ipp-crypto-mb-1.0.8-3.fc37.x86_64        intel-ipsec-mb-1.4.0-1.fc37.x86_64          qatengine-1.4.0-1.fc37.x86_64       
  qatlib-23.02.0-1.fc37.x86_64                   qatlib-service-23.02.0-1.fc37.x86_64

only then can the OpenSSL QAT Engine be used

[jaltman@fc36]$ ls /usr/lib64/engines-3/
afalg.so  capi.so  libpkcs11.so  loader_attic.so  padlock.so  pkcs11.so  qatengine.so
[jaltman@fc37]$ openssl engine -t -c -v qatengine
QAT_SW - Processor unsupported: AVX512F = 0, VAES = 0, VPCLMULQDQ = 0
(qatengine) Reference implementation of QAT crypto engine(qat_hw & qat_sw) v1.4.0
 [RSA, AES-128-CBC-HMAC-SHA256, AES-256-CBC-HMAC-SHA256, ChaCha20-Poly1305, id-aes128-GCM, id-aes192-GCM, id-aes256-GCM, SHA3-256, SHA3-384, SHA3-512, TLS1-PRF, X25519, X448, SM2]
     [ available ]
     ENABLE_EXTERNAL_POLLING, POLL, SET_INSTANCE_FOR_THREAD, 
     GET_NUM_OP_RETRIES, SET_MAX_RETRY_COUNT, SET_INTERNAL_POLL_INTERVAL, 
     GET_EXTERNAL_POLLING_FD, ENABLE_EVENT_DRIVEN_POLLING_MODE, 
     GET_NUM_CRYPTO_INSTANCES, DISABLE_EVENT_DRIVEN_POLLING_MODE, 
     SET_EPOLL_TIMEOUT, SET_CRYPTO_SMALL_PACKET_OFFLOAD_THRESHOLD, 
     ENABLE_INLINE_POLLING, ENABLE_HEURISTIC_POLLING, 
     GET_NUM_REQUESTS_IN_FLIGHT, INIT_ENGINE, SET_CONFIGURATION_SECTION_NAME, 
     ENABLE_SW_FALLBACK, HEARTBEAT_POLL, DISABLE_QAT_OFFLOAD, HW_ALGO_BITMAP, 
     SW_ALGO_BITMAP

As far as I can tell there is no qatengine.so packaged for OpenSSL 3.0, 3.1 or 3.2 on FreeBSD 14. Hence it cannot be installed and cannot be used.

stephenw10

Mmm, as I read it OpenSSL requires the qat engine module to use it in user mode. Interesting that it does use it in 23.05...

michmoor

@stephenw10 following this thread for a while and that’s the general concern here. Why is this behavior different?

stephenw10

It's almost certainly because we moved to OpenSSL 3 and there is fallout from that. Most of that has been resolved. Since user mode encryption off-load is generally not supported this was probably just overlooked. I'll see what I can do when I'm home tomorrow.

michmoor

@stephenw10 thank you. Appreciate the quick response

jimp

I'm still not convinced anyone has accurately demonstrated that it was working on 23.05.1. There isn't any evidence that it was, just what may be coincidental increased in interrupt usage.

And I think people missed the fact that there is support for userspace QAT in the 14 kernel driver but it's only for 4xxx devices. (See my post here: https://forum.netgate.com/post/1128163 )

And the 14 man page:

https://man.freebsd.org/cgi/man.cgi?query=qat&apropos=0&sektion=0&manpath=FreeBSD+14.0-STABLE&arch=default&format=html

   cfg_mode
     Override	the device mode	configuration  for  kernel  space  and
     user  space  instances.	 Possible values: "ks",	"us", "ks;us".
     Default value "ks;us".

If userspace QAT was working on 23.05.1, anyone could replicate the results being claimed, but so far nobody else has been able to.

RobbieTT

@stephenw10 said in 23.09d - Is QAT Broken?:

Mmm, as I read it OpenSSL requires the qat engine module to use it in user mode. Interesting that it does use it in 23.05...

Quite a few things have changed with 23.09d. The library of files used by OpenSSL is more expansive, the config files have changed and other new elements (eg Kea) have become users of OpenSSL.

Moving from the QAT-focused OpenSSL 1.1.1t-freebsd to the later OpenSSL 3.0.10 is also a significant delta.

There are other oddities between 23.05 and 23.09d. For example, the openssl engine on 23.05 used:

[23.05.1-RELEASE] /root: openssl engine 
(devcrypto) /dev/crypto engine
(rdrand) Intel RDRAND engine
(dynamic) Dynamic engine loading support
[23.05.1-RELEASE] /root:

With 23.09d the devcrypto line has been removed:

[23.09-DEVELOPMENT] /root: openssl engine
(rdrand) Intel RDRAND engine
(dynamic) Dynamic engine loading support
[23.09-DEVELOPMENT]/root:

There also appears to be no /usr/lib/engines/qatengine.so file or indeed a qatengine.so anywhere on the system.

I have no difficulty replicating the QAT interrupts on 23.05.1. They don't increment by themselves, only when the firewall is doing a relevant task eg TLS/SSL. A simple DoT Dig that is forwarded is enough to increment, as will a curl, package update etc. Not sure I am believed though, for reasons that escape me.

RobbieTT

@jimp said in 23.09d - Is QAT Broken?:

And I think people missed the fact that there is support for userspace QAT in the 14 kernel driver but it's only for 4xxx devices. (See my post here: https://forum.netgate.com/post/1128163 )

And the 14 man page:

https://man.freebsd.org/cgi/man.cgi?query=qat&apropos=0&sektion=0&manpath=FreeBSD+14.0-STABLE&arch=default&format=html

Jim, the 4xxx message could be linked to an errata elsewhere in pfSense as it has been missed from one of the lists. It is included in the actual FW lists though. There was a post on this subject a few days ago which @stephenw10 covered. Of course, being a later QAT generation, it will have key differences to the earlier generations QAT in the C3xxx and probably adds a brace of expanded capabilities.

The man pages you linked to makes no mention of userspace being limited to 4xxx either and it is grouped in the same list as the C3xxx. That does not make it untrue either, just less than clear.

I agree though that 23.09d is limited to kernel space (ks) only but I don't think that is attributed to freeBSD 14.0 alone. That change may have been brought about by pfSense+ and its current configuration.

pfSense 23.05.1 is also on freeBSD 14 and it is flagged to run in the default kernel space + user space (ks;us) mode.

23.05.1:

[23.05.1-RELEASE]/root: sysctl -a | grep "cfg"
hw.pci.mcfg: 1
dev.qat.0.dev_cfg: [GENERAL]
[23.05.1-RELEASE]/root:

23.09d - 'us' mode has been disabled, leaving only 'ks' mode enabled:

[23.09-DEVELOPMENT]/root: sysctl -a | grep "cfg"
hw.pci.mcfg: 1
dev.qat.0.dev_cfg: [GENERAL]
dev.qat.0.cfg_mode: ks
dev.qat.0.cfg_services: sym;dc
[23.09-DEVELOPMENT]/root:

I really hope someone will check my findings as not being believed feels pretty odd.

️

jaltman

@stephenw10 said in 23.09d - Is QAT Broken?:

Mmm, as I read it OpenSSL requires the qat engine module to use it in user mode. Interesting that it does use it in 23.05...

OpenSSL 1.1.x also requires the QAT Engine in order to support use of QuickAssist. The Intel QAT Engine for OpenSSL was developed against OpenSSL 1.1 on FreeBSD 12.4. However, that release doesn't package or ship the engine.

I have seen no evidence on my 4100 when running 23.05.1 that QAT is being used by userspace. There is a small increase in the qat counters in kernel but I cannot believe that they are result of any userspace cryptographic or compression or signing operations.

jaltman

@RobbieTT said in 23.09d - Is QAT Broken?:

[23.05.1-RELEASE] /root: openssl engine
(devcrypto) /dev/crypto engine

It is possible that QAT on 23.05.1 is triggered for random number generation since /dev/crypto operates in kernel and has access to QAT. Any such usage would not be for encryption, compression or signing of actual network traffic.

jaltman

@RobbieTT said in 23.09d - Is QAT Broken?:

I have no difficulty replicating the QAT interrupts on 23.05.1. They don't increment by themselves, only when the firewall is doing a relevant task eg TLS/SSL. A simple DoT Dig that is forwarded is enough to increment, as will a curl, package update etc. Not sure I am believed though, for reasons that escape me.

I believe that the interrupts occur because I see them as well. I do not believe that it has anything to do with use of QAT to encrypt, sign, or compress network traffic because I understand how QAT plugs into OpenSSL libcrypto and none of nginx, apache, curl, sshd, ssh, kerberos, etc that rely upon libcrypto for encryption, signing and compression primitives would contain any internal code to call into the QAT driver.

libz also has to be built custom in order to make use of QAT.

jaltman

I've sent private mail to Bernard Spil, the maintainer of OpenSSL for FreeBSD, asking him if and how QAT is supported in the FreeBSD builds.
I will report on his response when I receive it.

RobbieTT

@jaltman That would be enormously helpful - thank you.

️

stephenw10

Mmm, if it was supported in user-space I would expect to be able to see it very easily when using OpenVPN without DCO mode. With DCO is uses the kernel-mode crypto framework.

jaltman

@RobbieTT Bernard confirms QAT functionality has never been packaged by him for FreeBSD. He suggests that someone else should build it and submit a ports request.

He wants whoever supports it to have hardware on which to test it.

jaltman

@jaltman Moin Rahman did the earlier work on QAT support for the kernel and OpenSSL engine as one of his former employers was interested. However, that company went in a different direction leveraging programmable NICs instead after Intel abandoned the dedicated QAT add-on boards during the FreeBSD 13 time frame.

jimp

@RobbieTT said in 23.09d - Is QAT Broken?:

@jimp said in 23.09d - Is QAT Broken?:
Jim, the 4xxx message could be linked to an errata elsewhere in pfSense as it has been missed from one of the lists. It is included in the actual FW lists though. There was a post on this subject a few days ago which @stephenw10 covered. Of course, being a later QAT generation, it will have key differences to the earlier generations QAT in the C3xxx and probably adds a brace of expanded capabilities.

The message saying userspace QAT only supported on 4xxx is from FreeBSD, not pfSense.

https://github.com/freebsd/freebsd-src/blob/3523f0677ef514fe72710033c73cc58517b9cda8/sys/dev/qat/qat_common/adf_cfg_device.c#L700

The man pages you linked to makes no mention of userspace being limited to 4xxx either and it is grouped in the same list as the C3xxx. That does not make it untrue either, just less than clear.

Try setting a loader.conf.local tunable for dev.qat.0.cfg_mode="ks;us" yourself and see.

You can't compare 23.05.1 directly because it did not have that tunable so I don't get where your assertion is coming from that it had both enabled there. Your shell output doesn't show that. There is nothing in the 23.05.1 output showing userspace.