23.09d - Is QAT Broken?

jaltman

@stephenw10 said in 23.09d - Is QAT Broken?:

Mmm, as I read it OpenSSL requires the qat engine module to use it in user mode. Interesting that it does use it in 23.05...

OpenSSL 1.1.x also requires the QAT Engine in order to support use of QuickAssist. The Intel QAT Engine for OpenSSL was developed against OpenSSL 1.1 on FreeBSD 12.4. However, that release doesn't package or ship the engine.

I have seen no evidence on my 4100 when running 23.05.1 that QAT is being used by userspace. There is a small increase in the qat counters in kernel but I cannot believe that they are result of any userspace cryptographic or compression or signing operations.

jaltman

@RobbieTT said in 23.09d - Is QAT Broken?:

[23.05.1-RELEASE] /root: openssl engine
(devcrypto) /dev/crypto engine

It is possible that QAT on 23.05.1 is triggered for random number generation since /dev/crypto operates in kernel and has access to QAT. Any such usage would not be for encryption, compression or signing of actual network traffic.

jaltman

@RobbieTT said in 23.09d - Is QAT Broken?:

I have no difficulty replicating the QAT interrupts on 23.05.1. They don't increment by themselves, only when the firewall is doing a relevant task eg TLS/SSL. A simple DoT Dig that is forwarded is enough to increment, as will a curl, package update etc. Not sure I am believed though, for reasons that escape me.

I believe that the interrupts occur because I see them as well. I do not believe that it has anything to do with use of QAT to encrypt, sign, or compress network traffic because I understand how QAT plugs into OpenSSL libcrypto and none of nginx, apache, curl, sshd, ssh, kerberos, etc that rely upon libcrypto for encryption, signing and compression primitives would contain any internal code to call into the QAT driver.

libz also has to be built custom in order to make use of QAT.

jaltman

I've sent private mail to Bernard Spil, the maintainer of OpenSSL for FreeBSD, asking him if and how QAT is supported in the FreeBSD builds.
I will report on his response when I receive it.

RobbieTT

@jaltman That would be enormously helpful - thank you.

️

stephenw10

Mmm, if it was supported in user-space I would expect to be able to see it very easily when using OpenVPN without DCO mode. With DCO is uses the kernel-mode crypto framework.

jaltman

@RobbieTT Bernard confirms QAT functionality has never been packaged by him for FreeBSD. He suggests that someone else should build it and submit a ports request.

He wants whoever supports it to have hardware on which to test it.

jaltman

@jaltman Moin Rahman did the earlier work on QAT support for the kernel and OpenSSL engine as one of his former employers was interested. However, that company went in a different direction leveraging programmable NICs instead after Intel abandoned the dedicated QAT add-on boards during the FreeBSD 13 time frame.

jimp

@RobbieTT said in 23.09d - Is QAT Broken?:

@jimp said in 23.09d - Is QAT Broken?:
Jim, the 4xxx message could be linked to an errata elsewhere in pfSense as it has been missed from one of the lists. It is included in the actual FW lists though. There was a post on this subject a few days ago which @stephenw10 covered. Of course, being a later QAT generation, it will have key differences to the earlier generations QAT in the C3xxx and probably adds a brace of expanded capabilities.

The message saying userspace QAT only supported on 4xxx is from FreeBSD, not pfSense.

https://github.com/freebsd/freebsd-src/blob/3523f0677ef514fe72710033c73cc58517b9cda8/sys/dev/qat/qat_common/adf_cfg_device.c#L700

The man pages you linked to makes no mention of userspace being limited to 4xxx either and it is grouped in the same list as the C3xxx. That does not make it untrue either, just less than clear.

Try setting a loader.conf.local tunable for dev.qat.0.cfg_mode="ks;us" yourself and see.

You can't compare 23.05.1 directly because it did not have that tunable so I don't get where your assertion is coming from that it had both enabled there. Your shell output doesn't show that. There is nothing in the 23.05.1 output showing userspace.

RobbieTT

@jimp said in 23.09d - Is QAT Broken?:

You can't compare 23.05.1 directly because it did not have that tunable so I don't get where your assertion is coming from that it had both enabled there. Your shell output doesn't show that. There is nothing in the 23.05.1 output showing userspace.

Understood but the man pages lists ks;us as the default and the absence of an explicit command to demure from that usually equals that the default is set. Apologies if this is not the case.

️

jimp

@RobbieTT said in 23.09d - Is QAT Broken?:

@jimp said in 23.09d - Is QAT Broken?:

You can't compare 23.05.1 directly because it did not have that tunable so I don't get where your assertion is coming from that it had both enabled there. Your shell output doesn't show that. There is nothing in the 23.05.1 output showing userspace.

Understood but the man pages lists ks;us as the default and the absence of an explicit command to demure from that usually equals that the default is set. Apologies if this is not the case.

The man page is not complete/accurate there. It's only the default on 4xxx devices as well, for all others it defaults to kernel only.

https://github.com/freebsd/freebsd-src/blob/main/sys/dev/qat/qat_common/adf_cfg.c#L37

RobbieTT

@jimp said in 23.09d - Is QAT Broken?:

The man page is not complete/accurate there.

I didn't stand a chance. Back to the cup of tea.

️

marcosm

@RobbieTT said in 23.09d - Is QAT Broken?:

@jimp said in 23.09d - Is QAT Broken?:

The man page is not complete/accurate there.

I didn't stand a chance. Back to the cup of tea.

️

There are many nuances indeed! It's a good discussion to have, and the civil approach is appreciated :)

bcdouglas

@marcosm said in 23.09d - Is QAT Broken?:

@RobbieTT said in 23.09d - Is QAT Broken?:

@jimp said in 23.09d - Is QAT Broken?:

The man page is not complete/accurate there.

I didn't stand a chance. Back to the cup of tea.

️

There are many nuances indeed! It's a good discussion to have, and the civil approach is appreciated :)

Yes, thank you for a civil discussion @marcosm and @stephenw10. I don't know why these conversations often become aggressive with users trying to provide input to netgate. Thanks @marcosm and @stephenw10 .

RobbieTT

@bcdouglas
Sounds like I am being admonished; if so I will take it on the chin. It was not my intent to cause waves but clearly something unintended was triggered.

As users we don't always have the technical language for this kind of discourse but all I can say is that I did my best to read-into the topic to try and understand the apparent changes or limitations, only to find gaps in the documentation.

Threads such as this may put-off others from providing feedback but it shouldn't. Please set this aside and do comment when you think something does not make sense. Nothing can move forward without feedback.

Anyway, I'll take the thumping on this one.

stephenw10

I really don't think there was any issue here. Reading back I think there was a misunderstanding earlier on but in general this was a useful discussion. No admonishment required!

bcdouglas

@RobbieTT Not from me. The conversations from the Netgate side often take a weird turn when users try to ask honest questions and help.

RobbieTT

@stephenw10

No observable change in functionality with the newly-enabled QAT 200xx devices (tested on an Xeon D-1536NT with 23.09.b.20231020.0600 installed) from that of the C3xxx series on the same beta load.

️

stephenw10

The change we made there should now recognise that device as QAT capable on the dashboard. It should also load the qat module if it's not already.
However the driver itself already worked with the hardware so if it was loaded then the kernel could already use it for kernel mode crypto.

RobbieTT

@stephenw10
It does show correctly and I have provided feedback on issue #14844.

️