Intel Xeon D-1736NT QAT on pfSense Plus 23.05.1

eracerxrs

So I'm starting to familiarize myself with all that pfSense Plus offers, one feature which is of interest to me is QAT.

Currently I have proxmox passing through the 200xx Series Embedded QAT Coprocessor.

on pfSense CLI: pciconf -lv shows:

qat0@pci0:2:0:0:        class=0x0b4000 rev=0x11 hdr=0x00 vendor=0x8086 device=0x18ee subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = '200xx Series QAT'
    class      = processor

In the GUI under System -> Advanced -> Misc.
I selected Cryptographic Hardware =
Intel QuickAssist (QAT)

kldstat shows the driver being loaded:

Id Refs Address                Size Name
 1   58 0xffffffff80200000  339fc28 kernel
 2    1 0xffffffff835a0000     76f8 cryptodev.ko
 3    1 0xffffffff835a8000   5ba0d8 zfs.ko
 4    1 0xffffffff83b63000     aab0 opensolaris.ko
 5    1 0xffffffff83b6f000    abd98 ice_ddp.ko
 6    1 0xffffffff84118000     2220 cpuctl.ko
 7    1 0xffffffff8411b000     3248 ichsmb.ko
 8    1 0xffffffff8411f000     2178 smbus.ko
 9    1 0xffffffff84122000     4378 qat.ko
10    6 0xffffffff84127000    13568 qat_hw.ko
11    8 0xffffffff8413b000    2d840 qat_common.ko
12    7 0xffffffff84169000    68cd8 qat_api.ko
13    1 0xffffffff841d2000   122c18 qat_200xx_fw.ko
14    1 0xffffffff842f5000    11240 qat_c2xxx.ko

But the dashboard still shows:

Intel(R) Xeon(R) D-1736NT CPU @ 2.70GHz
8 CPUs : 1 package(s) x 8 core(s)
AES-NI CPU Crypto: Yes (inactive)
IPsec-MB Crypto: Yes (inactive)
QAT Crypto: No
Hardware crypto	Inactive

What am I missing here?

stephenw10

Do you see the driver attach without errors in the boot log?

Steve

eracerxrs

@stephenw10, thanks for the reply.

Oh yeah forgot about that...

qat0: <Intel 200xx QuickAssist> mem 0xfe600000-0xfe63ffff,0xfe640000-0xfe67ffff irq 16 at device 0.0 on pci2
qat0: qat_dev0 started 6 acceleration engines
qat0: FW version: 4.18.0
qat0: Excessive clock measure delay
qat_ocf0: <QAT engine>

What is "Excessive clock measure delay"?

stephenw10

Hmm, not sure what that message means but since it's not actually an error I wouldn't expect it to change what the GUI shows. And other devices working correctly show that: https://forum.netgate.com/post/1127856

Do you see it in the output of vmstat -i?

eracerxrs

@stephenw10

Yeah, I found that post after posting here and have been using to to try to figure out my problem.

Almost all of my output is the same as that poster's except for my QAT is 200xx and their's is c3xxx. But their problem is on a dev version but works on production.

QAT is not showing in vmstat -i

stephenw10

Ok, it looks like this isn't showing because that QAT device is not in the list of recognised devices:
https://github.com/pfsense/pfsense/blob/master/src/usr/local/www/includes/functions.inc.php#L237

It probably needs to be updated against the supported devices: https://github.com/pfsense/FreeBSD-src/tree/devel-main/sys/dev/qat/qat_hw

RobbieTT

@eracerxrs said in Intel Xeon D-1736NT QAT on pfSense Plus 23.05.1:

@stephenw10

Yeah, I found that post after posting here and have been using to to try to figure out my problem.

Almost all of my output is the same as that poster's except for my QAT is 200xx and their's is c3xxx. But their problem is on a dev version but works on production.

Guess who has a D-1736NT on order. I'm just born lucky with all things QAT.

pfSense does include some code for the 200xx QAT, including the firmware package:

/boot/kernel/qat_200xx_fw.ko

️

stephenw10

Should be an easy enough patch. https://redmine.pfsense.org/issues/14844

RobbieTT

@stephenw10

Thanks Steve

Any chance of pfSense taking on the full-range of QAT capabilities or did I hear that door firmly closed?

️

eracerxrs

@stephenw10 Good catch going straight to the source(code), thanks for your help!

@RobbieTT Yeah, that is unfortunate luck, but on the bright-side at least you have a c3xxx up and running on the stable release, if not the dev. version yet. And as stephenw10 points out it should be relatively easy to fix given the driver infrastructure is already included.

stephenw10

@RobbieTT said in Intel Xeon D-1736NT QAT on pfSense Plus 23.05.1:

Any chance of pfSense taking on the full-range of QAT capabilities

Which capabilities are you referring to?

RobbieTT

@stephenw10 said in Intel Xeon D-1736NT QAT on pfSense Plus 23.05.1:

Which capabilities are you referring to?

SSH, nginx, curl, TLS/SSL, openSSL etc - basically all the features listed in the Intel guide for Linux/FreeBSD, either as part of a pre-built Intel framework or via the API. Currently we seem to only have the kernel space active.

️

stephenw10

Ah, you mean user mode stuff. That would have been via cryptodev as an engine for OpenSSL as I understand it. I'm not sure if/when that might be coming back.

RobbieTT

@stephenw10 Basically just looking for pfSense to use QAT for anything it is running itself. I did thought it did just that on 23.05 but apparently that could be false increments on the QAT IRQs. They previously increased with things like TLS originating from the router itself.

It would seem a shame that a QAT equipped device would not QAT for its own activity. Performance on the table and all that.

️

stephenw10

Ah that might be something else then, cryptodev stopped working some time ago.

RobbieTT

@stephenw10

You may have caught this thread:

https://forum.netgate.com/topic/183123/23-09d-is-qat-broken/42?_=1696239799286

️