Sapphire Rapids QAT Support?

stephenw10

Nope it should still show as available in the CPU. Probably the pfSense GUI code just needs to catch up with the driver.

Does it automatically load the correct firmware? You shouldn't need the qat_c2xxx module for example so I assume you manually loaded that during testing?

stephenw10

Add a feature request to track: https://redmine.pfsense.org/issues/15233

Lurick

@stephenw10 I believe I did, or at least I presume that would happen with 'kldload qat' when I was testing things out.

Right now I'm on Proxmox 8 and have the pfsense VM CPU set to host which I believe is probably the best way to get it to see everything from the host CPU but if that's wrong let me know there.

stephenw10

The QAT device is actually a separate PCI device rather than something in the CPU directly (like AES-NI). You would need to pass it through to the guest VM in order to use it there.

Lurick

@stephenw10 Cool, yah I was looking back and realized I did have to do that in Proxmox. I tried undoing some stuff earlier and needed to get back to the working state again I showed prior. I wasn't sure since it's built into the CPU and some other discussions had me questioning if it really needed PCIe passthrough since it's technically in the 5416S.
After passing it through with:

qm set [VM ID] -hostpci2 0000:f7:00,pcie=1

I still had to do kldload qat to get pfsense to load the drivers after rebooting pfsense with the passthrough.
Once I did do that though I saw what I posted earlier.

Lurick

Full logs after a reboot of pfsense before and after kldload:

[23.09.1-RELEASE][admin@firewall]/root: pciconf -l
hostb0@pci0:0:0:0:      class=0x060000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x29c0 subvendor=0x1af4 subdevice=0x1100
vgapci0@pci0:0:1:0:     class=0x030000 rev=0x02 hdr=0x00 vendor=0x1234 device=0x1111 subvendor=0x1af4 subdevice=0x1100
uhci0@pci0:0:26:0:      class=0x0c0300 rev=0x03 hdr=0x00 vendor=0x8086 device=0x2937 subvendor=0x1af4 subdevice=0x1100
uhci1@pci0:0:26:1:      class=0x0c0300 rev=0x03 hdr=0x00 vendor=0x8086 device=0x2938 subvendor=0x1af4 subdevice=0x1100
uhci2@pci0:0:26:2:      class=0x0c0300 rev=0x03 hdr=0x00 vendor=0x8086 device=0x2939 subvendor=0x1af4 subdevice=0x1100
ehci0@pci0:0:26:7:      class=0x0c0320 rev=0x03 hdr=0x00 vendor=0x8086 device=0x293c subvendor=0x1af4 subdevice=0x1100
hdac0@pci0:0:27:0:      class=0x040300 rev=0x03 hdr=0x00 vendor=0x8086 device=0x293e subvendor=0x1af4 subdevice=0x1100
pcib1@pci0:0:28:0:      class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b36 device=0x000c subvendor=0x1b36 subdevice=0x0000
pcib2@pci0:0:28:1:      class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b36 device=0x000c subvendor=0x1b36 subdevice=0x0000
pcib3@pci0:0:28:2:      class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b36 device=0x000c subvendor=0x1b36 subdevice=0x0000
pcib4@pci0:0:28:3:      class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b36 device=0x000c subvendor=0x1b36 subdevice=0x0000
uhci3@pci0:0:29:0:      class=0x0c0300 rev=0x03 hdr=0x00 vendor=0x8086 device=0x2934 subvendor=0x1af4 subdevice=0x1100
uhci4@pci0:0:29:1:      class=0x0c0300 rev=0x03 hdr=0x00 vendor=0x8086 device=0x2935 subvendor=0x1af4 subdevice=0x1100
uhci5@pci0:0:29:2:      class=0x0c0300 rev=0x03 hdr=0x00 vendor=0x8086 device=0x2936 subvendor=0x1af4 subdevice=0x1100
ehci1@pci0:0:29:7:      class=0x0c0320 rev=0x03 hdr=0x00 vendor=0x8086 device=0x293a subvendor=0x1af4 subdevice=0x1100
pcib5@pci0:0:30:0:      class=0x060401 rev=0x92 hdr=0x01 vendor=0x8086 device=0x244e subvendor=0x0000 subdevice=0x0000
isab0@pci0:0:31:0:      class=0x060100 rev=0x02 hdr=0x00 vendor=0x8086 device=0x2918 subvendor=0x1af4 subdevice=0x1100
ahci0@pci0:0:31:2:      class=0x010601 rev=0x02 hdr=0x00 vendor=0x8086 device=0x2922 subvendor=0x1af4 subdevice=0x1100
ichsmb0@pci0:0:31:3:    class=0x0c0500 rev=0x02 hdr=0x00 vendor=0x8086 device=0x2930 subvendor=0x1af4 subdevice=0x1100
ix0@pci0:1:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1563 subvendor=0x15d9 subdevice=0x1563
ix1@pci0:2:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1563 subvendor=0x15d9 subdevice=0x1563
none0@pci0:3:0:0:       class=0x0b4000 rev=0x40 hdr=0x00 vendor=0x8086 device=0x4942 subvendor=0x8086 subdevice=0x0000
pcib6@pci0:5:1:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b36 device=0x0001 subvendor=0x0000 subdevice=0x0000
pcib7@pci0:5:2:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b36 device=0x0001 subvendor=0x0000 subdevice=0x0000
pcib8@pci0:5:3:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b36 device=0x0001 subvendor=0x0000 subdevice=0x0000
pcib9@pci0:5:4:0:       class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b36 device=0x0001 subvendor=0x0000 subdevice=0x0000
virtio_pci0@pci0:6:3:0: class=0x00ff00 rev=0x00 hdr=0x00 vendor=0x1af4 device=0x1002 subvendor=0x1af4 subdevice=0x0005
sym0@pci0:6:5:0:        class=0x010000 rev=0x00 hdr=0x00 vendor=0x1000 device=0x0012 subvendor=0x0000 subdevice=0x1000
vmx0@pci0:6:18:0:       class=0x020000 rev=0x01 hdr=0x00 vendor=0x15ad device=0x07b0 subvendor=0x15ad subdevice=0x07b0
[23.09.1-RELEASE][admin@firewall]/root: vmstat -i 
interrupt                          total       rate
irq1: atkbd0                           2          0
irq19: ehci0 ehci1                    21          0
irq22: sym0                         8040         86
cpu0:timer                          9845        105
cpu1:timer                          2842         30
cpu2:timer                          4630         49
cpu3:timer                          3600         38
cpu4:timer                          4641         49
cpu5:timer                          5584         60
cpu6:timer                          4466         48
cpu7:timer                          5067         54
irq25: ix0:rxq0                     1114         12
irq26: ix0:rxq1                     2844         30
irq27: ix0:rxq2                     4139         44
irq28: ix0:rxq3                     4063         43
irq29: ix0:rxq4                     5197         55
irq30: ix0:rxq5                     1754         19
irq31: ix0:rxq6                     3056         33
irq32: ix0:rxq7                     1800         19
irq33: ix0:aq                          3          0
irq34: ix1:rxq0                      846          9
irq35: ix1:rxq1                     1904         20
irq36: ix1:rxq2                     3666         39
irq37: ix1:rxq3                     4171         44
irq38: ix1:rxq4                     5366         57
irq39: ix1:rxq5                     1594         17
irq40: ix1:rxq6                     2997         32
irq41: ix1:rxq7                     1647         18
irq42: ix1:aq                          1          0
Total                              94900       1012
[23.09.1-RELEASE][admin@firewall]/root: dmesg | grep qat
[23.09.1-RELEASE][admin@firewall]/root: kldload qat
[23.09.1-RELEASE][admin@firewall]/root: dmesg | grep qat
qat0: <Intel 4xxx QuickAssist> mem 0xfd200000-0xfd27ffff,0xfc800000-0xfcffffff,0xfd000000-0xfd1fffff at device 0.0 on pci3
qat0: qat_dev0 started 9 acceleration engines
qat0: FW version: 1.0.10
qat0: Excessive clock measure delay
qat_ocf0: <QAT engine>
[23.09.1-RELEASE][admin@firewall]/root: pciconf -l | grep qat
qat0@pci0:3:0:0:        class=0x0b4000 rev=0x40 hdr=0x00 vendor=0x8086 device=0x4942 subvendor=0x8086 subdevice=0x0000

stephenw10

Hmm, interesting that actually appears to be a QAT 401xx device:
https://github.com/pfsense/FreeBSD-src/blob/devel-main/sys/dev/qat/include/common/adf_accel_devices.h#L38

Lurick

@stephenw10 That is interesting, everything I saw referenced 4xxx on Intel's side so I never even thought about that.

Lurick

@Lurick
root@proxmox:~# lspci | grep QAT
f3:00.0 Co-processor: Intel Corporation 4xxx Series QAT (rev 40)
f7:00.0 Co-processor: Intel Corporation 4xxx Series QAT (rev 40)

stephenw10

Yup, looks like it just groups them together as a gen4 devices.