Sapphire Rapids QAT Support?
-
@stephenw10 Correct, I triple checked. Perhaps it's a GUI display issue or, and I'm not 100% sure on this, do I need to disable IPSec-MB?
root: vmstat -i interrupt total rate irq1: atkbd0 2 0 irq19: ehci0 ehci1 21 0 irq22: sym0 3516058 13 cpu0:timer 31169653 114 cpu1:timer 35268031 129 cpu2:timer 11008937 40 cpu3:timer 10926755 40 cpu4:timer 17340435 64 cpu5:timer 11901727 44 cpu6:timer 9208785 34 cpu7:timer 11716605 43 irq25: ix0:rxq0 7656208 28 irq26: ix0:rxq1 33398446 122 irq27: ix0:rxq2 8286263 30 irq28: ix0:rxq3 21840050 80 irq29: ix0:rxq4 12992202 48 irq30: ix0:rxq5 24379698 89 irq31: ix0:rxq6 12540202 46 irq32: ix0:rxq7 9611903 35 irq33: ix0:aq 3 0 irq34: ix1:rxq0 4290260 16 irq35: ix1:rxq1 32394876 119 irq36: ix1:rxq2 8025425 29 irq37: ix1:rxq3 21524101 79 irq38: ix1:rxq4 13054790 48 irq39: ix1:rxq5 23072021 85 irq40: ix1:rxq6 12175240 45 irq41: ix1:rxq7 8467197 31 irq42: ix1:aq 3 0 Total 395765897 1451 root: vmstat -i | grep qat root: dmesg | grep qat qat0: <Intel 4xxx QuickAssist> mem 0xfd200000-0xfd27ffff,0xfc800000-0xfcffffff,0xfd000000-0xfd1fffff at device 0.0 on pci3 qat0: qat_dev0 started 9 acceleration engines qat0: FW version: 1.0.10 qat0: Excessive clock measure delay qat_ocf0: <QAT engine> root: kldstat Id Refs Address Size Name 1 75 0xffffffff80200000 339f830 kernel 2 1 0xffffffff835a0000 1e2b0 opensolaris.ko 3 1 0xffffffff835c0000 5d7790 zfs.ko 4 1 0xffffffff84418000 2220 cpuctl.ko 5 1 0xffffffff8441b000 3240 ichsmb.ko 6 1 0xffffffff8441f000 2178 smbus.ko 7 1 0xffffffff84422000 4378 qat.ko 8 6 0xffffffff84427000 14d60 qat_hw.ko 9 9 0xffffffff8443c000 2ff70 qat_common.ko 10 8 0xffffffff8446c000 68cd8 qat_api.ko 11 1 0xffffffff844d5000 c9228 qat_4xxx_fw.ko 12 1 0xffffffff84600000 666a08 iimb.ko 13 1 0xffffffff8459f000 11240 qat_c2xxx.ko 15 1 0xffffffff845e0000 20e8 coretemp.ko 16 1 0xffffffff845e3000 13808 dummynet.ko 17 1 0xffffffff845b1000 2e560 if_wg.ko 18 1 0xffffffff84e00000 4af8b8 qat_c4xxx_fw.ko -
Nope it should still show as available in the CPU. Probably the pfSense GUI code just needs to catch up with the driver.
Does it automatically load the correct firmware? You shouldn't need the qat_c2xxx module for example so I assume you manually loaded that during testing?
-
Add a feature request to track: https://redmine.pfsense.org/issues/15233
-
@stephenw10 I believe I did, or at least I presume that would happen with 'kldload qat' when I was testing things out.
Right now I'm on Proxmox 8 and have the pfsense VM CPU set to host which I believe is probably the best way to get it to see everything from the host CPU but if that's wrong let me know there.
-
The QAT device is actually a separate PCI device rather than something in the CPU directly (like AES-NI). You would need to pass it through to the guest VM in order to use it there.
-
@stephenw10 Cool, yah I was looking back and realized I did have to do that in Proxmox. I tried undoing some stuff earlier and needed to get back to the working state again I showed prior. I wasn't sure since it's built into the CPU and some other discussions had me questioning if it really needed PCIe passthrough since it's technically in the 5416S.
After passing it through with:qm set [VM ID] -hostpci2 0000:f7:00,pcie=1I still had to do kldload qat to get pfsense to load the drivers after rebooting pfsense with the passthrough.
Once I did do that though I saw what I posted earlier. -
Full logs after a reboot of pfsense before and after kldload:
[23.09.1-RELEASE][admin@firewall]/root: pciconf -l hostb0@pci0:0:0:0: class=0x060000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x29c0 subvendor=0x1af4 subdevice=0x1100 vgapci0@pci0:0:1:0: class=0x030000 rev=0x02 hdr=0x00 vendor=0x1234 device=0x1111 subvendor=0x1af4 subdevice=0x1100 uhci0@pci0:0:26:0: class=0x0c0300 rev=0x03 hdr=0x00 vendor=0x8086 device=0x2937 subvendor=0x1af4 subdevice=0x1100 uhci1@pci0:0:26:1: class=0x0c0300 rev=0x03 hdr=0x00 vendor=0x8086 device=0x2938 subvendor=0x1af4 subdevice=0x1100 uhci2@pci0:0:26:2: class=0x0c0300 rev=0x03 hdr=0x00 vendor=0x8086 device=0x2939 subvendor=0x1af4 subdevice=0x1100 ehci0@pci0:0:26:7: class=0x0c0320 rev=0x03 hdr=0x00 vendor=0x8086 device=0x293c subvendor=0x1af4 subdevice=0x1100 hdac0@pci0:0:27:0: class=0x040300 rev=0x03 hdr=0x00 vendor=0x8086 device=0x293e subvendor=0x1af4 subdevice=0x1100 pcib1@pci0:0:28:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b36 device=0x000c subvendor=0x1b36 subdevice=0x0000 pcib2@pci0:0:28:1: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b36 device=0x000c subvendor=0x1b36 subdevice=0x0000 pcib3@pci0:0:28:2: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b36 device=0x000c subvendor=0x1b36 subdevice=0x0000 pcib4@pci0:0:28:3: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b36 device=0x000c subvendor=0x1b36 subdevice=0x0000 uhci3@pci0:0:29:0: class=0x0c0300 rev=0x03 hdr=0x00 vendor=0x8086 device=0x2934 subvendor=0x1af4 subdevice=0x1100 uhci4@pci0:0:29:1: class=0x0c0300 rev=0x03 hdr=0x00 vendor=0x8086 device=0x2935 subvendor=0x1af4 subdevice=0x1100 uhci5@pci0:0:29:2: class=0x0c0300 rev=0x03 hdr=0x00 vendor=0x8086 device=0x2936 subvendor=0x1af4 subdevice=0x1100 ehci1@pci0:0:29:7: class=0x0c0320 rev=0x03 hdr=0x00 vendor=0x8086 device=0x293a subvendor=0x1af4 subdevice=0x1100 pcib5@pci0:0:30:0: class=0x060401 rev=0x92 hdr=0x01 vendor=0x8086 device=0x244e subvendor=0x0000 subdevice=0x0000 isab0@pci0:0:31:0: class=0x060100 rev=0x02 hdr=0x00 vendor=0x8086 device=0x2918 subvendor=0x1af4 subdevice=0x1100 ahci0@pci0:0:31:2: class=0x010601 rev=0x02 hdr=0x00 vendor=0x8086 device=0x2922 subvendor=0x1af4 subdevice=0x1100 ichsmb0@pci0:0:31:3: class=0x0c0500 rev=0x02 hdr=0x00 vendor=0x8086 device=0x2930 subvendor=0x1af4 subdevice=0x1100 ix0@pci0:1:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1563 subvendor=0x15d9 subdevice=0x1563 ix1@pci0:2:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1563 subvendor=0x15d9 subdevice=0x1563 none0@pci0:3:0:0: class=0x0b4000 rev=0x40 hdr=0x00 vendor=0x8086 device=0x4942 subvendor=0x8086 subdevice=0x0000 pcib6@pci0:5:1:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b36 device=0x0001 subvendor=0x0000 subdevice=0x0000 pcib7@pci0:5:2:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b36 device=0x0001 subvendor=0x0000 subdevice=0x0000 pcib8@pci0:5:3:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b36 device=0x0001 subvendor=0x0000 subdevice=0x0000 pcib9@pci0:5:4:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1b36 device=0x0001 subvendor=0x0000 subdevice=0x0000 virtio_pci0@pci0:6:3:0: class=0x00ff00 rev=0x00 hdr=0x00 vendor=0x1af4 device=0x1002 subvendor=0x1af4 subdevice=0x0005 sym0@pci0:6:5:0: class=0x010000 rev=0x00 hdr=0x00 vendor=0x1000 device=0x0012 subvendor=0x0000 subdevice=0x1000 vmx0@pci0:6:18:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x15ad device=0x07b0 subvendor=0x15ad subdevice=0x07b0 [23.09.1-RELEASE][admin@firewall]/root: vmstat -i interrupt total rate irq1: atkbd0 2 0 irq19: ehci0 ehci1 21 0 irq22: sym0 8040 86 cpu0:timer 9845 105 cpu1:timer 2842 30 cpu2:timer 4630 49 cpu3:timer 3600 38 cpu4:timer 4641 49 cpu5:timer 5584 60 cpu6:timer 4466 48 cpu7:timer 5067 54 irq25: ix0:rxq0 1114 12 irq26: ix0:rxq1 2844 30 irq27: ix0:rxq2 4139 44 irq28: ix0:rxq3 4063 43 irq29: ix0:rxq4 5197 55 irq30: ix0:rxq5 1754 19 irq31: ix0:rxq6 3056 33 irq32: ix0:rxq7 1800 19 irq33: ix0:aq 3 0 irq34: ix1:rxq0 846 9 irq35: ix1:rxq1 1904 20 irq36: ix1:rxq2 3666 39 irq37: ix1:rxq3 4171 44 irq38: ix1:rxq4 5366 57 irq39: ix1:rxq5 1594 17 irq40: ix1:rxq6 2997 32 irq41: ix1:rxq7 1647 18 irq42: ix1:aq 1 0 Total 94900 1012 [23.09.1-RELEASE][admin@firewall]/root: dmesg | grep qat [23.09.1-RELEASE][admin@firewall]/root: kldload qat [23.09.1-RELEASE][admin@firewall]/root: dmesg | grep qat qat0: <Intel 4xxx QuickAssist> mem 0xfd200000-0xfd27ffff,0xfc800000-0xfcffffff,0xfd000000-0xfd1fffff at device 0.0 on pci3 qat0: qat_dev0 started 9 acceleration engines qat0: FW version: 1.0.10 qat0: Excessive clock measure delay qat_ocf0: <QAT engine> [23.09.1-RELEASE][admin@firewall]/root: pciconf -l | grep qat qat0@pci0:3:0:0: class=0x0b4000 rev=0x40 hdr=0x00 vendor=0x8086 device=0x4942 subvendor=0x8086 subdevice=0x0000 -
Hmm, interesting that actually appears to be a QAT 401xx device:
https://github.com/pfsense/FreeBSD-src/blob/devel-main/sys/dev/qat/include/common/adf_accel_devices.h#L38 -
@stephenw10 That is interesting, everything I saw referenced 4xxx on Intel's side so I never even thought about that.
-
@Lurick
root@proxmox:~# lspci | grep QAT
f3:00.0 Co-processor: Intel Corporation 4xxx Series QAT (rev 40)
f7:00.0 Co-processor: Intel Corporation 4xxx Series QAT (rev 40) -
Yup, looks like it just groups them together as a gen4 devices.
