Intel Xeon D-2796NT and QAT
-
@stephenw10 said in Intel Xeon D-2796NT and QAT:
I'm not sure what QAT device is on those so I can't tell you. It seems quite likely though since they're from the same family of CPUs.
Intel C62x
none1@pci0:4:8:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000
vendor = 'Intel Corporation'
device = 'C62x Chipset QuickAssist Technology Virtual Function'
class = processorSo should I install the given - 1761331340268-180.diff ?
-
That device is already supported in 25.07.1. Check the IDs in the diff.
-
@stephenw10 said in Intel Xeon D-2796NT and QAT:
That device is already supported in 25.07.1. Check the IDs in the diff.
I still can't get it to work, I'll show you :)-------
(now I was able to focus on this more deeply, - this pfs box was completed.)-these are PCI QAT VFs /w pass-thr. from ESXi - no qat0
none1@pci0:4:8:0: to none10@pci0:4:17:0:
[25.07.1-RELEASE][root@ngfw.rm.arpa]/root: vmstat -i | grep qat [25.07.1-RELEASE][root@ngfw.rm.arpa]/root: sysctl -a | grep 'qat' qat_ocf0: <QAT engine> dev.qat_ocf.0.enable: 1 dev.qat_ocf.0.%iommu: dev.qat_ocf.0.%parent: nexus0 dev.qat_ocf.0.%pnpinfo: dev.qat_ocf.0.%location: dev.qat_ocf.0.%driver: qat_ocf dev.qat_ocf.0.%desc: QAT engine dev.qat_ocf.%parent: [25.07.1-RELEASE][root@ngfw.rm.arpa]/root: sysctl dev.qat_ocf.0.enable dev.qat_ocf.0.enable: 1 [25.07.1-RELEASE][root@ngfw.rm.arpa]/root: kldstat | egrep 'qat|ocf|crypto' 9 1 0xffffffff8412b000 4b18 cryptodev.ko 14 1 0xffffffff84142000 4390 qat.ko 15 6 0xffffffff84147000 14dd0 qat_hw.ko 16 9 0xffffffff8415c000 2ffe0 qat_common.ko 17 8 0xffffffff8418c000 68cd8 qat_api.ko 18 1 0xffffffff847d7000 11258 qat_c2xxx.ko [25.07.1-RELEASE][root@ngfw.rm.arpa]/root: kldstat -v | grep qat 14 1 0xffffffff84142000 4390 qat.ko (/boot/kernel/qat.ko) 698 nexus/qat 15 6 0xffffffff84147000 14dd0 qat_hw.ko (/boot/kernel/qat_hw.ko) 697 pci/qat_c4xxx 692 pci/qat_200xx 696 pci/qat_dh895xcc 693 pci/qat_4xxx 695 pci/qat_c3xxx 691 pci/qat_c62x 694 pci/qat_4xxxvf 16 9 0xffffffff8415c000 2ffe0 qat_common.ko (/boot/kernel/qat_common.ko) 689 qat_common 17 8 0xffffffff8418c000 68cd8 qat_api.ko (/boot/kernel/qat_api.ko) 690 qat_api 18 1 0xffffffff847d7000 11258 qat_c2xxx.ko (/boot/kernel/qat_c2xxx.ko) 699 pci/qat_c2xxx [25.07.1-RELEASE][root@ngfw.rm.arpa]/root: dmesg | grep -i qat qat_ocf0: <QAT engine> [25.07.1-RELEASE][root@ngfw.rm.arpa]/root: pciconf -lv none1@pci0:4:8:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none2@pci0:4:9:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none3@pci0:4:10:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none4@pci0:4:11:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none5@pci0:4:12:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none6@pci0:4:13:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none7@pci0:4:14:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none8@pci0:4:15:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none9@pci0:4:16:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none10@pci0:4:17:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor [root@vepesxi:~] vsish -e get /hardware/ipmiSupport 1 -> IPMI supported [root@vepesxi:~] lspci -vn | grep -E '37c9|18ef|18a1.*_VF_' 0000:b5:01.0 Class 0b40: 8086:37c9 0000:b5:01.1 Class 0b40: 8086:37c9 0000:b5:01.2 Class 0b40: 8086:37c9 0000:b5:01.3 Class 0b40: 8086:37c9 0000:b5:01.4 Class 0b40: 8086:37c9 0000:b5:01.5 Class 0b40: 8086:37c9 0000:b5:01.6 Class 0b40: 8086:37c9 0000:b5:01.7 Class 0b40: 8086:37c9 0000:b5:02.0 Class 0b40: 8086:37c9 0000:b5:02.1 Class 0b40: 8086:37c9 0000:b5:02.2 Class 0b40: 8086:37c9 0000:b5:02.3 Class 0b40: 8086:37c9 0000:b5:02.4 Class 0b40: 8086:37c9 0000:b5:02.5 Class 0b40: 8086:37c9 0000:b5:02.6 Class 0b40: 8086:37c9 0000:b5:02.7 Class 0b40: 8086:37c9 [root@vepesxi:~] lspci -vvv 0000:b5:00.0 Processor Co-processor: Intel Corporation c6xx QAT Class 0b40: 8086:37c8 0000:b5:01.0 Processor Co-processor: Intel Corporation c6xx QAT VF Class 0b40: 8086:37c9 0000:b5:01.1 Processor Co-processor: Intel Corporation c6xx QAT VF Class 0b40: 8086:37c9 0000:b5:01.2 Processor Co-processor: Intel Corporation c6xx QAT VF Class 0b40: 8086:37c9 0000:b5:01.3 Processor Co-processor: Intel Corporation c6xx QAT VF Class 0b40: 8086:37c9 0000:b5:01.4 Processor Co-processor: Intel Corporation c6xx QAT VF Class 0b40: 8086:37c9 0000:b5:01.5 Processor Co-processor: Intel Corporation c6xx QAT VF Class 0b40: 8086:37c9 0000:b5:01.6 Processor Co-processor: Intel Corporation c6xx QAT VF Class 0b40: 8086:37c9 0000:b5:01.7 Processor Co-processor: Intel Corporation c6xx QAT VF Class 0b40: 8086:37c9 0000:b5:02.0 Processor Co-processor: Intel Corporation c6xx QAT VF Class 0b40: 8086:37c9 0000:b5:02.1 Processor Co-processor: Intel Corporation c6xx QAT VF Class 0b40: 8086:37c9 0000:b5:02.2 Processor Co-processor: Intel Corporation c6xx QAT VF Class 0b40: 8086:37c9 0000:b5:02.3 Processor Co-processor: Intel Corporation c6xx QAT VF Class 0b40: 8086:37c9 0000:b5:02.4 Processor Co-processor: Intel Corporation c6xx QAT VF Class 0b40: 8086:37c9 0000:b5:02.5 Processor Co-processor: Intel Corporation c6xx QAT VF Class 0b40: 8086:37c9 0000:b5:02.6 Processor Co-processor: Intel Corporation c6xx QAT VF Class 0b40: 8086:37c9 0000:b5:02.7 Processor Co-processor: Intel Corporation c6xx QAT VF Class 0b40: 8086:37c9Cannot be (active), but is detected as a QAT device.
@stephenw10 - Any ideas?
As far as I know, Netgate pushed for this development in FreeBSD.
I can't really find anything about this on FreeBSD channels.PS:
Broadcom:
https://knowledge.broadcom.com/external/article/315470/intel-quickassist-technology-driver-for.html -
Hmm, no logs showing the driver trying to attach?
It could be the driver itself doesn't support the virtual function for some reason. That would be unusual though.
-
Nope it is there: https://github.com/pfsense/FreeBSD-src/blob/devel-main/sys/dev/qat/include/common/adf_accel_devices.h#L29
And it's in 25.07.1. So I'd expect to see it attach or attempt to attach.
-
@stephenw10 said in Intel Xeon D-2796NT and QAT:
Hmm, no logs showing the driver trying to attach?
I think I got it - I can see it in - dmesg.boot
pci4: <processor> at device 8.0 (no driver attached) pci4: <processor> at device 9.0 (no driver attached) pci4: <processor> at device 10.0 (no driver attached) pci4: <processor> at device 11.0 (no driver attached) pci4: <processor> at device 12.0 (no driver attached) pci4: <processor> at device 13.0 (no driver attached) pci4: <processor> at device 14.0 (no driver attached) pci4: <processor> at device 15.0 (no driver attached) pci4: <processor> at device 16.0 (no driver attached) pci4: <processor> at device 17.0 (no driver attached)& pciconf -lv --- pci4 --- 8 to 17 no driver attached
none1@pci0:4:8:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none2@pci0:4:9:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none3@pci0:4:10:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none4@pci0:4:11:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none5@pci0:4:12:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none6@pci0:4:13:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none7@pci0:4:14:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none8@pci0:4:15:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none9@pci0:4:16:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none10@pci0:4:17:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processorDash claims to see QAT capabilities, but no driver is attached. Why?
-
@DaddyGo said in Intel Xeon D-2796NT and QAT:
Dash claims to see QAT capabilities, but no driver is attached. Why?
Because the webgui code looks for the device IDs and the C62x is in that list. For some reason the driver isn't attaching to the devices on your hardware, or even trying to attach, so whilst pfSense thinks you have a QAT device there isn't actually one present.
I've never tried passing through a VF QAT device like that and I suspect that could be causing it. I also don't have any hardware to test it with though.
Can you pass-through the real PCIe device?
Can you test bare meta?
-
@stephenw10 said in Intel Xeon D-2796NT and QAT:
Can you pass-through the real PCIe device?
Can you test bare meta?
A little background on the case, this is exactly what this device is:
https://i.dell.com/sites/csdocuments/Shared-Content_data-Sheets_Documents/en/virtual-edge-platform-4600-spec-sheet.pdf
We have 15 of these in our system, with VersaOS, but the license is extremely expensive and we want to switch to pfSense.
What I am currently testing with pfS Plus is equipped with an 8C16T Intel D-2145NT CPU.
QAT on these not real PCI devices; they are C62x implemented in the CPU as chipset device(s).Since Dell also sells these with ESXi as base OS opc., we still have a license valid for four years, so I used that.
--and the other thing I would like to test before the mature version of "if_pppoe" is this - multi-threaded PPPoE in pfSense
https://www.neelc.org/posts/opnsense-pppoe-kvm/Also have a 16C32T D-2187NT version "behind me", which I have already started with a light mfsBSD. So I think pfSense can also run on it as bare metal. I'll try it out soon.
This is how it looks in ESXi before you enable SR-IOV - only PCI bridges and the base Chipset device
EDIT1: This is a screenshot of the D-2187NT CPU where 3 "ch" QAT is implemented.
After enabling SR-IOV, it creates 16 VFs for each Chipset device (C62x), and only these can be passed to the guest using "pass-thr".
What you see in the VM PCI list...
pfSense also sees this correctly
none1@pci0:4:8:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none2@pci0:4:9:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none3@pci0:4:10:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none4@pci0:4:11:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none5@pci0:4:12:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none6@pci0:4:13:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none7@pci0:4:14:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none8@pci0:4:15:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none9@pci0:4:16:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor none10@pci0:4:17:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processor- just doesn't load drivers for them for some reason (?)
Perhaps because they are VFs, but they are also listed as PCI devices, so they are present.
- just doesn't load drivers for them for some reason (?)
-
So before you enable SR-IOV you can't pass through the PCIe devices directly?
-
@stephenw10 said in Intel Xeon D-2796NT and QAT:
So before you enable SR-IOV you can't pass through the PCIe devices directly?
Yes, it was, thanks for the train of thought. :)
(control plane is needed)Only the pass-thr. the Virtual Functions does not work.
- I updated Intel FreeBSD 1.x VIB to the latest version on ESXi.
- I revoked all pass-thr. and PCI assignments in ESXi and pfS VM -- & SR-IOV off in ESXi on C62X
- After updating VIB, I was able to enable only the pass-thr. of C62x chipset device, which I transferred to the VM as a complete PCI device.
Seeing the complete device, pfSense was able to assign a driver to the QAT C62X.
[25.07.1-RELEASE][root@ngfw.rm.arpa]/root: sysctl -a | grep 'qat' qat0: <Intel c6xx QuickAssist> mem 0xffb40000-0xffb7ffff,0xffb00000-0xffb3ffff at device 8.0 numa-domain 0 on pci4 qat0: qat_dev0 started 8 acceleration engines qat0: FW version: 4.18.0 qat0: Excessive clock measure delay qat_ocf0: <QAT engine> irq135: qat0:b0:277 @cpu0(domain0): 0 irq136: qat0:b1:279 @cpu0(domain0): 0 irq137: qat0:b2:281 @cpu0(domain0): 0 irq138: qat0:b3:283 @cpu0(domain0): 0 irq139: qat0:b4:285 @cpu0(domain0): 0 irq140: qat0:b5:287 @cpu0(domain0): 0 irq141: qat0:b6:289 @cpu0(domain0): 0 irq142: qat0:b7:291 @cpu0(domain0): 0 irq143: qat0:b8:293 @cpu0(domain0): 0 irq144: qat0:b9:295 @cpu0(domain0): 0 irq145: qat0:b10:297 @cpu0(domain0): 0 irq146: qat0:b11:299 @cpu0(domain0): 0 irq147: qat0:b12:301 @cpu0(domain0): 0 irq148: qat0:b13:303 @cpu0(domain0): 0 irq149: qat0:b14:305 @cpu0(domain0): 0 irq150: qat0:b15:307 @cpu0(domain0): 0 irq151: qat0:ae:309 @cpu0(domain0): 0 dev.qat_ocf.0.enable: 1 dev.qat_ocf.0.%iommu: dev.qat_ocf.0.%parent: nexus0 dev.qat_ocf.0.%pnpinfo: dev.qat_ocf.0.%location: dev.qat_ocf.0.%driver: qat_ocf dev.qat_ocf.0.%desc: QAT engine dev.qat_ocf.%parent: dev.qat.0.frequency: 685000000 dev.qat.0.cnv_error: dev.qat.0.fw_counters: dev.qat.0.mmp_version: 6.0.0 dev.qat.0.hw_version: 4 dev.qat.0.fw_version: 4.18.0 dev.qat.0.heartbeat: 1 dev.qat.0.heartbeat_failed: 0 dev.qat.0.heartbeat_sent: 1 dev.qat.0.dev_cfg: [GENERAL] dev.qat.0.num_user_processes: 0 dev.qat.0.cfg_mode: ks dev.qat.0.cfg_services: sym;dc dev.qat.0.state: up dev.qat.0.%domain: 0 dev.qat.0.%iommu: rid=0x440 dev.qat.0.%parent: pci4 dev.qat.0.%pnpinfo: vendor=0x8086 device=0x37c8 subvendor=0x8086 subdevice=0x0000 class=0x0b4000 dev.qat.0.%location: slot=8 function=0 dbsf=pci0:4:8:0 handle=\_SB_.PC0G.S9F0 dev.qat.0.%driver: qat dev.qat.0.%desc: Intel c6xx QuickAssist dev.qat.%parent:D-2145NT CPU only 1 "ch" QAT implemented
The original goal was not to assign all VFs to pfSense, but with this CPU, it is necessary because there is only one "ch" QAT.
For example, D-2187NT CPU has 3 channels (3x16VFs), where other VMs can also get a QAT device if needed.
The lesson is that cannot use SR-IOV first, but must need pass-thr. the entire device to the pfS guest.
On Linux, for example handles this and explicitly recommends that in case multiple QAT "CHs", - VFs should be symmetrically distributed to the guest(s).
D-2187NT offers greater flexibility if you want to run more than just pfSense guests on ESXi, as QAT capability remains available for other VMs.
I should note that I am STILL a fan of bare metal pfSense, but there are cases where paravirtualization can help. This is now the case.
And honestly, after so many years away, it was good to return to pfSense. It never disappoints. I did skip the PLUS licensing thing (but the TAC was fair), but Netgate is right: counterfeiters MUST be stopped somehow!
@stephenw10 - THX
