IQA89601G1P5 QAT not working
-
Found this on xen host, not sure, but maybe it is related
[21:44 dallas ~]# dmesg | grep pciback [ 0.000000] Command line: root=LABEL=root-bhmxgw ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:c5:00.0)(0000:c6:00.0)(0000:c7:00.0) [ 1.811384] Kernel command line: root=LABEL=root-bhmxgw ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:c5:00.0)(0000:c6:00.0)(0000:c7:00.0) [ 4.419889] pciback 0000:c5:00.0: seizing device [ 4.419895] pciback 0000:c6:00.0: seizing device [ 4.419900] pciback 0000:c7:00.0: seizing device [ 4.771730] pciback 0000:c7:00.0: enabling device (0140 -> 0142) [ 4.771804] pciback 0000:c7:00.0: Disabling System Error reporting on root port c2:04.0 [ 4.876894] pciback 0000:c6:00.0: enabling device (0140 -> 0142) [ 4.984902] pciback 0000:c5:00.0: enabling device (0140 -> 0142) [ 5.092811] xen_pciback: backend is vpcic2:04.0 is:
c2:04.0 PCI bridge: Intel Corporation Device 347c (rev 04) -
Hmm, I don't think we've seen that before. Could just be no-one has tried that particular combination yet. It does look like it's not passing that device through correctly. Or not completely.
-
I am seeing this error on my Super Micro 1541, seems to be related. Although in my case I am not virtualized, running a fresh installation of 23.01.
Any ideas what could cause these events and how they might be fixed:
Apr 27 01:56:29 kernel qat_ocf0: <QAT engine> Apr 27 01:56:29 kernel qat_ocf0: no QAT IRQ instances available Apr 27 01:56:29 kernel device_attach: qat_ocf0 attach returned 6@Impovich, do you have SRIOV enabled on the hypervisor?
Full disclosure, I am just guessing and grasping at straws as I have been unable to activate QAT on my particular box.
-
That's a different error. Is it the same QAT adapter?
-
This is the on-chip QAT that is present on the Xeon-D1541. Its kind-of-puzzling, there is not much on the web about this particular problem.
-
@impovich said in IQA89601G1P5 QAT not working:
Unable to find AER capability of the device
I would possibly test by verifying that a few BIOS features are enabled on the system that is running the Xen Hypervisor. Then passthru the QAT device using SR-IOV so that VM can get more direct access to the QAT device thru virtual function,
I find the error about AER interesting as there is a related BIOS feature that can be found on some systems to present this functionality. When it comes to AER, it may be because the QAT device is not passed thru with SR-IOV so its unable to read the AER Assertions.
I hope my opinion helps, this is my best guess based on some other things I have worked on.
VT-d - (Intel Virtualization Technology for Directed I/O)
PCI-E AER - (PCI Advanced Error Reporting)
SR-IOV - (Single Root I/O Virtualization)https://01.org/sites/default/files/downloads/intelr-quickassist-technology/330689qatvirtualizationappnoter006usreview.pdf
This guide has some information on how this is done with KVM, I know its not Xen, but it might be worth a look and some consideration.
-
@orange-guru said in IQA89601G1P5 QAT not working:
This is the on-chip QAT that is present on the Xeon-D1541
- Activate it in the BIOS
- Open-VM-Tools packet installation
- Set up in pfSense to take advantage of that QAT
-
The Xeon D-1541 does not have any on chip QAT hardware.
That's why the driver is unable to attach.Steve
-
-
@stephenw10 said in IQA89601G1P5 QAT not working:
Xeon D-1541
Oh Jeez, you are right, QAT didn't hit the Xeon-D until the next generation. I will see myself out, sorry for that.
-
@orange-guru said in IQA89601G1P5 QAT not working:
VT-d - (Intel Virtualization Technology for Directed I/O)
PCI-E AER - (PCI Advanced Error Reporting)
SR-IOV - (Single Root I/O Virtualization)Hi, thank you for your suggestion.
VT-d is enabled
SR-IOV is enabled
PCI-E-AER is missing from biosbesides that, I have Nvidia T400 that is passed to Ubuntu and works
-
-
updates from my investigation
Pfsense on bare metal - QAT IQA89601G1P5 works!
Pfsense virtualized with Proxmox - exactly the same story as with xcp-ng :(
Intel states that it is possible to passthrough some QAT adapters including mine to KVM
Document which describes how to do it -
Does it work in vanilla FreeBSD? Or in Linux even?
-
@stephenw10
Didn't try vanilla FreeBSD, only in PFsense 23.01-RELEASE
As for Linux, I installed drivers on a Proxmox host successfully and it looked like everything was fine -
Right but did you try passing though the QAT device to a Linux VM?
-
@stephenw10 no, will try
-
@stephenw10
So another attempt:
Fresh proxmox installation
installed qat drivers and enabled SR-IOV
restored pfsense from backup
followed howto mentioned earlier in the threadand now I'm getting this, any thoughts?
[23.01-RELEASE][root@pfSense.home.lan]/root: dmesg | grep qat qat_ocf0: <QAT engine> qat_ocf0: no QAT IRQ instances available device_attach: qat_ocf0 attach returned 6 qat_ocf0: <QAT engine> qat_ocf0: no QAT IRQ instances available device_attach: qat_ocf0 attach returned 6vmstat -i | grep qat - returns nothing
23.01-RELEASE][root@pfSense.home.lan]/root: kldstat -v | grep qat 8 1 0xffffffff84751000 4348 qat.ko (/boot/kernel/qat.ko) 691 nexus/qat 9 5 0xffffffff84756000 10e10 qat_hw.ko (/boot/kernel/qat_hw.ko) 690 pci/qat_c4xxx 686 pci/qat_c62x 689 pci/qat_dh895xcc 687 pci/qat_200xx 688 pci/qat_c3xxx 10 7 0xffffffff84767000 29840 qat_common.ko (/boot/kernel/qat_common.ko) 684 qat_common 11 6 0xffffffff84791000 66b78 qat_api.ko (/boot/kernel/qat_api.ko) 685 qat_api 12 1 0xffffffff847f8000 11240 qat_c2xxx.ko (/boot/kernel/qat_c2xxx.ko) 692 pci/qat_c2xxx[23.01-RELEASE][root@pfSense.home.lan]/root: /usr/bin/openssl engine -t -c (dynamic) Dynamic engine loading support [ unavailable ]none0@pci0:0:16:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processorOn Proxmox host everything looks fine and the device uses the correct driver
54:01.0 0b40: 8086:37c9 (rev 04) Subsystem: 8086:0000 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- NUMA node: 0 IOMMU group: 150 Region 0: Memory at d0b90000 (64-bit, non-prefetchable) [virtual] [size=4K] Region 2: Memory at d0b80000 (64-bit, non-prefetchable) [virtual] [size=4K] Capabilities: [50] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <128ns, L1 <1us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM not supported ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed unknown (downgraded), Width x0 (downgraded) TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR- 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- TPHComp- ExtTPHComp- AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled, AtomicOpsCtl: ReqEn- LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1- EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Capabilities: [138 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 0 ARICtl: MFVC- ACS-, Function Group: 0 Capabilities: [1b0 v1] Access Control Services ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- Kernel driver in use: vfio-pci Kernel modules: qat_c62xvf -
Maybe this is because drivers in pfsense are not configured with --enable-icp-sriov=guest
From intel manual
2. Install the Intel® QAT Software package on the Guest. 3. Enable the SR-IOV build on the host by using: # ./configure --enable-icp-sriov=guest 4. Install the QAT software: # make install -
@impovich said in IQA89601G1P5 QAT not working:
qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6That is expected if there is simply no device present. or none with a driver attached.
And that appears to be the case here:none0@pci0:0:16:0: class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000 vendor = 'Intel Corporation' device = 'C62x Chipset QuickAssist Technology Virtual Function' class = processorIt looks like the FreeBSD driver simply doesn't support the virtual devices yet:
https://redmine.pfsense.org/issues/14173Though looking at the code it does appear to be listed.
https://github.com/freebsd/freebsd-src/blob/main/sys/dev/qat/include/common/adf_accel_devices.h#L12
That is in the 23.01 code so you might expect it to at least try to attach.
