IQA89601G1P5 QAT not working

Impovich

Hello all, having an issue with QAT that I do not understand, so need some assistance.
Pfsense Plus: 23.01-RELEASE(Virtualized under XCP-NG)
adapter passed to VM

c5:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
c6:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
c7:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)

QAT adapter: IQA89601G1P5
QAT is activated but it is inactive.

[23.01-RELEASE][root@pfSense.home.lan]/root: kldstat -v | grep qat
11    1 0xffffffff8475d000     4348 qat.ko (/boot/kernel/qat.ko)
		695 nexus/qat
12    5 0xffffffff84762000    10e10 qat_hw.ko (/boot/kernel/qat_hw.ko)
		694 pci/qat_c4xxx
		690 pci/qat_c62x
		693 pci/qat_dh895xcc
		691 pci/qat_200xx
		692 pci/qat_c3xxx
13    7 0xffffffff84773000    29840 qat_common.ko (/boot/kernel/qat_common.ko)
		688 qat_common
14    6 0xffffffff8479d000    66b78 qat_api.ko (/boot/kernel/qat_api.ko)
		689 qat_api
15    1 0xffffffff84804000    11240 qat_c2xxx.ko (/boot/kernel/qat_c2xxx.ko)
		696 pci/qat_c2xxx

[23.01-RELEASE][root@pfSense.home.lan]/root: dmesg | grep qat
qat0: <Intel c6xx QuickAssist> mem 0xf1800000-0xf183ffff,0xf1840000-0xf187ffff irq 21 at device 9.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1880000-0xf18bffff,0xf18c0000-0xf18fffff irq 25 at device 10.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1900000-0xf193ffff,0xf1940000-0xf197ffff irq 29 at device 11.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1800000-0xf183ffff,0xf1840000-0xf187ffff irq 21 at device 9.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1880000-0xf18bffff,0xf18c0000-0xf18fffff irq 25 at device 10.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1900000-0xf193ffff,0xf1940000-0xf197ffff irq 29 at device 11.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6
qat0: <Intel c6xx QuickAssist> mem 0xf1800000-0xf183ffff,0xf1840000-0xf187ffff irq 21 at device 9.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1880000-0xf18bffff,0xf18c0000-0xf18fffff irq 25 at device 10.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1900000-0xf193ffff,0xf1940000-0xf197ffff irq 29 at device 11.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19

Impovich

Found this on xen host, not sure, but maybe it is related

[21:44 dallas ~]# dmesg | grep pciback
[    0.000000] Command line: root=LABEL=root-bhmxgw ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:c5:00.0)(0000:c6:00.0)(0000:c7:00.0)
[    1.811384] Kernel command line: root=LABEL=root-bhmxgw ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:c5:00.0)(0000:c6:00.0)(0000:c7:00.0)
[    4.419889] pciback 0000:c5:00.0: seizing device
[    4.419895] pciback 0000:c6:00.0: seizing device
[    4.419900] pciback 0000:c7:00.0: seizing device
[    4.771730] pciback 0000:c7:00.0: enabling device (0140 -> 0142)
[    4.771804] pciback 0000:c7:00.0: Disabling System Error reporting on root port c2:04.0
[    4.876894] pciback 0000:c6:00.0: enabling device (0140 -> 0142)
[    4.984902] pciback 0000:c5:00.0: enabling device (0140 -> 0142)
[    5.092811] xen_pciback: backend is vpci

c2:04.0 is:

c2:04.0 PCI bridge: Intel Corporation Device 347c (rev 04)

stephenw10

Hmm, I don't think we've seen that before. Could just be no-one has tried that particular combination yet. It does look like it's not passing that device through correctly. Or not completely.

orange.guru

I am seeing this error on my Super Micro 1541, seems to be related. Although in my case I am not virtualized, running a fresh installation of 23.01.

Any ideas what could cause these events and how they might be fixed:

Apr 27 01:56:29	kernel		qat_ocf0: <QAT engine>
Apr 27 01:56:29	kernel		qat_ocf0: no QAT IRQ instances available
Apr 27 01:56:29	kernel		device_attach: qat_ocf0 attach returned 6

@Impovich, do you have SRIOV enabled on the hypervisor?

Full disclosure, I am just guessing and grasping at straws as I have been unable to activate QAT on my particular box.

stephenw10

That's a different error. Is it the same QAT adapter?

orange.guru

@stephenw10

This is the on-chip QAT that is present on the Xeon-D1541. Its kind-of-puzzling, there is not much on the web about this particular problem.

orange.guru

@impovich said in IQA89601G1P5 QAT not working:

Unable to find AER capability of the device

I would possibly test by verifying that a few BIOS features are enabled on the system that is running the Xen Hypervisor. Then passthru the QAT device using SR-IOV so that VM can get more direct access to the QAT device thru virtual function,

I find the error about AER interesting as there is a related BIOS feature that can be found on some systems to present this functionality. When it comes to AER, it may be because the QAT device is not passed thru with SR-IOV so its unable to read the AER Assertions.

I hope my opinion helps, this is my best guess based on some other things I have worked on.

VT-d - (Intel Virtualization Technology for Directed I/O)
PCI-E AER - (PCI Advanced Error Reporting)
SR-IOV - (Single Root I/O Virtualization)

https://01.org/sites/default/files/downloads/intelr-quickassist-technology/330689qatvirtualizationappnoter006usreview.pdf

This guide has some information on how this is done with KVM, I know its not Xen, but it might be worth a look and some consideration.

Dobby_

@orange-guru said in IQA89601G1P5 QAT not working:

This is the on-chip QAT that is present on the Xeon-D1541

Activate it in the BIOS
Open-VM-Tools packet installation
Set up in pfSense to take advantage of that QAT

Erweiterte Einstellungen.jpg

stephenw10

The Xeon D-1541 does not have any on chip QAT hardware.
That's why the driver is unable to attach.

Steve

orange.guru

@dobby_

This is what I have in the GUI:

orange.guru

@stephenw10 said in IQA89601G1P5 QAT not working:

Xeon D-1541

Oh Jeez, you are right, QAT didn't hit the Xeon-D until the next generation. I will see myself out, sorry for that.

Impovich

@orange-guru said in IQA89601G1P5 QAT not working:

VT-d - (Intel Virtualization Technology for Directed I/O)
PCI-E AER - (PCI Advanced Error Reporting)
SR-IOV - (Single Root I/O Virtualization)

Hi, thank you for your suggestion.

VT-d is enabled
SR-IOV is enabled
PCI-E-AER is missing from bios

besides that, I have Nvidia T400 that is passed to Ubuntu and works

stephenw10

@orange-guru said in IQA89601G1P5 QAT not working:

I will see myself out,

Easy mistake to make.

Impovich

updates from my investigation
Pfsense on bare metal - QAT IQA89601G1P5 works!
Pfsense virtualized with Proxmox - exactly the same story as with xcp-ng :(
Intel states that it is possible to passthrough some QAT adapters including mine to KVM
Document which describes how to do it

stephenw10

Does it work in vanilla FreeBSD? Or in Linux even?

Impovich

@stephenw10
Didn't try vanilla FreeBSD, only in PFsense 23.01-RELEASE
As for Linux, I installed drivers on a Proxmox host successfully and it looked like everything was fine

stephenw10

Right but did you try passing though the QAT device to a Linux VM?

Impovich

@stephenw10 no, will try

Impovich

@stephenw10
So another attempt:
Fresh proxmox installation
installed qat drivers and enabled SR-IOV
restored pfsense from backup
followed howto mentioned earlier in the thread

and now I'm getting this, any thoughts?

[23.01-RELEASE][root@pfSense.home.lan]/root: dmesg | grep qat
qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6
qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6

vmstat -i | grep qat - returns nothing

23.01-RELEASE][root@pfSense.home.lan]/root: kldstat -v | grep qat
 8    1 0xffffffff84751000     4348 qat.ko (/boot/kernel/qat.ko)
		691 nexus/qat
 9    5 0xffffffff84756000    10e10 qat_hw.ko (/boot/kernel/qat_hw.ko)
		690 pci/qat_c4xxx
		686 pci/qat_c62x
		689 pci/qat_dh895xcc
		687 pci/qat_200xx
		688 pci/qat_c3xxx
10    7 0xffffffff84767000    29840 qat_common.ko (/boot/kernel/qat_common.ko)
		684 qat_common
11    6 0xffffffff84791000    66b78 qat_api.ko (/boot/kernel/qat_api.ko)
		685 qat_api
12    1 0xffffffff847f8000    11240 qat_c2xxx.ko (/boot/kernel/qat_c2xxx.ko)
		692 pci/qat_c2xxx

[23.01-RELEASE][root@pfSense.home.lan]/root: /usr/bin/openssl engine -t -c
(dynamic) Dynamic engine loading support
     [ unavailable ]

none0@pci0:0:16:0:	class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'C62x Chipset QuickAssist Technology Virtual Function'
    class      = processor

On Proxmox host everything looks fine and the device uses the correct driver

54:01.0 0b40: 8086:37c9 (rev 04)
        Subsystem: 8086:0000
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        NUMA node: 0
        IOMMU group: 150
        Region 0: Memory at d0b90000 (64-bit, non-prefetchable) [virtual] [size=4K]
        Region 2: Memory at d0b80000 (64-bit, non-prefetchable) [virtual] [size=4K]
        Capabilities: [50] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <128ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM not supported
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed unknown (downgraded), Width x0 (downgraded)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [138 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [1b0 v1] Access Control Services
                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Kernel driver in use: vfio-pci
        Kernel modules: qat_c62xvf

Impovich

Maybe this is because drivers in pfsense are not configured with --enable-icp-sriov=guest

From intel manual

2. Install the Intel® QAT Software package on the Guest.
3. Enable the SR-IOV build on the host by using:
# ./configure --enable-icp-sriov=guest
4. Install the QAT software:
# make install