IQA89601G1P5 QAT not working

Impovich

@orange-guru said in IQA89601G1P5 QAT not working:

VT-d - (Intel Virtualization Technology for Directed I/O)
PCI-E AER - (PCI Advanced Error Reporting)
SR-IOV - (Single Root I/O Virtualization)

Hi, thank you for your suggestion.

VT-d is enabled
SR-IOV is enabled
PCI-E-AER is missing from bios

besides that, I have Nvidia T400 that is passed to Ubuntu and works

stephenw10

@orange-guru said in IQA89601G1P5 QAT not working:

I will see myself out,

Easy mistake to make.

Impovich

updates from my investigation
Pfsense on bare metal - QAT IQA89601G1P5 works!
Pfsense virtualized with Proxmox - exactly the same story as with xcp-ng :(
Intel states that it is possible to passthrough some QAT adapters including mine to KVM
Document which describes how to do it

stephenw10

Does it work in vanilla FreeBSD? Or in Linux even?

Impovich

@stephenw10
Didn't try vanilla FreeBSD, only in PFsense 23.01-RELEASE
As for Linux, I installed drivers on a Proxmox host successfully and it looked like everything was fine

stephenw10

Right but did you try passing though the QAT device to a Linux VM?

Impovich

@stephenw10 no, will try

Impovich

@stephenw10
So another attempt:
Fresh proxmox installation
installed qat drivers and enabled SR-IOV
restored pfsense from backup
followed howto mentioned earlier in the thread

and now I'm getting this, any thoughts?

[23.01-RELEASE][root@pfSense.home.lan]/root: dmesg | grep qat
qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6
qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6

vmstat -i | grep qat - returns nothing

23.01-RELEASE][root@pfSense.home.lan]/root: kldstat -v | grep qat
 8    1 0xffffffff84751000     4348 qat.ko (/boot/kernel/qat.ko)
		691 nexus/qat
 9    5 0xffffffff84756000    10e10 qat_hw.ko (/boot/kernel/qat_hw.ko)
		690 pci/qat_c4xxx
		686 pci/qat_c62x
		689 pci/qat_dh895xcc
		687 pci/qat_200xx
		688 pci/qat_c3xxx
10    7 0xffffffff84767000    29840 qat_common.ko (/boot/kernel/qat_common.ko)
		684 qat_common
11    6 0xffffffff84791000    66b78 qat_api.ko (/boot/kernel/qat_api.ko)
		685 qat_api
12    1 0xffffffff847f8000    11240 qat_c2xxx.ko (/boot/kernel/qat_c2xxx.ko)
		692 pci/qat_c2xxx

[23.01-RELEASE][root@pfSense.home.lan]/root: /usr/bin/openssl engine -t -c
(dynamic) Dynamic engine loading support
     [ unavailable ]

none0@pci0:0:16:0:	class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'C62x Chipset QuickAssist Technology Virtual Function'
    class      = processor

On Proxmox host everything looks fine and the device uses the correct driver

54:01.0 0b40: 8086:37c9 (rev 04)
        Subsystem: 8086:0000
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        NUMA node: 0
        IOMMU group: 150
        Region 0: Memory at d0b90000 (64-bit, non-prefetchable) [virtual] [size=4K]
        Region 2: Memory at d0b80000 (64-bit, non-prefetchable) [virtual] [size=4K]
        Capabilities: [50] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <128ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM not supported
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed unknown (downgraded), Width x0 (downgraded)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [138 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [1b0 v1] Access Control Services
                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Kernel driver in use: vfio-pci
        Kernel modules: qat_c62xvf

Impovich

Maybe this is because drivers in pfsense are not configured with --enable-icp-sriov=guest

From intel manual

2. Install the Intel® QAT Software package on the Guest.
3. Enable the SR-IOV build on the host by using:
# ./configure --enable-icp-sriov=guest
4. Install the QAT software:
# make install

stephenw10

@impovich said in IQA89601G1P5 QAT not working:

qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6

That is expected if there is simply no device present. or none with a driver attached.
And that appears to be the case here:

none0@pci0:0:16:0:	class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'C62x Chipset QuickAssist Technology Virtual Function'
    class      = processor

It looks like the FreeBSD driver simply doesn't support the virtual devices yet:
https://redmine.pfsense.org/issues/14173

Though looking at the code it does appear to be listed.
https://github.com/freebsd/freebsd-src/blob/main/sys/dev/qat/include/common/adf_accel_devices.h#L12
That is in the 23.01 code so you might expect it to at least try to attach.