Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    IQA89601G1P5 QAT not working

    Scheduled Pinned Locked Moved Hardware
    21 Posts 4 Posters 2.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator
      last edited by

      The Xeon D-1541 does not have any on chip QAT hardware.
      That's why the driver is unable to attach.

      Steve

      O 1 Reply Last reply Reply Quote 1
      • O
        orange.guru
        last edited by orange.guru

        @dobby_

        This is what I have in the GUI:

        145258ea-de48-40bd-964f-9167b864ea04-image.png

        866b10ae-e9c6-4180-b876-b85cf8ad781f-image.png

        1 Reply Last reply Reply Quote 0
        • O
          orange.guru @stephenw10
          last edited by orange.guru

          @stephenw10 said in IQA89601G1P5 QAT not working:

          Xeon D-1541

          Oh Jeez, you are right, QAT didn't hit the Xeon-D until the next generation. I will see myself out, sorry for that.

          stephenw10S 1 Reply Last reply Reply Quote 0
          • I
            Impovich @orange.guru
            last edited by

            @orange-guru said in IQA89601G1P5 QAT not working:

            VT-d - (Intel Virtualization Technology for Directed I/O)
            PCI-E AER - (PCI Advanced Error Reporting)
            SR-IOV - (Single Root I/O Virtualization)

            Hi, thank you for your suggestion.

            VT-d is enabled
            SR-IOV is enabled
            PCI-E-AER is missing from bios

            besides that, I have Nvidia T400 that is passed to Ubuntu and works

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator @orange.guru
              last edited by

              @orange-guru said in IQA89601G1P5 QAT not working:

              I will see myself out,

              😁

              Easy mistake to make.

              I 1 Reply Last reply Reply Quote 0
              • I
                Impovich @stephenw10
                last edited by Impovich

                updates from my investigation
                Pfsense on bare metal - QAT IQA89601G1P5 works!
                Pfsense virtualized with Proxmox - exactly the same story as with xcp-ng :(
                Intel states that it is possible to passthrough some QAT adapters including mine to KVM
                Document which describes how to do it

                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by

                  Does it work in vanilla FreeBSD? Or in Linux even?

                  I 1 Reply Last reply Reply Quote 0
                  • I
                    Impovich @stephenw10
                    last edited by

                    @stephenw10
                    Didn't try vanilla FreeBSD, only in PFsense 23.01-RELEASE
                    As for Linux, I installed drivers on a Proxmox host successfully and it looked like everything was fine

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      Right but did you try passing though the QAT device to a Linux VM?

                      I 2 Replies Last reply Reply Quote 0
                      • I
                        Impovich @stephenw10
                        last edited by Impovich

                        @stephenw10 no, will try

                        1 Reply Last reply Reply Quote 1
                        • I
                          Impovich @stephenw10
                          last edited by Impovich

                          @stephenw10
                          So another attempt:
                          Fresh proxmox installation
                          installed qat drivers and enabled SR-IOV
                          restored pfsense from backup
                          followed howto mentioned earlier in the thread

                          and now I'm getting this, any thoughts?

                          [23.01-RELEASE][root@pfSense.home.lan]/root: dmesg | grep qat
qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6
qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6

                          vmstat -i | grep qat - returns nothing

                          23.01-RELEASE][root@pfSense.home.lan]/root: kldstat -v | grep qat
 8    1 0xffffffff84751000     4348 qat.ko (/boot/kernel/qat.ko)
		691 nexus/qat
 9    5 0xffffffff84756000    10e10 qat_hw.ko (/boot/kernel/qat_hw.ko)
		690 pci/qat_c4xxx
		686 pci/qat_c62x
		689 pci/qat_dh895xcc
		687 pci/qat_200xx
		688 pci/qat_c3xxx
10    7 0xffffffff84767000    29840 qat_common.ko (/boot/kernel/qat_common.ko)
		684 qat_common
11    6 0xffffffff84791000    66b78 qat_api.ko (/boot/kernel/qat_api.ko)
		685 qat_api
12    1 0xffffffff847f8000    11240 qat_c2xxx.ko (/boot/kernel/qat_c2xxx.ko)
		692 pci/qat_c2xxx

                          [23.01-RELEASE][root@pfSense.home.lan]/root: /usr/bin/openssl engine -t -c
(dynamic) Dynamic engine loading support
     [ unavailable ]

                          none0@pci0:0:16:0:	class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'C62x Chipset QuickAssist Technology Virtual Function'
    class      = processor

                          On Proxmox host everything looks fine and the device uses the correct driver

                          54:01.0 0b40: 8086:37c9 (rev 04)
        Subsystem: 8086:0000
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        NUMA node: 0
        IOMMU group: 150
        Region 0: Memory at d0b90000 (64-bit, non-prefetchable) [virtual] [size=4K]
        Region 2: Memory at d0b80000 (64-bit, non-prefetchable) [virtual] [size=4K]
        Capabilities: [50] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <128ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM not supported
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed unknown (downgraded), Width x0 (downgraded)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [138 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [1b0 v1] Access Control Services
                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Kernel driver in use: vfio-pci
        Kernel modules: qat_c62xvf
                          I stephenw10S 2 Replies Last reply Reply Quote 0
                          • I
                            Impovich @Impovich
                            last edited by Impovich

                            Maybe this is because drivers in pfsense are not configured with --enable-icp-sriov=guest

                            From intel manual

                            2. Install the Intel® QAT Software package on the Guest.
3. Enable the SR-IOV build on the host by using:
# ./configure --enable-icp-sriov=guest
4. Install the QAT software:
# make install
                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator @Impovich
                              last edited by

                              @impovich said in IQA89601G1P5 QAT not working:

                              qat_ocf0: <QAT engine>
                              qat_ocf0: no QAT IRQ instances available
                              device_attach: qat_ocf0 attach returned 6

                              That is expected if there is simply no device present. or none with a driver attached.
                              And that appears to be the case here:

                              none0@pci0:0:16:0:	class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'C62x Chipset QuickAssist Technology Virtual Function'
    class      = processor

                              It looks like the FreeBSD driver simply doesn't support the virtual devices yet:
                              https://redmine.pfsense.org/issues/14173

                              Though looking at the code it does appear to be listed.
                              https://github.com/freebsd/freebsd-src/blob/main/sys/dev/qat/include/common/adf_accel_devices.h#L12
                              That is in the 23.01 code so you might expect it to at least try to attach.

                              1 Reply Last reply Reply Quote 1
                              • First post
                                Last post
                              Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.