Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    IQA89601G1P5 QAT not working

    Scheduled Pinned Locked Moved
    Hardware
    4
    21
    1.3k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Dobby_D
      Dobby_ @orange.guru
      last edited by

      @orange-guru said in IQA89601G1P5 QAT not working:

      This is the on-chip QAT that is present on the Xeon-D1541

      • Activate it in the BIOS
      • Open-VM-Tools packet installation
      • Set up in pfSense to take advantage of that QAT

      Erweiterte Einstellungen.jpg

      QAT1.jpg

      QAT2.jpg

      #~. @Dobby

      Turris Omnia - 4 Ports - 2 GB RAM / TurrisOS 7 Release (Btrfs)
      PC Engines APU4D4 - 4 Ports - 4 GB RAM / pfSense CE 2.7.2 Release (ZFS)
      PC Engines APU6B4 - 4 Ports - 4 GB RAM / pfSense+ (Plus) 24.03_1 Release (ZFS)

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        The Xeon D-1541 does not have any on chip QAT hardware.
        That's why the driver is unable to attach.

        Steve

        O 1 Reply Last reply Reply Quote 1
        • O
          orange.guru
          last edited by orange.guru

          @dobby_

          This is what I have in the GUI:

          145258ea-de48-40bd-964f-9167b864ea04-image.png

          866b10ae-e9c6-4180-b876-b85cf8ad781f-image.png

          1 Reply Last reply Reply Quote 0
          • O
            orange.guru @stephenw10
            last edited by orange.guru

            @stephenw10 said in IQA89601G1P5 QAT not working:

            Xeon D-1541

            Oh Jeez, you are right, QAT didn't hit the Xeon-D until the next generation. I will see myself out, sorry for that.

            stephenw10S 1 Reply Last reply Reply Quote 0
            • I
              Impovich @orange.guru
              last edited by

              @orange-guru said in IQA89601G1P5 QAT not working:

              VT-d - (Intel Virtualization Technology for Directed I/O)
              PCI-E AER - (PCI Advanced Error Reporting)
              SR-IOV - (Single Root I/O Virtualization)

              Hi, thank you for your suggestion.

              VT-d is enabled
              SR-IOV is enabled
              PCI-E-AER is missing from bios

              besides that, I have Nvidia T400 that is passed to Ubuntu and works

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator @orange.guru
                last edited by

                @orange-guru said in IQA89601G1P5 QAT not working:

                I will see myself out,

                😁

                Easy mistake to make.

                I 1 Reply Last reply Reply Quote 0
                • I
                  Impovich @stephenw10
                  last edited by Impovich

                  updates from my investigation
                  Pfsense on bare metal - QAT IQA89601G1P5 works!
                  Pfsense virtualized with Proxmox - exactly the same story as with xcp-ng :(
                  Intel states that it is possible to passthrough some QAT adapters including mine to KVM
                  Document which describes how to do it

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Does it work in vanilla FreeBSD? Or in Linux even?

                    I 1 Reply Last reply Reply Quote 0
                    • I
                      Impovich @stephenw10
                      last edited by

                      @stephenw10
                      Didn't try vanilla FreeBSD, only in PFsense 23.01-RELEASE
                      As for Linux, I installed drivers on a Proxmox host successfully and it looked like everything was fine

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        Right but did you try passing though the QAT device to a Linux VM?

                        I 2 Replies Last reply Reply Quote 0
                        • I
                          Impovich @stephenw10
                          last edited by Impovich

                          @stephenw10 no, will try

                          1 Reply Last reply Reply Quote 1
                          • I
                            Impovich @stephenw10
                            last edited by Impovich

                            @stephenw10
                            So another attempt:
                            Fresh proxmox installation
                            installed qat drivers and enabled SR-IOV
                            restored pfsense from backup
                            followed howto mentioned earlier in the thread

                            and now I'm getting this, any thoughts?

                            [23.01-RELEASE][root@pfSense.home.lan]/root: dmesg | grep qat
qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6
qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6

                            vmstat -i | grep qat - returns nothing

                            23.01-RELEASE][root@pfSense.home.lan]/root: kldstat -v | grep qat
 8    1 0xffffffff84751000     4348 qat.ko (/boot/kernel/qat.ko)
		691 nexus/qat
 9    5 0xffffffff84756000    10e10 qat_hw.ko (/boot/kernel/qat_hw.ko)
		690 pci/qat_c4xxx
		686 pci/qat_c62x
		689 pci/qat_dh895xcc
		687 pci/qat_200xx
		688 pci/qat_c3xxx
10    7 0xffffffff84767000    29840 qat_common.ko (/boot/kernel/qat_common.ko)
		684 qat_common
11    6 0xffffffff84791000    66b78 qat_api.ko (/boot/kernel/qat_api.ko)
		685 qat_api
12    1 0xffffffff847f8000    11240 qat_c2xxx.ko (/boot/kernel/qat_c2xxx.ko)
		692 pci/qat_c2xxx

                            [23.01-RELEASE][root@pfSense.home.lan]/root: /usr/bin/openssl engine -t -c
(dynamic) Dynamic engine loading support
     [ unavailable ]

                            none0@pci0:0:16:0:	class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'C62x Chipset QuickAssist Technology Virtual Function'
    class      = processor

                            On Proxmox host everything looks fine and the device uses the correct driver

                            54:01.0 0b40: 8086:37c9 (rev 04)
        Subsystem: 8086:0000
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        NUMA node: 0
        IOMMU group: 150
        Region 0: Memory at d0b90000 (64-bit, non-prefetchable) [virtual] [size=4K]
        Region 2: Memory at d0b80000 (64-bit, non-prefetchable) [virtual] [size=4K]
        Capabilities: [50] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <128ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM not supported
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed unknown (downgraded), Width x0 (downgraded)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [138 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [1b0 v1] Access Control Services
                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Kernel driver in use: vfio-pci
        Kernel modules: qat_c62xvf
                            I stephenw10S 2 Replies Last reply Reply Quote 0
                            • I
                              Impovich @Impovich
                              last edited by Impovich

                              Maybe this is because drivers in pfsense are not configured with --enable-icp-sriov=guest

                              From intel manual

                              2. Install the Intel® QAT Software package on the Guest.
3. Enable the SR-IOV build on the host by using:
# ./configure --enable-icp-sriov=guest
4. Install the QAT software:
# make install
                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator @Impovich
                                last edited by

                                @impovich said in IQA89601G1P5 QAT not working:

                                qat_ocf0: <QAT engine>
                                qat_ocf0: no QAT IRQ instances available
                                device_attach: qat_ocf0 attach returned 6

                                That is expected if there is simply no device present. or none with a driver attached.
                                And that appears to be the case here:

                                none0@pci0:0:16:0:	class=0x0b4000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37c9 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'C62x Chipset QuickAssist Technology Virtual Function'
    class      = processor

                                It looks like the FreeBSD driver simply doesn't support the virtual devices yet:
                                https://redmine.pfsense.org/issues/14173

                                Though looking at the code it does appear to be listed.
                                https://github.com/freebsd/freebsd-src/blob/main/sys/dev/qat/include/common/adf_accel_devices.h#L12
                                That is in the 23.01 code so you might expect it to at least try to attach.

                                1 Reply Last reply Reply Quote 1
                                • First post
                                  Last post