• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

23.09d - Is QAT Broken?

Plus 23.09 Development Snapshots (Retired)
10
86
16.5k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • R
    RobbieTT @jaltman
    last edited by RobbieTT Oct 8, 2023, 12:55 PM Oct 8, 2023, 12:42 PM

    @jaltman So it just comes down to the version of OpenSSL being used is not built with QAT support?

    I ask because openSSL v3.0.10 is specifically called for in the freeBSD QAT requirements and pfSense uses that very same version:

    /root: openssl version
    OpenSSL 3.0.10 1 Aug 2023 (Library: OpenSSL 3.0.10 1 Aug 2023)
    
    

    β˜•οΈ

    J 1 Reply Last reply Oct 8, 2023, 1:39 PM Reply Quote 0
    • J
      jaltman @RobbieTT
      last edited by Oct 8, 2023, 1:39 PM

      @RobbieTT OpenSSL 3.0 is used by FreeBSD but the QAT Engine and its dependencies (ipp-crypto-mb, ipsec-mb, qatlib) are not part of the base OpenSSL 3.0 build.

      For example, on Fedora Linux you need to install

        intel-ipp-crypto-mb-1.0.8-3.fc37.x86_64        intel-ipsec-mb-1.4.0-1.fc37.x86_64          qatengine-1.4.0-1.fc37.x86_64       
        qatlib-23.02.0-1.fc37.x86_64                   qatlib-service-23.02.0-1.fc37.x86_64      
      

      only then can the OpenSSL QAT Engine be used

      [jaltman@fc36]$ ls /usr/lib64/engines-3/
      afalg.so  capi.so  libpkcs11.so  loader_attic.so  padlock.so  pkcs11.so  qatengine.so
      [jaltman@fc37]$ openssl engine -t -c -v qatengine
      QAT_SW - Processor unsupported: AVX512F = 0, VAES = 0, VPCLMULQDQ = 0
      (qatengine) Reference implementation of QAT crypto engine(qat_hw & qat_sw) v1.4.0
       [RSA, AES-128-CBC-HMAC-SHA256, AES-256-CBC-HMAC-SHA256, ChaCha20-Poly1305, id-aes128-GCM, id-aes192-GCM, id-aes256-GCM, SHA3-256, SHA3-384, SHA3-512, TLS1-PRF, X25519, X448, SM2]
           [ available ]
           ENABLE_EXTERNAL_POLLING, POLL, SET_INSTANCE_FOR_THREAD, 
           GET_NUM_OP_RETRIES, SET_MAX_RETRY_COUNT, SET_INTERNAL_POLL_INTERVAL, 
           GET_EXTERNAL_POLLING_FD, ENABLE_EVENT_DRIVEN_POLLING_MODE, 
           GET_NUM_CRYPTO_INSTANCES, DISABLE_EVENT_DRIVEN_POLLING_MODE, 
           SET_EPOLL_TIMEOUT, SET_CRYPTO_SMALL_PACKET_OFFLOAD_THRESHOLD, 
           ENABLE_INLINE_POLLING, ENABLE_HEURISTIC_POLLING, 
           GET_NUM_REQUESTS_IN_FLIGHT, INIT_ENGINE, SET_CONFIGURATION_SECTION_NAME, 
           ENABLE_SW_FALLBACK, HEARTBEAT_POLL, DISABLE_QAT_OFFLOAD, HW_ALGO_BITMAP, 
           SW_ALGO_BITMAP
      

      As far as I can tell there is no qatengine.so packaged for OpenSSL 3.0, 3.1 or 3.2 on FreeBSD 14. Hence it cannot be installed and cannot be used.

      1 Reply Last reply Reply Quote 0
      • S
        stephenw10 Netgate Administrator
        last edited by Oct 8, 2023, 4:32 PM

        Mmm, as I read it OpenSSL requires the qat engine module to use it in user mode. Interesting that it does use it in 23.05... πŸ€”

        M R J 3 Replies Last reply Oct 8, 2023, 4:40 PM Reply Quote 1
        • M
          michmoor LAYER 8 Rebel Alliance @stephenw10
          last edited by Oct 8, 2023, 4:40 PM

          @stephenw10 following this thread for a while and that’s the general concern here. Why is this behavior different?

          Firewall: NetGate,Palo Alto-VM,Juniper SRX
          Routing: Juniper, Arista, Cisco
          Switching: Juniper, Arista, Cisco
          Wireless: Unifi, Aruba IAP
          JNCIP,CCNP Enterprise

          1 Reply Last reply Reply Quote 0
          • S
            stephenw10 Netgate Administrator
            last edited by Oct 8, 2023, 4:47 PM

            It's almost certainly because we moved to OpenSSL 3 and there is fallout from that. Most of that has been resolved. Since user mode encryption off-load is generally not supported this was probably just overlooked. I'll see what I can do when I'm home tomorrow.

            M 1 Reply Last reply Oct 8, 2023, 4:48 PM Reply Quote 1
            • M
              michmoor LAYER 8 Rebel Alliance @stephenw10
              last edited by Oct 8, 2023, 4:48 PM

              @stephenw10 thank you. Appreciate the quick response

              Firewall: NetGate,Palo Alto-VM,Juniper SRX
              Routing: Juniper, Arista, Cisco
              Switching: Juniper, Arista, Cisco
              Wireless: Unifi, Aruba IAP
              JNCIP,CCNP Enterprise

              1 Reply Last reply Reply Quote 0
              • J
                jimp Rebel Alliance Developer Netgate
                last edited by Oct 8, 2023, 4:53 PM

                I'm still not convinced anyone has accurately demonstrated that it was working on 23.05.1. There isn't any evidence that it was, just what may be coincidental increased in interrupt usage.

                And I think people missed the fact that there is support for userspace QAT in the 14 kernel driver but it's only for 4xxx devices. (See my post here: https://forum.netgate.com/post/1128163 )

                And the 14 man page:

                https://man.freebsd.org/cgi/man.cgi?query=qat&apropos=0&sektion=0&manpath=FreeBSD+14.0-STABLE&arch=default&format=html

                   cfg_mode
                     Override	the device mode	configuration  for  kernel  space  and
                     user  space  instances.	 Possible values: "ks",	"us", "ks;us".
                     Default value "ks;us".
                

                If userspace QAT was working on 23.05.1, anyone could replicate the results being claimed, but so far nobody else has been able to.

                Remember: Upvote with the πŸ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                Need help fast? Netgate Global Support!

                Do not Chat/PM for help!

                R 1 Reply Last reply Oct 8, 2023, 5:48 PM Reply Quote 0
                • R
                  RobbieTT @stephenw10
                  last edited by Oct 8, 2023, 5:03 PM

                  @stephenw10 said in 23.09d - Is QAT Broken?:

                  Mmm, as I read it OpenSSL requires the qat engine module to use it in user mode. Interesting that it does use it in 23.05... πŸ€”

                  Quite a few things have changed with 23.09d. The library of files used by OpenSSL is more expansive, the config files have changed and other new elements (eg Kea) have become users of OpenSSL.

                  Moving from the QAT-focused OpenSSL 1.1.1t-freebsd to the later OpenSSL 3.0.10 is also a significant delta.

                  There are other oddities between 23.05 and 23.09d. For example, the openssl engine on 23.05 used:

                  [23.05.1-RELEASE] /root: openssl engine 
                  (devcrypto) /dev/crypto engine
                  (rdrand) Intel RDRAND engine
                  (dynamic) Dynamic engine loading support
                  [23.05.1-RELEASE] /root: 
                  

                  With 23.09d the devcrypto line has been removed:

                  [23.09-DEVELOPMENT] /root: openssl engine
                  (rdrand) Intel RDRAND engine
                  (dynamic) Dynamic engine loading support
                  [23.09-DEVELOPMENT]/root:
                  

                  There also appears to be no /usr/lib/engines/qatengine.so file or indeed a qatengine.so anywhere on the system.

                  I have no difficulty replicating the QAT interrupts on 23.05.1. They don't increment by themselves, only when the firewall is doing a relevant task eg TLS/SSL. A simple DoT Dig that is forwarded is enough to increment, as will a curl, package update etc. Not sure I am believed though, for reasons that escape me.

                  J 2 Replies Last reply Oct 8, 2023, 6:03 PM Reply Quote 0
                  • R
                    RobbieTT @jimp
                    last edited by RobbieTT Oct 8, 2023, 8:25 PM Oct 8, 2023, 5:48 PM

                    @jimp said in 23.09d - Is QAT Broken?:

                    And I think people missed the fact that there is support for userspace QAT in the 14 kernel driver but it's only for 4xxx devices. (See my post here: https://forum.netgate.com/post/1128163 )

                    And the 14 man page:

                    https://man.freebsd.org/cgi/man.cgi?query=qat&apropos=0&sektion=0&manpath=FreeBSD+14.0-STABLE&arch=default&format=html

                    Jim, the 4xxx message could be linked to an errata elsewhere in pfSense as it has been missed from one of the lists. It is included in the actual FW lists though. There was a post on this subject a few days ago which @stephenw10 covered. Of course, being a later QAT generation, it will have key differences to the earlier generations QAT in the C3xxx and probably adds a brace of expanded capabilities.

                    The man pages you linked to makes no mention of userspace being limited to 4xxx either and it is grouped in the same list as the C3xxx. That does not make it untrue either, just less than clear.

                    I agree though that 23.09d is limited to kernel space (ks) only but I don't think that is attributed to freeBSD 14.0 alone. That change may have been brought about by pfSense+ and its current configuration.

                    pfSense 23.05.1 is also on freeBSD 14 and it is flagged to run in the default kernel space + user space (ks;us) mode.

                    23.05.1:

                    [23.05.1-RELEASE]/root: sysctl -a | grep "cfg"
                    hw.pci.mcfg: 1
                    dev.qat.0.dev_cfg: [GENERAL]
                    [23.05.1-RELEASE]/root: 
                    

                    23.09d - 'us' mode has been disabled, leaving only 'ks' mode enabled:

                    [23.09-DEVELOPMENT]/root: sysctl -a | grep "cfg"
                    hw.pci.mcfg: 1
                    dev.qat.0.dev_cfg: [GENERAL]
                    dev.qat.0.cfg_mode: ks
                    dev.qat.0.cfg_services: sym;dc
                    [23.09-DEVELOPMENT]/root: 
                    

                    I really hope someone will check my findings as not being believed feels pretty odd.

                    β˜•οΈ

                    J 1 Reply Last reply Oct 9, 2023, 1:29 PM Reply Quote 0
                    • J
                      jaltman @stephenw10
                      last edited by Oct 8, 2023, 5:58 PM

                      @stephenw10 said in 23.09d - Is QAT Broken?:

                      Mmm, as I read it OpenSSL requires the qat engine module to use it in user mode. Interesting that it does use it in 23.05... πŸ€”

                      OpenSSL 1.1.x also requires the QAT Engine in order to support use of QuickAssist. The Intel QAT Engine for OpenSSL was developed against OpenSSL 1.1 on FreeBSD 12.4. However, that release doesn't package or ship the engine.

                      I have seen no evidence on my 4100 when running 23.05.1 that QAT is being used by userspace. There is a small increase in the qat counters in kernel but I cannot believe that they are result of any userspace cryptographic or compression or signing operations.

                      1 Reply Last reply Reply Quote 0
                      • J
                        jaltman @RobbieTT
                        last edited by Oct 8, 2023, 6:03 PM

                        @RobbieTT said in 23.09d - Is QAT Broken?:

                        [23.05.1-RELEASE] /root: openssl engine
                        (devcrypto) /dev/crypto engine

                        It is possible that QAT on 23.05.1 is triggered for random number generation since /dev/crypto operates in kernel and has access to QAT. Any such usage would not be for encryption, compression or signing of actual network traffic.

                        1 Reply Last reply Reply Quote 0
                        • J
                          jaltman @RobbieTT
                          last edited by jaltman Oct 8, 2023, 6:29 PM Oct 8, 2023, 6:07 PM

                          @RobbieTT said in 23.09d - Is QAT Broken?:

                          I have no difficulty replicating the QAT interrupts on 23.05.1. They don't increment by themselves, only when the firewall is doing a relevant task eg TLS/SSL. A simple DoT Dig that is forwarded is enough to increment, as will a curl, package update etc. Not sure I am believed though, for reasons that escape me.

                          I believe that the interrupts occur because I see them as well. I do not believe that it has anything to do with use of QAT to encrypt, sign, or compress network traffic because I understand how QAT plugs into OpenSSL libcrypto and none of nginx, apache, curl, sshd, ssh, kerberos, etc that rely upon libcrypto for encryption, signing and compression primitives would contain any internal code to call into the QAT driver.

                          libz also has to be built custom in order to make use of QAT.

                          J 1 Reply Last reply Oct 8, 2023, 7:52 PM Reply Quote 0
                          • J
                            jaltman @jaltman
                            last edited by Oct 8, 2023, 7:52 PM

                            I've sent private mail to Bernard Spil, the maintainer of OpenSSL for FreeBSD, asking him if and how QAT is supported in the FreeBSD builds.
                            I will report on his response when I receive it.

                            R 1 Reply Last reply Oct 8, 2023, 8:29 PM Reply Quote 1
                            • R
                              RobbieTT @jaltman
                              last edited by Oct 8, 2023, 8:29 PM

                              @jaltman That would be enormously helpful - thank you. πŸ‘

                              β˜•οΈ

                              J 1 Reply Last reply Oct 8, 2023, 9:02 PM Reply Quote 0
                              • S
                                stephenw10 Netgate Administrator
                                last edited by Oct 8, 2023, 8:47 PM

                                Mmm, if it was supported in user-space I would expect to be able to see it very easily when using OpenVPN without DCO mode. With DCO is uses the kernel-mode crypto framework.

                                1 Reply Last reply Reply Quote 0
                                • J
                                  jaltman @RobbieTT
                                  last edited by Oct 8, 2023, 9:02 PM

                                  @RobbieTT Bernard confirms QAT functionality has never been packaged by him for FreeBSD. He suggests that someone else should build it and submit a ports request.

                                  He wants whoever supports it to have hardware on which to test it.

                                  J 1 Reply Last reply Oct 9, 2023, 11:45 AM Reply Quote 0
                                  • J
                                    jaltman @jaltman
                                    last edited by Oct 9, 2023, 11:45 AM

                                    @jaltman Moin Rahman did the earlier work on QAT support for the kernel and OpenSSL engine as one of his former employers was interested. However, that company went in a different direction leveraging programmable NICs instead after Intel abandoned the dedicated QAT add-on boards during the FreeBSD 13 time frame.

                                    1 Reply Last reply Reply Quote 0
                                    • J
                                      jimp Rebel Alliance Developer Netgate @RobbieTT
                                      last edited by jimp Oct 9, 2023, 1:30 PM Oct 9, 2023, 1:29 PM

                                      @RobbieTT said in 23.09d - Is QAT Broken?:

                                      @jimp said in 23.09d - Is QAT Broken?:
                                      Jim, the 4xxx message could be linked to an errata elsewhere in pfSense as it has been missed from one of the lists. It is included in the actual FW lists though. There was a post on this subject a few days ago which @stephenw10 covered. Of course, being a later QAT generation, it will have key differences to the earlier generations QAT in the C3xxx and probably adds a brace of expanded capabilities.

                                      The message saying userspace QAT only supported on 4xxx is from FreeBSD, not pfSense.

                                      https://github.com/freebsd/freebsd-src/blob/3523f0677ef514fe72710033c73cc58517b9cda8/sys/dev/qat/qat_common/adf_cfg_device.c#L700

                                      The man pages you linked to makes no mention of userspace being limited to 4xxx either and it is grouped in the same list as the C3xxx. That does not make it untrue either, just less than clear.

                                      Try setting a loader.conf.local tunable for dev.qat.0.cfg_mode="ks;us" yourself and see.

                                      You can't compare 23.05.1 directly because it did not have that tunable so I don't get where your assertion is coming from that it had both enabled there. Your shell output doesn't show that. There is nothing in the 23.05.1 output showing userspace.

                                      Remember: Upvote with the πŸ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                      Need help fast? Netgate Global Support!

                                      Do not Chat/PM for help!

                                      R 1 Reply Last reply Oct 9, 2023, 1:41 PM Reply Quote 0
                                      • R
                                        RobbieTT @jimp
                                        last edited by Oct 9, 2023, 1:41 PM

                                        @jimp said in 23.09d - Is QAT Broken?:

                                        You can't compare 23.05.1 directly because it did not have that tunable so I don't get where your assertion is coming from that it had both enabled there. Your shell output doesn't show that. There is nothing in the 23.05.1 output showing userspace.

                                        Understood but the man pages lists ks;us as the default and the absence of an explicit command to demure from that usually equals that the default is set. Apologies if this is not the case.

                                        β˜•οΈ

                                        J 1 Reply Last reply Oct 9, 2023, 1:50 PM Reply Quote 0
                                        • J
                                          jimp Rebel Alliance Developer Netgate @RobbieTT
                                          last edited by Oct 9, 2023, 1:50 PM

                                          @RobbieTT said in 23.09d - Is QAT Broken?:

                                          @jimp said in 23.09d - Is QAT Broken?:

                                          You can't compare 23.05.1 directly because it did not have that tunable so I don't get where your assertion is coming from that it had both enabled there. Your shell output doesn't show that. There is nothing in the 23.05.1 output showing userspace.

                                          Understood but the man pages lists ks;us as the default and the absence of an explicit command to demure from that usually equals that the default is set. Apologies if this is not the case.

                                          The man page is not complete/accurate there. It's only the default on 4xxx devices as well, for all others it defaults to kernel only.

                                          https://github.com/freebsd/freebsd-src/blob/main/sys/dev/qat/qat_common/adf_cfg.c#L37

                                          Remember: Upvote with the πŸ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                          Need help fast? Netgate Global Support!

                                          Do not Chat/PM for help!

                                          R 1 Reply Last reply Oct 9, 2023, 2:15 PM Reply Quote 1
                                          63 out of 86
                                          • First post
                                            63/86
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.