Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    TX queues on VMX interface

    Scheduled Pinned Locked Moved General pfSense Questions
    4 Posts 3 Posters 1.4k Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J Offline
      Josef
      last edited by

      Hi,
      Hoping someone can help.
      I am using a virtual pfsense instance on ESX 6.7 with 8 vCPU cores, and I am using vmxnet3 interfaces.
      I only have 2 interfaces, and I want to use multi-queue to better utilize all CPU cores.
      I cannot get more than 1 TX queue to be enabled. No matter what I do, there is always only 1.
      I can get multiple RX queues, but only 1 TX queue.

      In the FreeBSD vmx driver manual, it says to set the following parameters in the /boot/loader.conf file, which I have done:
      hw.pci.honor_msi_blacklist="0"
      hw.vmx.txnqueue="4"
      hw.vmx.rxnqueue="4"

      What ever I set for rxnqueue is honoured . But txnqueue seems to be ignored.
      Likewise, if I remove the two settings, leaving just the msi_blacklist line, the defaults (for rx) apply - which should be 8 for both.
      But again, I see 8 rx queues and still only 1 tx queue.

      root: vmstat -i | grep vmx0
      irq266: vmx0:tq0 11663 13
      irq270: vmx0:rq0 5302 6
      irq271: vmx0:rq1 1065 1
      irq272: vmx0:rq2 4444 5
      irq273: vmx0:rq3 4821 5

      Thanks!!

      1 Reply Last reply Reply Quote 0
      • stephenw10S Offline
        stephenw10 Netgate Administrator
        last edited by

        Hmm, I would expect it to use them there if they are available.

        Check the boot log perhaps.

        You have no other loader lines limiting that?

        Steve

        1 Reply Last reply Reply Quote 0
        • J Offline
          Josef
          last edited by

          Thanks.
          If I boot with verbose, I see:
          /root: cat /var/log/dmesg.boot | grep vmx0
          vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x3000-0x300f mem 0xfe103000-0xfe103fff,0xfe102000-0xfe102fff,0xfe100000-0xfe101fff at device 0.0 on pci4
          vmx0: attempting to allocate 9 MSI-X vectors (25 supported)
          vmx0: using IRQs 266-274 for MSI-X
          vmx0: bpf attached
          vmx0: Ethernet address: 00:50:56:b4:54:ae
          random: harvesting attach, 8 bytes (4 bits) from vmx0

          So its correctly using MSI-X and its only trying to allocate 9 vectors - which is what I expect.
          But vmstat -i still shows only 1 tx queue.
          vmstat -i | grep vmx0
          irq266: vmx0:tq0 5089 5
          irq270: vmx0:rq0 3519 3
          irq271: vmx0:rq1 394 0
          irq272: vmx0:rq2 2618 3
          irq273: vmx0:rq3 2057 2

          However, when I look at the process list, I see:
          root: ps -axdHO lwp | grep vmx0
          35771 100328 0 S+ 0:00.00 | `-- grep vmx0
          12 100114 - WL 0:00.04 - [intr/irq266: vmx0:t]
          12 100115 - WL 0:00.00 - [intr/irq267: vmx0:t]
          12 100116 - WL 0:00.00 - [intr/irq268: vmx0:t]
          12 100117 - WL 0:00.00 - [intr/irq269: vmx0:t]
          12 100118 - WL 0:00.09 - [intr/irq270: vmx0:r]
          12 100119 - WL 0:00.02 - [intr/irq271: vmx0:r]
          12 100120 - WL 0:00.11 - [intr/irq272: vmx0:r]
          12 100121 - WL 0:00.10 - [intr/irq273: vmx0:r]

          Which if I understand correctly (that vmx0:t refers to a TX queue), does indicate there are 4 of each queue.
          So perhaps I am misunderstanding how to actually see how many queues there are?
          In all the examples I have seen, all TX and RX queues are shown in vmstat -i

          Cheers

          1 Reply Last reply Reply Quote 0
          • C Offline
            chrcoluk
            last edited by chrcoluk

            add this to /boot/loader.conf.local and reboot

            # allow multi queue support on vmx
            hw.pci.honor_msi_blacklist="0"
            

            pfSense CE 2.8.0

            1 Reply Last reply Reply Quote 0
            • First post
              Last post
            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.