TX queues on VMX interface



  • Hi,
    Hoping someone can help.
    I am using a virtual pfsense instance on ESX 6.7 with 8 vCPU cores, and I am using vmxnet3 interfaces.
    I only have 2 interfaces, and I want to use multi-queue to better utilize all CPU cores.
    I cannot get more than 1 TX queue to be enabled. No matter what I do, there is always only 1.
    I can get multiple RX queues, but only 1 TX queue.

    In the FreeBSD vmx driver manual, it says to set the following parameters in the /boot/loader.conf file, which I have done:
    hw.pci.honor_msi_blacklist="0"
    hw.vmx.txnqueue="4"
    hw.vmx.rxnqueue="4"

    What ever I set for rxnqueue is honoured . But txnqueue seems to be ignored.
    Likewise, if I remove the two settings, leaving just the msi_blacklist line, the defaults (for rx) apply - which should be 8 for both.
    But again, I see 8 rx queues and still only 1 tx queue.

    root: vmstat -i | grep vmx0
    irq266: vmx0:tq0 11663 13
    irq270: vmx0:rq0 5302 6
    irq271: vmx0:rq1 1065 1
    irq272: vmx0:rq2 4444 5
    irq273: vmx0:rq3 4821 5

    Thanks!!


  • Netgate Administrator

    Hmm, I would expect it to use them there if they are available.

    Check the boot log perhaps.

    You have no other loader lines limiting that?

    Steve



  • Thanks.
    If I boot with verbose, I see:
    /root: cat /var/log/dmesg.boot | grep vmx0
    vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x3000-0x300f mem 0xfe103000-0xfe103fff,0xfe102000-0xfe102fff,0xfe100000-0xfe101fff at device 0.0 on pci4
    vmx0: attempting to allocate 9 MSI-X vectors (25 supported)
    vmx0: using IRQs 266-274 for MSI-X
    vmx0: bpf attached
    vmx0: Ethernet address: 00:50:56:b4:54:ae
    random: harvesting attach, 8 bytes (4 bits) from vmx0

    So its correctly using MSI-X and its only trying to allocate 9 vectors - which is what I expect.
    But vmstat -i still shows only 1 tx queue.
    vmstat -i | grep vmx0
    irq266: vmx0:tq0 5089 5
    irq270: vmx0:rq0 3519 3
    irq271: vmx0:rq1 394 0
    irq272: vmx0:rq2 2618 3
    irq273: vmx0:rq3 2057 2

    However, when I look at the process list, I see:
    root: ps -axdHO lwp | grep vmx0
    35771 100328 0 S+ 0:00.00 | `-- grep vmx0
    12 100114 - WL 0:00.04 - [intr/irq266: vmx0:t]
    12 100115 - WL 0:00.00 - [intr/irq267: vmx0:t]
    12 100116 - WL 0:00.00 - [intr/irq268: vmx0:t]
    12 100117 - WL 0:00.00 - [intr/irq269: vmx0:t]
    12 100118 - WL 0:00.09 - [intr/irq270: vmx0:r]
    12 100119 - WL 0:00.02 - [intr/irq271: vmx0:r]
    12 100120 - WL 0:00.11 - [intr/irq272: vmx0:r]
    12 100121 - WL 0:00.10 - [intr/irq273: vmx0:r]

    Which if I understand correctly (that vmx0:t refers to a TX queue), does indicate there are 4 of each queue.
    So perhaps I am misunderstanding how to actually see how many queues there are?
    In all the examples I have seen, all TX and RX queues are shown in vmstat -i

    Cheers



  • add this to /boot/loader.conf.local and reboot

    # allow multi queue support on vmx
    hw.pci.honor_msi_blacklist="0"
    

Log in to reply