TX queues on VMX interface

Josef

Hi,
Hoping someone can help.
I am using a virtual pfsense instance on ESX 6.7 with 8 vCPU cores, and I am using vmxnet3 interfaces.
I only have 2 interfaces, and I want to use multi-queue to better utilize all CPU cores.
I cannot get more than 1 TX queue to be enabled. No matter what I do, there is always only 1.
I can get multiple RX queues, but only 1 TX queue.

In the FreeBSD vmx driver manual, it says to set the following parameters in the /boot/loader.conf file, which I have done:
hw.pci.honor_msi_blacklist="0"
hw.vmx.txnqueue="4"
hw.vmx.rxnqueue="4"

What ever I set for rxnqueue is honoured . But txnqueue seems to be ignored.
Likewise, if I remove the two settings, leaving just the msi_blacklist line, the defaults (for rx) apply - which should be 8 for both.
But again, I see 8 rx queues and still only 1 tx queue.

root: vmstat -i | grep vmx0
irq266: vmx0:tq0 11663 13
irq270: vmx0:rq0 5302 6
irq271: vmx0:rq1 1065 1
irq272: vmx0:rq2 4444 5
irq273: vmx0:rq3 4821 5

Thanks!!

stephenw10

Hmm, I would expect it to use them there if they are available.

Check the boot log perhaps.

You have no other loader lines limiting that?

Steve

Josef

Thanks.
If I boot with verbose, I see:
/root: cat /var/log/dmesg.boot | grep vmx0
vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x3000-0x300f mem 0xfe103000-0xfe103fff,0xfe102000-0xfe102fff,0xfe100000-0xfe101fff at device 0.0 on pci4
vmx0: attempting to allocate 9 MSI-X vectors (25 supported)
vmx0: using IRQs 266-274 for MSI-X
vmx0: bpf attached
vmx0: Ethernet address: 00:50:56:b4:54:ae
random: harvesting attach, 8 bytes (4 bits) from vmx0

So its correctly using MSI-X and its only trying to allocate 9 vectors - which is what I expect.
But vmstat -i still shows only 1 tx queue.
vmstat -i | grep vmx0
irq266: vmx0:tq0 5089 5
irq270: vmx0:rq0 3519 3
irq271: vmx0:rq1 394 0
irq272: vmx0:rq2 2618 3
irq273: vmx0:rq3 2057 2

However, when I look at the process list, I see:
root: ps -axdHO lwp | grep vmx0
35771 100328 0 S+ 0:00.00 | `-- grep vmx0
12 100114 - WL 0:00.04 - [intr/irq266: vmx0:t]
12 100115 - WL 0:00.00 - [intr/irq267: vmx0:t]
12 100116 - WL 0:00.00 - [intr/irq268: vmx0:t]
12 100117 - WL 0:00.00 - [intr/irq269: vmx0:t]
12 100118 - WL 0:00.09 - [intr/irq270: vmx0:r]
12 100119 - WL 0:00.02 - [intr/irq271: vmx0:r]
12 100120 - WL 0:00.11 - [intr/irq272: vmx0:r]
12 100121 - WL 0:00.10 - [intr/irq273: vmx0:r]

Which if I understand correctly (that vmx0:t refers to a TX queue), does indicate there are 4 of each queue.
So perhaps I am misunderstanding how to actually see how many queues there are?
In all the examples I have seen, all TX and RX queues are shown in vmstat -i

Cheers

chrcoluk

add this to /boot/loader.conf.local and reboot

# allow multi queue support on vmx
hw.pci.honor_msi_blacklist="0"