Is there a limit to the number of VMXNET3 NICs I can use in VMware?

MilkmanX

Hello, I'm currently attempting to install using the latest version (pfSense-CE-2.3.3-RELEASE-amd64.iso) on a virtual machine in VMware vCloud.

For the VM I"m using the following settings:
Other Linux 64 bit
2 GB disk
512 MB RAM
5 x VMXNET 3 Network Adapters

I'm doing a quick install with all default settings and everything appears to go fine. However, following the install, I'm unable to ping any of the interfaces and DHCP, etc. doesn't appear to be working. The console is up but doesn't show any errors that I can see. I'm able to assign interfaces and assign IPs from the console without any errors but this doesn't resolve the issue either.

What's interesting is that if I only use 4 VMXNET NICs everything works perfectly. I can also get it to work with 5 E1000 NICs. This leads me to believe there's some kind of limit or some other issue I'm running in to?

Thanks!

jimp

There is no limit I know of BUT ESX does wonky things with the NICs after 4. They get interleaved in the probe order.

So you end up with:

1
5
2
3
4

or

1
5
2
6
3
7
4
8

So you have to be careful to line up the detected NICs, MAC addresses, interface assignments in pfSense and so on.

MilkmanX

Yes, I have seen it do strange things but I made sure to double check the order and confirmed the mac addresses matched the interfaces in the UI. As it stands now 5 VMX nics do not work but the identical set up with 5 e1000 nics do. For now I've just switched to e1000 so I can continue working but I do think there's something strange going on here.

A Former User

I'm currently running pfSense 2.3.4 under ESXi 6.5 using 6 vmxnet3 nics. No issues here.

fwcheck

As far as i know, the maximum number of adapters within a virtual esxi-machine is 10.
https://www.vmware.com/pdf/vsphere5/r55/vsphere-55-configuration-maximums.pdf

You can use 10 Adapters with no problems (ESXi 5.5/6.0/6.5).

Be aware that adapters might be reordered as mentioned above. As this is nasty you can either

add 10 Adapters at start, leave unused adapters disconnected.
use a different way (e.g. VLAN-Trunking) with one adapter

A Former User

@fwcheck:

use a different way (e.g. VLAN-Trunking) with one adapter

I was considering of going down that path of VLANing it. I didn't think I changed around my stuff that often but I've had to straighten out the nics 3-4 times now after about 4 months.

Anyone aware of anything in pfSense that cannot be used or is limited in some form when using VLANs vs dedicated nics? Looking specifically for pfSense items, I'm aware of the virtualization layer pieces like threaded interrupt requests, multi queues, etc.

I would hate to switch off of dedicated nics only to find something I may want to use and have to recreate what I have already.

k-bull

Hello,

I too am running into this problem. I'm running ESXi 6.5d. I've been trying to add a 5th and 6th NIC using vmxnet3. But there seem to be problem with the MAC's. PFsense has MAC's for the extra nics but when I check the ESXi, it has no MAC addresses. I have redone the VM a few times , with fresh installs and still the same problem. I believe there's a compatibility issue. Mostly likely the problem is with ESXi. Can someone please confirm this.

A Former User

I'm running 6.5 build 5310538 with a pfSense VM using 6 nics with no issues here (other than the reordering which is known).

cbarber2

similar problem here. esxi 5.5, pfs 2.3.4, all interfaces are vmxnet at this point.

I'm ok with 8, total disaster when I add the 9th. In my case something gets misconfigured and most or all of the interfaces stop talking. No pings, no nothing on the lan or other configured and known working subnets. But the basic interface list on the console screen is still the same. Remove the 9th nic, and it all goes back to normal. Really weird.

I have the ordering problems, etc, others have described too. took a long time to comprehend that the mac address was the only constant linking the pfs interface to the esxi virtual adapter.

I'll try the E1000 types. Why vmxnet? That's what the appliance is built with. Ironic that some other documentation I found about this appliance says the best choice is to use the E1000.

VLANs and trunking - as a last resort. These are all virtual subnets that will never leave the virtual environment, except possibly to traverse from one esxi host to another. Seems like overkill, more moving parts, more things to keep track of.

-Chris

cbarber2

follow-up: with 8 vmx nics, adding a ninth nic with an E1000 causes no problem.

zxvv

https://www.vmware.com/pdf/vsphere6/r65/vsphere-65-configuration-maximums.pdf

Page 10 says:

maximum Virtual NICs per virtual machine: 10 (Any combination of supported virtual NICs)

ZsZs

@jimp:

There is no limit I know of BUT ESX does wonky things with the NICs after 4. They get interleaved in the probe order.

So you end up with:

1
5
2
3
4

or

1
5
2
6
3
7
4
8

So you have to be careful to line up the detected NICs, MAC addresses, interface assignments in pfSense and so on.

According to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=198406#c9 this can be fixed by editing ethernetX.pciSlotNumber lines respectively the vmx file directly on the ESXi.host.
Up to 8 vmxnet3 interfaces they should look like:

ethernet0.pciSlotNumber = "160"
ethernet1.pciSlotNumber = "1184"
ethernet2.pciSlotNumber = "192"
ethernet3.pciSlotNumber = "1216"
ethernet4.pciSlotNumber = "224"
ethernet5.pciSlotNumber = "1248"
ethernet6.pciSlotNumber = "256"
ethernet7.pciSlotNumber = "1280"