VmWare ESXi : 4th vmx does not work

[alfredo]

Dear List,

Setting up 3 vmx network adapters works, setting up 3 vmx and one ethernet E1000 works, but when configuring a fourth vmx adaptor, all networking stops to function. Please see attached image; when changing the bottom E1000 to vmx3, everything fails.

Is this a bug from pfSense or vmWare?

[heper]

https://forum.pfsense.org/index.php?topic=109901.0

[alfredo]

So this is a bug with FreeBSD and nothing can be done to make it work?

Note: We're using vmWare ESXi 6 and not M$.

[heper]

no clue … it sounded familiar so i linked the post.

if i find the time i'll try this on a vm @ work in the next couple of days

[alfredo]

Yes; maybe true for all hypervisors. What do the pfSense creators have to say?

This is really kinda annoying since we deal with 10GB Ethernet speeds. E1000 is a big hindrance.

Thanks.

Alfredo

[cmb]

That has nothing to do with Hyper-V.

I'm sure it's working, it's that ESX shifts your NICs when you add beyond a certain number of vmxnet NICs. Network adapter 1 is no longer vmxnet0, etc. It changes how they're presented to the guest, so you have to reassign the NICs and figure out their order.

[alfredo]

Could you please explain in terms of this vmx file what to do:

Changing ethernet3.virtualDev = "e1000" to "vmxnet3" makes it fail.

.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "11"
vmci0.present = "TRUE"
floppy0.present = "FALSE"
numvcpus = "8"
memSize = "8192"
bios.bootRetry.delay = "10"
sched.cpu.units = "mhz"
sched.cpu.affinity = "all"
powerType.suspend = "soft"
tools.upgrade.policy = "manual"
scsi0.virtualDev = "lsisas1068"
scsi0.present = "TRUE"
sata0.present = "TRUE"
scsi0:0.deviceType = "scsi-hardDisk"
scsi0:0.fileName = "firewall0.vmdk"
sched.scsi0:0.shares = "normal"
scsi0:0.present = "TRUE"
ethernet0.virtualDev = "vmxnet3"
ethernet0.networkName = "External Network"
ethernet0.addressType = "generated"
ethernet0.wakeOnPcktRcv = "FALSE"
ethernet0.present = "TRUE"
ethernet1.virtualDev = "vmxnet3"
ethernet1.networkName = "Private Network"
ethernet1.addressType = "generated"
ethernet1.wakeOnPcktRcv = "FALSE"
ethernet1.present = "TRUE"
displayName = "firewall"
guestOS = "freebsd-64"
toolScripts.afterPowerOn = "TRUE"
toolScripts.afterResume = "TRUE"
toolScripts.beforeSuspend = "TRUE"
toolScripts.beforePowerOff = "TRUE"
tools.syncTime = "FALSE"
uuid.bios = "56 4d ba 5f c2 81 83 9f-6d c4 fd 17 bc 06 a8 36"
uuid.location = "56 4d ba 5f c2 81 83 9f-6d c4 fd 17 bc 06 a8 36"
vc.uuid = "52 3f 34 02 2c 55 a7 db-3b 2c 08 bc 55 ad 59 c2"
sched.cpu.min = "0"
sched.cpu.shares = "normal"
sched.mem.min = "0"
sched.mem.minSize = "0"
sched.mem.shares = "normal"
ethernet1.generatedAddress = "00:0c:29:06:a8:40"
vmci0.id = "-1140414410"
cleanShutdown = "FALSE"
ethernet2.virtualDev = "vmxnet3"
ethernet2.networkName = "Hosting Network"
ethernet2.addressType = "generated"
ethernet2.wakeOnPcktRcv = "false"
ethernet2.present = "TRUE"
config.readOnly = "FALSE"
ethernet2.generatedAddress = "00:0c:29:06:a8:4a"
ethernet0.generatedAddress = "00:0c:29:06:a8:36"
extendedConfigFile = "firewall.vmxf"
ethernet3.virtualDev = "e1000"
ethernet3.networkName = "Customer Network"
ethernet3.addressType = "generated"
ethernet3.wakeOnPcktRcv = "FALSE"
ethernet3.present = "TRUE"

tools.guest.desktop.autolock = "false"
nvram = "firewall.nvram"
pciBridge0.present = "true"
svga.present = "true"
pciBridge4.present = "true"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "true"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "true"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "true"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
hpet0.present = "true"
RemoteDisplay.maxConnections = "-1"
sched.cpu.latencySensitivity = "normal"
sched.scsi0:0.throughputCap = "off"
ethernet0.uptCompatibility = "TRUE"
ethernet1.uptCompatibility = "TRUE"
ethernet2.uptCompatibility = "TRUE"
ethernet3.uptCompatibility = "TRUE"
virtualHW.productCompatibility = "hosted"
sched.swap.derivedName = "/vmfs/volumes/572d1574-e07d8ac8-03f4-0cc47a8e1078/firewall/firewall-1fe91f28.vswp"
replay.supported = "false"
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
scsi0.pciSlotNumber = "160"
ethernet0.pciSlotNumber = "192"
ethernet1.pciSlotNumber = "224"
ethernet2.pciSlotNumber = "256"
ethernet3.pciSlotNumber = "34"
vmci0.pciSlotNumber = "32"
sata0.pciSlotNumber = "33"
scsi0.sasWWID = "50 05 05 6f c2 81 83 90"
ethernet0.generatedAddressOffset = "0"
ethernet1.generatedAddressOffset = "10"
ethernet2.generatedAddressOffset = "20"
ethernet3.generatedAddressOffset = "30"
monitor.phys_bits_used = "42"
vmotion.checkpointFBSize = "4194304"
vmotion.checkpointSVGAPrimarySize = "4194304"
softPowerOff = "FALSE"
tools.remindInstall = "FALSE"
migrate.hostLog = "./firewall-1fe91f28.hlog"
replay.filename = ""
scsi0:0.redo = ""
ethernet3.generatedAddress = "00:0c:29:06:a8:54"

[johnpoz]

you can have more than 4 vmx nics - its just sometimes the order gets moved around and you have to tell pfsense which interface is which again is all.

Match up your mac addresses of the interfaces in pfsense and assign them where they need to be for your different networks.

If lets say vmx0 use to be your wan, and now your wan is vmx1 then yeah nothing going to work - just assign them correctly in your pfsense console.

[alfredo]

Thanks johnpoz and colleagues. That worked out very well.  :)

It turns out that VMWare shifts the adaptors around when going from 3rd to 4th "vmxnet3" adapter. The fourth Adaptor becomes vmx0…

All the best,

Alfredo

[richtj99]

So is this a case where passthrough would work better?

[johnpoz]

No ;) i really can not think of use case where it makes sense to passthru unless esxi has no driver for nic and vm does and u dont have time to switch out nic that vmware supports

If u did that u should plan on changing to different nic or get with vmware for driver