Configuration of assigned interfaces is deployed to unassigned ones
-
Hey everyone,
I've discovered something I believe is a bug, but I'd like to consult it first for possible confirmation or help with additional troubleshooting.Background:
We are running pfSense virtualized on VMware vSphere platform with 10 vmxnet3 NICs (vmx0-vmx9), hardware version 13, using it as a router for multiple networks. Last week, we moved few of the networks to different router, so I've removed configuration for those networks on pfSense and unassigned now disconnected interfaces. This is where strange things started to happen, so to say.After reboot, routing was completely mismatched. Routes for still untouched and connected networks had wrong devices (netif) configured in routing:
Interfaces vmx0, vmx3, vmx4, vmx7 and vmx6 are those which were left untouched, I've only removed configuration and unassigned vmx1, vmx2, vmx5, vmx8 and vmx9.
As you can see, routing just doesn't correspond with interface settings. Situation was same in GUI, I've also triple-checked configuration and restarted the machine multiple times, nothing helped.At this point, I've tried to assign the interfaces (thinking that maybe their unassigned status is the problem) and leave them disabled with no configuration whatsoever. This helped until I did another reboot.
Routing situation remained the same, but it's cause was revealed:
Here you can see that configuration of vmx4 was deployed to vmx4 and vmx5, which has no configuration and is disabled. Same for vmx3 etc.Manually re-saving all interfaces helped only until reboot. Adding random ip addresses to OPT4 - OPT8 also didn't help.
I've discovered that the only thing that helps and persists through reboots is enabling those interfaces, even without configuration.I can repeat the same issue on fresh installation with 10 NICs. However, if I only add 4 NICs (2 assigned and configured, 2 unassigned), everything is working as expected.
Any thoughts?
EDIT:
I forgot to add one more important screenshot of this behavior. It's ifconfig command readout:
Here you can see that vmx2 and vmx6 has duplicate configuration, but vmx2 is not enabled in GUI, has ip config set to "none" and has status "no carrier", because there isn't a network connected to it.
Same goes for vmx5 and vmx4. -
This sort of thing happens all the time. The order of NICs may not necessarily be what you imagine they should be. For example, just last week there was a fellow who had a mainboard with 2 NICs, em0 and em1. He added an Intel quad-port card and was surprised to find out that everything stopped working. That's because FreeBSD enumerated the NICs differently than he expected. The new quad-port card took over em0-em3, and his old onboard NICs were punted to em4 and em5 which borked his config.
-
I came across similar NIC ordering issue some time ago. When I added more than 4 vmxnet3 adapters to the machine, FreeBSD numbered interfaces differently in comparison to VM ethernet adapter order, that issue is described here. When I used E1000 interfaces, the problem disappeared. I had to use workaround script that renamed vmx interfaces in FreeBSD based on their mac address, so that interface order was the same on VM and FreeBSD.
This is however something different. I'm not touching interfaces on the VM, I'm just unassigning/disabling interfaces in pfSense GUI under Interfaces -> Assignments. Though it definitely bears some similarity with aforementioned NIC order bug (meaning that everything is fine with only 4 interfaces on the machine).
