PfSense 2.1 as ESXi 5.5 guest – interface names change when new vNIC is added



  • I'm evaluating pfSense as a potential vRouter, running as an ESXi 5.5 guest VM.  First-time installer/user.

    Installed pfSense 2.1 from ISO on a 4-vNIC VM, installed VMWare Tools using the workarounds provided elsewhere in the forums (for VMXNET3 support), and got a test rig up and running fine.

    However, I found I needed to add an additional VMXNET3 vNIC to the 4 I had configured the VM with.

    Shut down the VM, added the NIC, and powered it back on…and found I could no longer access the GUI nor route packets through the pfSense VM.

    Got on the console and found that the "vmx3f*" interface names referenced completely different MAC addresses than when I had originally installed it.

    My interface ID to MAC list is:

    | IF name | Original | MAC | MAC |
    | | Assignment | (from original install) | (after adding vNIC) |
    | |
    | vmx3f0 | WAN | 00:50:56:96:f1:d2 | 00:50:56:96:f1:d2 |
    | vmx3f1 | LAN | 00:50:56:96:dc:8a | 00:50:56:96:93:17 |
    | vmx3f2 | OPT1 | 00:50:56:96:3b:95 | 00:50:56:96:dc:8a |
    | vmx3f3 | OPT2 | 00:50:56:96:48:4f | 00:50:56:96:3b:95 |
    | vmx3f4 | | | 00:50:56:96:48:4f |

    The newly-added vNIC (Network adapter 5 according to ESXi) has MAC address 00:50:56:96:93:17.

    I tried reassigning the new IFs based on their MAC addresses from the original assignment, but got hopelessly tangled up and finally reverted to the 4-vNIC install, at which point I got my WAN/LAN interfaces back (but lost configs for the OPT interfaces, as I had configured VLANs on them originally).

    I searched the forums (and Google) for about an hour, but did not run across any other folks who posted regarding this issue; please point me to other people's experiences if they exist.

    Otherwise, I'd love to know what the "proper" practice would be to add a vNIC card to a pfSense VM.

    Thanks in advance,


    Troy



  • Usually, the vNICs are presented to the VM in the order you assigned them to the VM.

    So if you have port groups named LAN, WAN, OPT1, OPT2 and you attach them to pfSense in that order, they should be presented to it in that order - as vmx3f0=LAN, vmx3f1=WAN, vmx3f2=OPT1, etc.

    Hard to see how it could re-order them like that - with the new vNIC's MAC being assigned to LAN and the others being pushed down.

    You say you had VLANs assigned.  Is each vNIC on a separate vSwitch?  Maybe the rule above doesn't apply when the VM has more than one interface on a vSwitch.



  • @biggsy:

    Usually, the vNICs are presented to the VM in the order you assigned them to the VM.

    So if you have port groups named LAN, WAN, OPT1, OPT2 and you attach them to pfSense in that order, they should be presented to it in that order - as vmx3f0=LAN, vmx3f1=WAN, vmx3f2=OPT1, etc.

    So I would have thought as well…which is why it surprised me that the reordering did occur the way it did.

    Ran into a similar situation with a Ubuntu-based VM install due to udev ignoring vNIC MAC addresses when populating /etc/network/interfaces; that led me to looking at pfSense as a potential alternative.

    My FreeBSD-fu is weak, however...and I have not found yet any information regarding how the loader determines the order in which a device is detected; I'd take your information (above) as doctrine, but my experience is showing me something completely different.

    @biggsy:

    You say you had VLANs assigned.  Is each vNIC on a separate vSwitch?  Maybe the rule above doesn't apply when the VM has more than one interface on a vSwitch.

    My configuration layout :

    | vNIC ID | pfSense ID | ESXi NIC # | PortGrp | vSwitch# |
    | vmx3f0 | WAN | 1 | External | 3 |
    | vmx3f1 | LAN | 2 | IT-MGMT | 0 |
    | vmx3f2 | OPT1 | 3 | Internal | 0 |
    | vmx3f3 | OPT2 | 4 | INTRASITE | 2 |

    My VLANs were on vmx3f0 and vmx3f3; the target vSwitch for each vNIC is a "bubble network" with no uplinks to the backing pSwitch (to provide routing layer isolation)

    The 5th vNIC was added and attached to an additional portgroup ("External 2") on vSwitch0.

    And now, having read that, I'm curious to find out what happens if I create another no-uplink vSwitch/portgroup and add a vNIC to the pfSense VM tied to the new vSwitch (as opposed to linking it back to vSwitch0 where I already have two vNICs connected).

    Back in a second.

    […]

    Okay, back.

    Doesn't seem to matter whether the vNIC is on its' own vSwitch or on a vSwitch with several portgroups; the new vNIC's MAC address is still being detected by the kernel as vmx3f1, not vmx3f4…and the rest of the vNICs are renamed as before.

    Running dmesg, I think I'm beginning to see a method to this madness, as:

    Original dmesg output:

    | vNIC ID | Port | IRQ | Device |
    | vmx3f0 | 0x4000-0x400f | 18 | 0.0, pci3 |
    | vmx3f1 | 0x5000-0x500f | 19 | 0.0, pci11 |
    | vmx3f2 | 0x6000-0x600f | 16 | 0.0, pci19 |
    | vmx3f3 | 0x7000-0x700f | 17 | 0.0, pci27 |

    dmesg output after vNIC add:

    | vNIC ID | Port | IRQ | Device |
    | vmx3f0 | 0x4000-0x400f | 18 | 0.0, pci3 |
    | vmx3f1 | 0x8000-0x800f | 18 | 0.0, pci4 |
    | vmx3f2 | 0x5000-0x500f | 19 | 0.0, pci11 |
    | vmx3f3 | 0x6000-0x600f | 16 | 0.0, pci19 |
    | vmx3f4 | 0x7000-0x700f | 17 | 0.0, pci27 |

    So it would appear that FreeBSD is assigning the "vmx3f_n_" name at boot time based on the PCI bus #, and not on MAC address or other criteria.

    Since I suck in FreeBSD (and not much better in Linux!), does anyone know what I might be able to tweak to change the detection methodology?  None of the answers I've found so far has given me a clue; I know (roughly) where in the boot process the drivers are loaded, and I've now figured out how the names are assigned…but nothing telling me of a "udev" equivalent that could be used to "force" assignment of a name to an artificial identifier (such as vNIC MAC address).

    --
    Troy



  • So after a bit more research, it looks like ESXi is responsible for determining which PCI bus # a given device gets added to…and there seems to be no specific way to configure it from ESXi-land.

    FreeBSD (and pfSense) are behaving as designed, so no fault there.

    It looks like what I would have to do is:

    • Develop a manual script to create a mapping of interface names and roles (e.g. "external", "intrasite", etc., arbitrary definitions) to MAC addresses (similar to /etc/network/interfaces in Linux);

    • Run the script once I've got a configuration ready for production;

    • Develop an rc script that runs at each boot that a) checks to see that the mapping is as originally set; b) renames interfaces if their PCI bus order changes (and reapplies any other configurations as needed); and c) calls the "Assign Interfaces" function if all of the MACs have changed; and

    • Make all this scripting something that can be transitioned to other *BSD-based systems (for example, the next upgrade of pfSense).

    This, of course, in my Copious Spare TimeTM.

    So, not an issue with pfSense or FreeBSD per se…but one that I'll have to work out how to handle from within FreeBSD in order to ensure a consistent configuration template that I don't have to re-work for each potential instance.

    Biggsy: thank you for your input on the issue; you stimulated much research and learning.

    --
    Troy