ESXi 5.5 ENT+ Multiple NICs on vSwitch issue



  • Up until now I've been running a machine with 6 NICs and 6 subnets on pfSense. I just built a lab box with 11 NICs which will allow me to eliminate two 8-port switches I'm using.

    I'm finding that when I put multiple NICs on a vSwitch and then build up pfSense it doesn't like that configuration. Its only happy with one physical NIC per vSwitch. Is it NIC teaming I need to use? Essentially I want pfSense to not know the difference between a virtual or a physical NIC so I can use a quad-card paired with a virtual switch and attach the VMs as well as four external machines all to the same subnet.

    Any advice is much appreciated. I really don't want to do LAN1, LAN2, LAN3 and LAN4 and give each copper NIC its own vSwitch and pfSense subnet.

    Thanks!


  • Banned

    Use Vlans…



  • @Supermule:

    Use Vlans…

    Having never set up vlans before that's a bit out of my wheelhouse. I hate to ask for such remedial advice but could someone please point me to a DIY or tutorial to this? Is it exactly like NIC teaming or load balancing a physical switch?


  • Banned

    Nope. You split your nics into virtual LAN'S and keep the traffic seperated that way.

    Then you can have 4096 virtual networks in one physical NIC.


  • Banned


  • LAYER 8 Global Moderator

    So you want to use your physical nics on your esxi to connect hosts - like the physical nics are just ports on the switch?  And then connect pfsense so that its 1 virtual nic is also connected to this vswitch with 4 physical ports all on say 192.168.0.0/24

    like this - attached.  Where this is only 1 network?  Why???  Why not just use a switch?  If you need more bandwidth to the VM through the vswitch then you would connect all 4 ports to your switch lagged/bonded/teamed/etc  and then each PC would be connected to port on this switch.

    Even if  you brokeout vswitches 1 for each nic and created 4 nics in VM, for them all to be on 1 network you would have to bridge the interfaces in pfsense.

    If you want them all on different networks you could create trunk to vswitch from VM and then create port groups each with on vlan tag and different networks.

    Hmmm - this seems like a really odd sort of thing to want to do to be honest.  And the reason for this is you want to save yourself using an actual physical switch?



  • Banned

    :D Exactly!



  • @johnpoz:

    So you want to use your physical nics on your esxi to connect hosts - like the physical nics are just ports on the switch?  And then connect pfsense so that its 1 virtual nic is also connected to this vswitch with 4 physical ports all on say 192.168.0.0/24

    like this - attached.  Where this is only 1 network?  Why???  Why not just use a switch?  If you need more bandwidth to the VM through the vswitch then you would connect all 4 ports to your switch lagged/bonded/teamed/etc  and then each PC would be connected to port on this switch.

    Even if  you brokeout vswitches 1 for each nic and created 4 nics in VM, for them all to be on 1 network you would have to bridge the interfaces in pfsense.

    If you want them all on different networks you could create trunk to vswitch from VM and then create port groups each with on vlan tag and different networks.

    Hmmm - this seems like a really odd sort of thing to want to do to be honest.  And the reason for this is you want to save yourself using an actual physical switch?

    Yes exactly.

    I have a quad-NIC that I'd like to attach to my LAN vSwitch and be able to connect VMs as well as up to 4 external devices to the NIC. The issue is it is not resolving. I just want multiple copper NICs on the same subnet/vSwitch/network.

    I have 11 NICs on this lab server and only 7 networks/subnets. I used to have only 7 NICs but realized I was using racked 8-port switches for two of the subnets (2 - LAN and 5 - wifi). I added a quad NIC giving me four extra physical interfaces. This would allow me to eliminate two 8-port switches (yes I could use vlans on the switches and combine the two switches into one).

    Goal is to be as light and minimal as possible in terms of power consumption and devices. Ideally I would be able to run this all out of one machine vs. the need for switches and other.

    1 - WAN (FiOS)
    2 - LAN (all of the ESXi VMs)
    3 - ESXi (management interface and WOL interface for the server)
    4 - Archive data (limited and throttled access)
    5 - wifi (external WAP connects here, public hotspot attached)
    6 - realtime data (working directories, profile z drives, etc)
    7 - web and FTP (as it says, web and FTP servers on this subnet)

    I'd like to add an extra wifi WAP to #5 and I'd like to add three external machines to the LAN #2 (a drobo, a raspberry Pi and a PC).

    All I want to be able to do is plug an external device into the NIC and have it resolve just as the internal VMs do on their virtual switches. How do I make the external physical NICs part of the vSwitch like they are internally?

    Sorry, I have reasons for wanting/needing to eliminate the use of switches. Hard to explain but this is a fairly mobile server and moves around quite a bit. being able to unplug and move one machine is a lot easier than having to carry switches with it.

    Thanks.



  • I only have ONE vNIC per subnet on the pfSense machine. I don't see how vLANs would be applicable.

    I think that there is a disconnect. #1 this is on a hypervisor and #2 pfsense has only the number of interfaces it needs; one virtual interface per subnet.

    I attach each subnet to a virtual switch in ESXi. From there I assign physical NICs on the ESXi host to each vswitch. My issue comes in when I assign more than one physical NIC to a virtual switch.

    All I want is:

    pfSense >> e1000 vNIC >> vSwitch >> 4x physical NICs.

    This way I can connect as many VMs as needed internally and hardwire/plug in up to four physical cables into the same subnet… without a switch.


  • LAYER 8 Global Moderator

    I don't think vmware accounts for such a setup.  You can clearly assign more than 1 physical nic to a vswitch.. But they would be used in teaming/failover/etc. sort of situation.  I don't think you can just use them like physical ports on a switch and connect specific hosts that are on the same network.

    Let me do a little digging and play with my setup, I am currently running esxi 5.5 as well and have multiple physical nics on the host I can play with.

    edit:  So a read of this
    http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf

    I really don't see how you can do what your after - the connection of physical nics to a vswitch is not meant to be like ports on a normal switch.  They are used for teaming or load balancing or failover.

    you might be able to setup a load balance and then use route based on ip hash, both source and destination IP..  But I think your going to run into problems.

    What you could do is create 4 vswitches, with each switch connected to a physical nic and then in pfsense with 4 vnics 1 each connected to each switch and then bridge these vnics in pfsense.. But don't think would be a great performer?


Log in to reply