ESXi vSwitch + quad port NIC odd behavior
-
Setup:
- ESXi 6.0 U2
- pfSense VM 2.3.1 (issue also occurs with 2.3.1 Update1)
- Asus Z10PA-U8 with 2 onboard Intel I210 NICs
- 1 NIC is assigned to the ESXi management network
- 2nd NIC is assigned to the pfSense VM via PCIe/VT-d passthrough
- Quad port Intel NIC PCIe I340-T4
- A couple of VMs running, 1 Ubuntu and the other Windows Server 2016, both have an E1000 NIC assigned
pfSense VM config:
- I210 NIC passthrough
- ESXi network adapter, type E1000
So what I'm trying to do is have the quad port NICs available on my LAN, eventually to be used in combination with VLANs. But I'm having some issues getting this to work. I've tried 2 methods, one being to just add the 4 NICs to the ESXi vSwitch and letting ESXi handle it and the other method being to pass them through directly to the pfSense VM and bridging everything.
Method 1 / vSwitch-assigned
When using this method, I'm able to access the pfSense web interface with every VM. The problem I have though is that I can only use 1 NIC of the quad port adapter. Whenever I plug in a second device, one of the 2 will lose its IP address. Also, the one that retains its IP address will be the "first" one on the quad port adapter (with the order being determined by the MAC address). F.e. NIC1 has MAC …a1 and NIC4 has MAC ...a4 then NIC1 will retain its IP address.
Method 2 / PCIe passthrough of the 4 NICs
With this method all 4 NICs are passed through to the pfSense VM directly. Then I followed the steps outlined here to set up the bridge. This is working so far, though only when using E1000 adapters, when using VMXNET3 the VMs suddenly don't have access anymore. Furthermore, from what I understood this isn't the preferred solution.
So now I'm wondering why method 1 doesn't work or more specifically, what I'm doing wrong. :-
-
method1 & 2 should technically be possible but are bad practice 99% of time (bridging = evil)
perhaps exi needs some additional settings to allow for bridging multiple adaptors? google (never attemted such a thing)
when using passthrough you shouldn't be able to specify the nic type …. the hypervisor isn't handling the device anymore and the guest OS has "full" access to it.
-
If you want your multiple nics to be used by multiple interfaces in pfsense why would you not just assign each physical nic to its own vswitch and then assign a vnic in pfsense to this vnic.
When you put more than 1 physical nic in the same vswitch then you would need to setup lag or loadbalancing in esxi to leverage those connections but you wouldn't be able to use them as a "bridge" You could put them all on the same vswitch with different port groups for each physical nic.
I really don't understand why you would use passthru. Unless for example the esxi didn't have a driver such a nic. And the vm OS did. I am fan of letting pfsense handle the physical nics and then assign vnics to the vms you want to go through those physical nics.
-
method1 & 2 should technically be possible but are bad practice 99% of time (bridging = evil)
perhaps exi needs some additional settings to allow for bridging multiple adaptors? google (never attemted such a thing)
when using passthrough you shouldn't be able to specify the nic type …. the hypervisor isn't handling the device anymore and the guest OS has "full" access to it.
Ah sorry, might not have made myself clear on that last part, but what I meant was that I can only get the VMs working if their vNIC is of the E1000 type :). Of course, for the physical NICs, I don't have to specify a type.
Furthermore, indeed, you're correct in that ESXi would need perhaps additional settings to support this. Other sources also seem to be mentioning it to be a bad idea to try anyway.
If you want your multiple nics to be used by multiple interfaces in pfsense why would you not just assign each physical nic to its own vswitch and then assign a vnic in pfsense to this vnic.
When you put more than 1 physical nic in the same vswitch then you would need to setup lag or loadbalancing in esxi to leverage those connections but you wouldn't be able to use them as a "bridge" You could put them all on the same vswitch with different port groups for each physical nic.
I really don't understand why you would use passthru. Unless for example the esxi didn't have a driver such a nic. And the vm OS did. I am fan of letting pfsense handle the physical nics and then assign vnics to the vms you want to go through those physical nics.
Well, at first I was going to use the extra physical NICs just to expand my LAN. Only afterwards I was going to play around with VLANs to build the home network I want. But it looks like it's not a good idea to use them as LAN extension anyway and instead I should get a real switch if necessary. I looked around the forum for quite some time during the analysis of my issue, but looks like I didn't dig deep enough as I found some topics where others (including you ;)) also hinted at just buying a real switch :)
-
"I should get a real switch if necessary"
if you need ports then yes a switch is what you want, I would highly suggest a smart/managed switch that does vlans. You can smart that does vlans for very cheap if need be. I picked up one for my av cabinet the other day 8 port get from netgear that does vlans for <$40
nics on a router should be used for networks/vlans they are not switch ports.
As to using only e1000, that makes no sense since pfsense since it went to freebsd 10.x has native support for vmx3.
-
About the E1000 adapters, I actually meant the adapter type of the extra VMs (one Ubuntu VM and the other a Windows Server 2016 VM). If those VMs have the VMX3 adapter type, then I can't get them on the LAN when using method 2. If I use E1000 as adapter type, then it's working fine, so for now I'm using E1000 on my additional VMs till I can start setting up a more definite home network :). Might be interesting to find out why this happens, but for now I don't feel the need to investigate :P
-
if your vm does not have drivers then you would need to install it. Window and ubuntu you would install either the native tools from vmware for the vmx3 driver or ubuntu you could use the openvmtools package.
-
I had this exact same problem yesterday with my ESXi 5,5u3 and my pfSense 2.3.1 VM's. It seem like when adding an additional vmxnet3 NIC to my pfSense VM the MAC Addresses get shifted around on all the vmxnet3 NIC's on VM. It took me a while to figure it out but you can seen it in the vSphere client going into the VM settings and clicking on the vmxnet3 NIC. You can see the MAC that is assigned to the NIC does not match what is showing in pfSense but it shows up on a different NIC in the VM. After I changed all the NIC to E1000 every worked as expected. Seems like a bug in the vmxnet3 driver. Maybe a new vmxnet3 driver was added to 2.3 because I didn't have this issue in 2.2.
-
he wasn't having issues with pfsense.. Looks more like he was having issues with other VMs that did not have drivers at all.
"I actually meant the adapter type of the extra VMs (one Ubuntu VM and the other a Windows Server 2016 VM)."
-
Not quite, I have the appropriate VMware tools installed in both VMs and when connected with a VMXNET3 adapter while using the first method, everything is working fine in both VMs (so no driver issue). Only when I use the second method and have VMXNET3 adapters assigned to my VMs do I see the issue of not being able to access my LAN from those VMs. Switching to E1000 adapters fixes that specific issue. You are correct in stating that my issue is not with pfSense though, it's something with ESXi which I don't understand yet. But perhaps it does have something to do with the MAC addresses, I'll have to look into that.
-
huh?? If you pass thru the nic to the VM.. The vm would use the driver for the actual physical nic type, it wouldn't use a vmx driver.. The only time you would use vmx driver would be if the esxi is presenting a nic to your vm and not passthru to it..
-
Maybe we need a recap on this :D
Both of my test VMs (Ubuntu and WS2016) are only connected to the vSwitch via vNICs, be it E1000 or VMXNET3, no physical NICs at all. The difference between method 1 and method 2 is where the 4 physical NICs are connected to. With method 1 they are connected to the vSwitch which is then connected to the LAN interface of my pfSense VM. In that case there is no problem at all for the VMs to connect to the LAN. Method 2 has all 4 physical NICs passed through to the pfSense VM and are bridged together (total of 5 NICs, 4 physical + the vNIC which is connected to the LAN vSwitch).
With the latter method I notice a difference for the Ubuntu and WS2016 VMs when using E1000 adapters or VMXNET3 adapters. Using VMXNET3 adapters to connect these VMs to the vSwitch LAN, I'm unable to connect to the LAN, they don't get an IP address. When using E1000 adapters, it works like a charm.
Sorry if I made this confusing :)
-
"all 4 physical NICs passed through to the pfSense VM and are bridged together "
WTF would you do that for??? So in the setup where you pass through nics to pfsense and bridge them into 1 mess. And then you have another vnic that is connected to what vswitch?? And what vswitch is the other vms on??
Why would you bridge 4 physical nics that you passthru to pfsense.. What is esxi connected to then for it to get to the network, say its vmkern? And how do you have the vswitch your other vms are connected too?
What do you expect to do with 4 physical nics all connected to the same vswitch, and what are they connected to in the real world - did you setup a lagg/etherchannel? You have esxi doing failover/load balancing??
How many networks do you have in the physical world that are either native untagged or vlans with tags? Let me post up my setup as example to talk through this.
So vmkern in is same vlan as as lan, I have 2 physical nics in esxi that connect to switch that are in same untagged vlan. I broke vmkern out to its own nic because I have noticed when you put the vmkern on the same vswitch and same nic as another port group the performance to and from the datastore when moving files takes a hit.
Now you see pfsense vnic in the lan switch, and I have some vms on this same network segment they can talk to anything on the physical network that is in this same vlan this is untagged. I then have my wlan vswitch and its connected to another physical switch, pfsense interface in this has an untagged network and then some tagged networks for my AP ssids, etc.. and some other physical networks that are in other vlans that talk to pfsense vnic through this phsyical nic in esxi, you will notice on that vswitch it has a 4095 setting to pass tag through. The port on the physical switch this is connected to is trunked and there is a native untagged vlan and then tagged vlans.
Then there is the wan vswitch, this physical nic connects direct to my cable modem and this is how pfsense gets a connect via that vnic to the public internet.
I then have my dmz vswitch that has no physical nic, but has pfsense vnic also connected to vswitch. There is nothing in the physical world that is on this network, so it has no need for physical nic to tie that vswitch to the physical world. All vms that need to talk to the internet or other networks be they virtual or physical get routed through pfsense.
You will find that is very common way to set it up, and and for the life of me can not figure out what your trying to do with 4 nics all on the same vswitch… Unless you had a SHIT load of vms and machine and needed 4 gig to your physical world that you would load balance across them?
So most of my vms have vmx3 vnics, pfsense currently is running e1000 vnics, but it runs vmx3 just fine as well. The only reason I have it running e1000 is that with the ladvd package to provide lldp and cdp to and from pfsense my switch keeps reporting duplex mismatches in the log because the vmx3 vnic doesn't report its speed and duplex correctly it just says autoselect. But using e1000 it reports speed and duplex as 1000 full. So switches log doesn't report any problems.
If have not seen a performance issue using the e1000 that that would justify either turning off the cdp stuff or living with the flood of noise to the log so I just run e1000 on pfsense.
edit: Just noticed my pfsense vm is still called pf22, should prob change that since its running 2.3.1 ;
-
The reasoning behind it was that I wrongfully assumed that those physical NICs could easily be used as if it was a switch. But now I understand they cannot. Eventually I will have a similar setup as yours, though I'm still in the process of ordering switches, wireless APs,… So I just wanted to try out some things already with the physical devices (an HTPC, Xbox,...) I'll be connecting to my LAN :)
-
It seems to be a common misconception that multiple port nics are little switches.. Not sure how we kill off this misconception but it really needs to die..
The other misconception is that bridging these interfaces turns them into switches.. The closest it would come to would be a hub, and a shitty one at that.. Bridging has some specific use cases where it makes sense to do so. Actual use case is when you change media type, say going to from a fiber connection to copper, or wifi to ethernet.
You can use multiple interfaces a lagg to loadbalance traffic through, again not optimal performance here.. If you need more than 1 gig for example you should use a 10ge interface ;)
