ESXi vSwitch + quad port NIC odd behavior
-
"I should get a real switch if necessary"
if you need ports then yes a switch is what you want, I would highly suggest a smart/managed switch that does vlans. You can smart that does vlans for very cheap if need be. I picked up one for my av cabinet the other day 8 port get from netgear that does vlans for <$40
nics on a router should be used for networks/vlans they are not switch ports.
As to using only e1000, that makes no sense since pfsense since it went to freebsd 10.x has native support for vmx3.
-
About the E1000 adapters, I actually meant the adapter type of the extra VMs (one Ubuntu VM and the other a Windows Server 2016 VM). If those VMs have the VMX3 adapter type, then I can't get them on the LAN when using method 2. If I use E1000 as adapter type, then it's working fine, so for now I'm using E1000 on my additional VMs till I can start setting up a more definite home network :). Might be interesting to find out why this happens, but for now I don't feel the need to investigate :P
-
if your vm does not have drivers then you would need to install it. Window and ubuntu you would install either the native tools from vmware for the vmx3 driver or ubuntu you could use the openvmtools package.
-
I had this exact same problem yesterday with my ESXi 5,5u3 and my pfSense 2.3.1 VM's. It seem like when adding an additional vmxnet3 NIC to my pfSense VM the MAC Addresses get shifted around on all the vmxnet3 NIC's on VM. It took me a while to figure it out but you can seen it in the vSphere client going into the VM settings and clicking on the vmxnet3 NIC. You can see the MAC that is assigned to the NIC does not match what is showing in pfSense but it shows up on a different NIC in the VM. After I changed all the NIC to E1000 every worked as expected. Seems like a bug in the vmxnet3 driver. Maybe a new vmxnet3 driver was added to 2.3 because I didn't have this issue in 2.2.
-
he wasn't having issues with pfsense.. Looks more like he was having issues with other VMs that did not have drivers at all.
"I actually meant the adapter type of the extra VMs (one Ubuntu VM and the other a Windows Server 2016 VM)."
-
Not quite, I have the appropriate VMware tools installed in both VMs and when connected with a VMXNET3 adapter while using the first method, everything is working fine in both VMs (so no driver issue). Only when I use the second method and have VMXNET3 adapters assigned to my VMs do I see the issue of not being able to access my LAN from those VMs. Switching to E1000 adapters fixes that specific issue. You are correct in stating that my issue is not with pfSense though, it's something with ESXi which I don't understand yet. But perhaps it does have something to do with the MAC addresses, I'll have to look into that.
-
huh?? If you pass thru the nic to the VM.. The vm would use the driver for the actual physical nic type, it wouldn't use a vmx driver.. The only time you would use vmx driver would be if the esxi is presenting a nic to your vm and not passthru to it..
-
Maybe we need a recap on this :D
Both of my test VMs (Ubuntu and WS2016) are only connected to the vSwitch via vNICs, be it E1000 or VMXNET3, no physical NICs at all. The difference between method 1 and method 2 is where the 4 physical NICs are connected to. With method 1 they are connected to the vSwitch which is then connected to the LAN interface of my pfSense VM. In that case there is no problem at all for the VMs to connect to the LAN. Method 2 has all 4 physical NICs passed through to the pfSense VM and are bridged together (total of 5 NICs, 4 physical + the vNIC which is connected to the LAN vSwitch).
With the latter method I notice a difference for the Ubuntu and WS2016 VMs when using E1000 adapters or VMXNET3 adapters. Using VMXNET3 adapters to connect these VMs to the vSwitch LAN, I'm unable to connect to the LAN, they don't get an IP address. When using E1000 adapters, it works like a charm.
Sorry if I made this confusing :)
-
"all 4 physical NICs passed through to the pfSense VM and are bridged together "
WTF would you do that for??? So in the setup where you pass through nics to pfsense and bridge them into 1 mess. And then you have another vnic that is connected to what vswitch?? And what vswitch is the other vms on??
Why would you bridge 4 physical nics that you passthru to pfsense.. What is esxi connected to then for it to get to the network, say its vmkern? And how do you have the vswitch your other vms are connected too?
What do you expect to do with 4 physical nics all connected to the same vswitch, and what are they connected to in the real world - did you setup a lagg/etherchannel? You have esxi doing failover/load balancing??
How many networks do you have in the physical world that are either native untagged or vlans with tags? Let me post up my setup as example to talk through this.
So vmkern in is same vlan as as lan, I have 2 physical nics in esxi that connect to switch that are in same untagged vlan. I broke vmkern out to its own nic because I have noticed when you put the vmkern on the same vswitch and same nic as another port group the performance to and from the datastore when moving files takes a hit.
Now you see pfsense vnic in the lan switch, and I have some vms on this same network segment they can talk to anything on the physical network that is in this same vlan this is untagged. I then have my wlan vswitch and its connected to another physical switch, pfsense interface in this has an untagged network and then some tagged networks for my AP ssids, etc.. and some other physical networks that are in other vlans that talk to pfsense vnic through this phsyical nic in esxi, you will notice on that vswitch it has a 4095 setting to pass tag through. The port on the physical switch this is connected to is trunked and there is a native untagged vlan and then tagged vlans.
Then there is the wan vswitch, this physical nic connects direct to my cable modem and this is how pfsense gets a connect via that vnic to the public internet.
I then have my dmz vswitch that has no physical nic, but has pfsense vnic also connected to vswitch. There is nothing in the physical world that is on this network, so it has no need for physical nic to tie that vswitch to the physical world. All vms that need to talk to the internet or other networks be they virtual or physical get routed through pfsense.
You will find that is very common way to set it up, and and for the life of me can not figure out what your trying to do with 4 nics all on the same vswitch… Unless you had a SHIT load of vms and machine and needed 4 gig to your physical world that you would load balance across them?
So most of my vms have vmx3 vnics, pfsense currently is running e1000 vnics, but it runs vmx3 just fine as well. The only reason I have it running e1000 is that with the ladvd package to provide lldp and cdp to and from pfsense my switch keeps reporting duplex mismatches in the log because the vmx3 vnic doesn't report its speed and duplex correctly it just says autoselect. But using e1000 it reports speed and duplex as 1000 full. So switches log doesn't report any problems.
If have not seen a performance issue using the e1000 that that would justify either turning off the cdp stuff or living with the flood of noise to the log so I just run e1000 on pfsense.
edit: Just noticed my pfsense vm is still called pf22, should prob change that since its running 2.3.1 ;
-
The reasoning behind it was that I wrongfully assumed that those physical NICs could easily be used as if it was a switch. But now I understand they cannot. Eventually I will have a similar setup as yours, though I'm still in the process of ordering switches, wireless APs,… So I just wanted to try out some things already with the physical devices (an HTPC, Xbox,...) I'll be connecting to my LAN :)
-
It seems to be a common misconception that multiple port nics are little switches.. Not sure how we kill off this misconception but it really needs to die..
The other misconception is that bridging these interfaces turns them into switches.. The closest it would come to would be a hub, and a shitty one at that.. Bridging has some specific use cases where it makes sense to do so. Actual use case is when you change media type, say going to from a fiber connection to copper, or wifi to ethernet.
You can use multiple interfaces a lagg to loadbalance traffic through, again not optimal performance here.. If you need more than 1 gig for example you should use a 10ge interface ;)
