When changing from "E1000" to "VMXNET3" I can't access the GUI (and more)?



  • Hi!

    My first post here - yay!
    Hello guys and girls!

    Now this is going to be quite a long post and hopefully the first of many but I wanted to give background info about my setup:
    Recently bought an Intel NUC D54250WYK and paired it with 2* Startech USB 3.0 to Gigabit Ethernet adapters for use with pfSense.
    I've read a post here - http://www.virtuallyghetto.com/2016/03/functional-usb-3-0-ethernet-adapter-nic-driver-for-esxi-5-5-6-0.html that the USB-adapters used a driver based on ASIX AX88179 and thanks to William Lam (the blog/link I've posted) and VMware's Songtao Zheng who helped him make it I integrated this driver in my ESXi ISO using "ESXi-Customizer-PS".

    I'm running ESXi 6.0.0 (Build 3825889), my NUC D54250WYK BIOS is 0041 (the latest one), I've changed the USB options (xHCI Mode) for the NUC from "Auto" (don't remember what it said exactly) to "Enabled". My pfSense build is 2.3-RELEASE which I've updated to 2.3.1 when I have entered the webinterface. I also downloaded "Open-VM-Tools" from pfSense as a package.

    The USB-interfaces work (YAY!) and I've had A LOT of problems with accessing the webinterface but what I think solved it was that I simply connected my ethernet cable going to my ethernet jack in my wall (using a fibre setup so I don't have any modem etc., just an ethernet jack in my wall directly which connects to my router) to my "WAN" interface during setup and the "LAN" interface to well, my switch.

    So my setup is as follows:
    Ethernet jack in my wall ("Internet") > WAN-interface (1* USB to Ethernet) > Intel NUC
    Devices (computer, NAS, soon to connect Ubiquiti AP) > Zyxel 8-port smart / managed switch w. PoE > LAN-interface (1* USB to Ethernet) > Intel NUC
    ESXi management interface (the onboard Intel NIC which I think is Intel i218v) > Zyxel switch

    WAN: DHCP
    LAN: <chosen private="" ip-range="">Now everything works as I said which I'm very pleased with since its been a hassle (at least for me and with these USB to Ethernet adapters) but if I make changes for the pfSense VM in ESXi that the network interfaces should be "VMXNET3" instead of "E1000" the following happens:

    • I can't access the webinterface; however sometimes (really bad description I know) I can ping the LAN-interface of pfSense
    • The DHCP-server seems to die so I don't get an IP and need to configure a static IP-address and even then I can only access ESXi and my switch, not the webinterface
    • WAN does not seem to get an IP via DHCP
      This happens even if I reinstall pfSense and from the beginning chose to have "VMXNET3" instead of "E1000" or if I start out with E1000 and then switch to VMXNET3.

    I have correctly re-assigned the interfaces (E1000=vusb0, vusb1 and VMXNET3=vmxn0, vmxn1) [unsure exactly what pfSense abrev. the adapters as when in VMXNET3] after switching from E1000 to VMXNET3 but it still won't work. A fresh install and chosing E1000 works fine though.

    What I didn't do however is to check the virtual switch settings in ESXi after changing the adapters!
    Might it be that since the adapters go from "vusb0" and "vusb1" I need to add "vmxn0" and vmxn1" to the virtual switches "WAN" and "LAN" or should that automatically just change in the virtual switches in ESXi?

    The thing is that my internet connection is 250/100Mbit and with the adapters as E1000 I still get really good speed (255,16 / 114,58). Should I even bother to change?
    And if so what might I have overlooked that "breaks" my pfSense installation? Will troubleshoot this especially with the re-assignment of the interfaces internally in ESXi later and give updates but wanted to post this first.

    Thank you very much for your time reading this even if you don't answer and hopefully I can give back to the community since I've made a write up of all the problems affecting me making an Intel NUC D54250WYK work together with 2* Startech USB 3.0 to Gigabit Ethernet (USB31000SW) and pfSense 2.3 :) Might be problems I've made for myself but hopefully it might benefit someone.

    Best Regards - TheSwede86</chosen>


  • LAYER 8 Global Moderator

    "ESXi that the network interfaces should be "VMXNET3" instead of "E1000" the following happens:"

    Says who… There is nothing saying you have to vmxnet3.. I run e1000 because vmx3net and cdp don't play nice keep reporting duplex mismatch.

    I am confused where you think you need to run vmx3??  That being said they should work.. Not sure what your switching.. You would add a new nic as vmx3net and then use that in pfsense as whatever it is you want to use it for wan, lan, opt connected to whatever vswitch you want it to connect to..



  • @johnpoz:

    "ESXi that the network interfaces should be "VMXNET3" instead of "E1000" the following happens:"

    Says who… There is nothing saying you have to vmxnet3.. I run e1000 because vmx3net and cdp don't play nice keep reporting duplex mismatch.

    I am confused where you think you need to run vmx3??  That being said they should work.. Not sure what your switching.. You would add a new nic as vmx3net and then use that in pfsense as whatever it is you want to use it for wan, lan, opt connected to whatever vswitch you want it to connect to..

    Thanks for your reply.
    I've just read these articles:
    VMware KB article - https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1001805
    VMXNET3 vs E1000E and E1000 - http://rickardnobel.se/vmxnet3-vs-e1000e-and-e1000-part-2/
    VMware vSphere 5.5 Virtual Network Adapter Performance - http://longwhiteclouds.com/2014/08/01/vmware-vsphere-5-5-virtual-network-adapter-performance/

    I don't have to run vmx3 but plenty of sources seem to say that VMX3NET is the better choice. Now as you pointed out there seem to be some drawbacks and since I'm new to pfSense and ESXi in general I did not know that :) I just based my assumption on whats "out there".

    Anyway I tried shutting down pfSense and then changing "WAN" and "LAN" from E1000 to VMX3NET, I get to assign the new interfaces and then my WAN-link just sits there not getting an IP from my ISP (DHCP) and likewise I am unable to either ping OR connecting to the webinterface. My local PC does not get an IP from pfSense DHCP-server either on the LAN-side.
    If I change the network adapters back to E1000 and then revert back to an earlier snapshot before changing the network adapters then it works great.

    I found this comment (by "Gareth Dawson") on a blog post - http://www.v-front.de/2015/01/pfsense-22-was-released-how-to-install.html?showComment=1435064227146#c5858289543876959124 which lead me to believe that if I would enable "Disable hardware checksum offload" (i.e. disabling it) in System > Advanced > Networking it might solve it! But I later found this post - https://redmine.pfsense.org/issues/5321 which seems to indicate that disabling hardware checksum offload does not affect  tx/rx checksumming for IPv6. Now I have pfSense setup to use both IPv4 and IPv6 so hey, I tried disabling hardware checksum offload and then changing to VMX3NET again.

    Now what I actually saw which might explain all this is the following:

    "Network interface mismatch – Running interface assignment option.
    vmx0: link state changed to UP
    vmx1: link state changed to UP

    Valid interfaces are:

    vmx0 <mac-address>(down) VMware VMXNET3 Ethernet Adapter
    vmx1 <mac-address>(down) VMware VMXNET3 Ethernet Adapter"

    So initially the interfaces are changed to "UP" but that don't seem to take effect since they are still "down" just a little bit later on?
    Might this be the cause of the problem?

    (This did not solve it either by disabling hardware checksum offload, get the same problems)</mac-address></mac-address>


  • LAYER 8 Global Moderator

    "not getting an IP from my ISP (DHCP)"

    So your nic changed macs, if your on cable modem this normally means you have to reboot that device to clear its cache.

    Yes the vmx3net is suppose to have better performance..  But there have been many a test done here, and none of them show any significant difference.  So yeah maybe when your running on some enterprise hardware with lots and lots of Vms on the host, etc. then you can see a difference between running e1000 and vmx3 on all your vms.

    But in a small setup with a couple of vms its not going to matter..



  • @johnpoz:

    "not getting an IP from my ISP (DHCP)"

    So your nic changed macs, if your on cable modem this normally means you have to reboot that device to clear its cache.

    Yes the vmx3net is suppose to have better performance..  But there have been many a test done here, and none of them show any significant difference.  So yeah maybe when your running on some enterprise hardware with lots and lots of Vms on the host, etc. then you can see a difference between running e1000 and vmx3 on all your vms.

    But in a small setup with a couple of vms its not going to matter..

    As I wrote earlier I'm not on a modem, I have an ethernet jack in my wall connecting it directly to my router (pfSense) and even though there might be a delay getting a new IP from my ISP etc. the "LAN"-side should still work right? After changing from E1000 to VMXNET3 all access and connectivity except for using ESXi to interface with pfSense just dies.

    Anyway I'm very happy just to have this up and running and to start exploring the world of pfSense and my next step will be to get my Ubiquiti AP running :) I'll leave the network adapters as E1000 since they give me as good a performance as my earlier router (Ubiquiti EdgeRouter PoE).
    Saw a project where they were making a pfSense package for Ubiquiti's "UniFi"-app but that sadly seems to be having dependency problems due to the fact that Ubiquiti updates it UniFi-app quite often and also related to the new verison of pfSense 2.3. But I'll keep an eye on it and am grateful for that guys work - https://github.com/gozoinks/unifi-pfsense.

    Thank you very much for your assistance and time, have a nice sunday :)


  • LAYER 8 Global Moderator

    Dude I am not sure what your doing.. But pfsense works out of the box with vmx3 on esxi.. Your not trying to install the native tools are you?



  • @johnpoz:

    Dude I am not sure what your doing.. But pfsense works out of the box with vmx3 on esxi.. Your not trying to install the native tools are you?

    No I am not trying to install the native tools.
    As I wrote earlier I installed the "Open-VM-Tools" via pfSense.

    What I was trying to do was to simply change the two network adapters assigned to my pfSense VM from E1000 to VMXNET3 but that got me in a world of trouble.
    Now since I'm using 2* USB to Gigabit Ethernet adapters with the drivers "injected" into an ESXi image that might explain why I am running into weird problems.
    These NIC:s are "WAN" and "LAN" on my NUC, with the NUCs own integrated NIC is only used for management of ESXi.

    If you want me to clarify something then please go ahead and I'll supply you with the info if you tell me what you need.
    I might have missed something totally obvious since it's my first time installing pfSense and VMware ESXi so I am not an expert by any means. But since E1000 seems to give perfectly reasonable performance then I'll just stick with that for now.


  • LAYER 8 Global Moderator

    Yeah maybe the vmx3 doesn't work with your physical hardware..


Log in to reply