ESXi 6.1 - E1000 vs VMXNET3

Hello,

i'm aware that there were some issues with VMXNET3 adapters in the past.

Because we need more throughput we're thinking of switching or boxes from E1000 to VMXNET3 soon.

All boxes are running on the latest 2.3.1_1 release.

Are there currently any known issues or limitations with that from a pfSense point-of-view?

Thanks in advance

heper

i use them on a couple of installs. I'm not aware that there are downsides to using vmxnet adaptors.

cmb

vmxnet is preferred. Either/or is fine, but vmxnet will perform better.

johnpoz

one thing I can tell you as downside is if using the cdp, lldp via ladvd package that your going to see duplex mismatch in your logs. From what I recall using the native vmx3 drivers in 10.1 speed is only shown as autoselect vs 10gig and this seems to cause a issue with cdp, etc. When I ran vmx3 my switch log was flooded with duplex mismatch errors.

I am not concerned with the slight performance increase you can see with vmx3 so I just went back to e1000 that shows no issues with the duplex mismatch errors.

agomezp

i have the same question.

i have two pfsense 2.3.1 (pfsense1, pfsense2) and windows server (vmxnet3 10GB), install in all iperf3

pfsense1 have E1000 adapter (1000baseT <full-duplex>):

em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:50:56:b9:25:35
        inet6 fe80::250:56ff:feb9:2535%em0 prefixlen 64 scopeid 0x1
        inet XXXXXXX netmask 0xfffffc00 broadcast XXXXXXX
        nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast>

pfsense2 have VMXNET3 adapter, with Open-VM-Tools :

vmx0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=60009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,rxcsum_ipv6,txcsum_ipv6>ether 00:50:56:ac:0a:1d
        inet6 fe80::250:56ff:feac:a1d%vmx0 prefixlen 64 scopeid 0xa
        inet XXXXXXXX netmask 0xfffffc00 broadcast XXXXXX
        nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect
        status: active</performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,simplex,multicast>

The performance test :

from pfsense1 to WinServer

[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  16.5 GBytes   282 MBytes/sec    0             sender
[  4]   0.00-60.00  sec  16.5 GBytes   282 MBytes/sec                  receiver

from pfsense2 to WinServer

[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  22.3 GBytes   381 MBytes/sec    0             sender
[  4]   0.00-60.00  sec  22.3 GBytes   381 MBytes/sec                  receiver

the performance increase this vmx adapter.

i have a question is use the official Vmtools the performance increase?.

i found this article (http://www.v-front.de/2015/01/pfsense-22-was-released-how-to-install.html) for install the official vmware tools and use vmx3f0 but have a warning:

Update 2016-05-04: If you are using pfSense 2.3 (or newer) then please do not follow this guide, or it will break your system! Use the pfSense Package Manager to install the open-vm-tools package instead!

em
vmx - using Open-VM-Tools and VMXNET3 adapter
vmx3f - using official Vmware Tools and VMXNET3 adapter</full-duplex>

Thank you all for the input.

Just switched the first box from E1000 to VMXNET3 and i can report it all works great so far :-)

Unlike some people said to do a fresh install, i did the "dirty" way of removing the old NICs and re-adding them.

I powered off the pfsense VM, removed the old E1000 NICs and re-add the NICs with VMXNET3 afterwards.
After booting the box again, Ihad to re-assign the interfaces via the shell. All rules were present again.

Thanks

cmb

Yeah no need to reinstall, just adding new NICs then re-assigning suffices.

agomezp

dominic1134 in your pfsense, the network adapter is vmx o vmx3f ?

vmx - using Open-VM-Tools and VMXNET3 adapter
vmx3f - using official Vmware Tools and VMXNET3 adapter

I have a question on the difference between the two one is with OpenTools and the other with the vmware tools, but not if you have direncias in performance

@agomezp:

dominic1134 in your pfsense, the network adapter is vmx o vmx3f ?

vmx - using Open-VM-Tools and VMXNET3 adapter
vmx3f - using official Vmware Tools and VMXNET3 adapter

I have a question on the difference between the two one is with OpenTools and the other with the vmware tools, but not if you have direncias in performance

We currently use vmx with Open-VM-Tools.

cmb

You don't want or need the official VMware tools driver, vmx is fine. vmx3f has a history of causing problems.

bbrendon

Any issues with traffic shaper on esxi6, 2.3, and vmxnet3 ? I'm doing a new setup and can't get the expected results. I'm trying to rule out possible problems, vmxnet3 being one.

pppfsense

I had issues with promiscuous mode (packages that use that mode) issues with vmware's driver, just before
pfsense (FreeBSD) included the driver.

I have not done any testing since then as this is my Internet router so I only need GB speeds and don't need
the 10GB vmxnet driver.

The way I put it is that, everything gets tested against E1000, but only a portion of all the possible/usual configs get tested against vmxnet or vmxnet3, right??

That's for the speed. As far as the cpu cycles, I can afford the 10-15% extra cpu.

None the less, I really wish I could trust the vmxnet driver to be not only as good as the E1000 (100% 'compatibility'), but better!

@dominic1134:

Hello,

i'm aware that there were some issues with VMXNET3 adapters in the past.

Because we need more throughput we're thinking of switching or boxes from E1000 to VMXNET3 soon.

All boxes are running on the latest 2.3.1_1 release.

Are there currently any known issues or limitations with that from a pfSense point-of-view?

Thanks in advance

cmb

vmxnet3 in 2.2.x and newer has proven very solid, I'm not aware of any problems with the built-in driver in FreeBSD 10.x versions and it's widely used.

pppfsense

Thank you for the comment.

I did believe that the built-in driver would be solid at some point and your post has motivated to go back and
convert all my E1000 to vmxnet3 adapters.

Hopefully it won't have the interface reassignment issues right after upgrade, that I ran into a while ago :->

@cmb:

vmxnet3 in 2.2.x and newer has proven very solid, I'm not aware of any problems with the built-in driver in FreeBSD 10.x versions and it's widely used.