[How-To] Using VMXNET2/3 NICs in pfSense 2.0

AhnHEL

Doesnt seem to support ALTQ either so traffic shaping isnt possible. E1000 seems to be the tried and true method on pfSense.

pfSense.User.1138

So basicly using the VMXNET2 NIC, you'll get better throughput and lower cpu utilization at the expense of features.
If just pfSense/FreeBSD had a working driver for the VMXNET3 NIC :-(
Don't think the VMXNET2 NIC or driver will ever get those features, afaik VMware have stopped development of that NIC entirely.

pfSense.User.1138

It seems that the Open-VM-Tools package is currently broken.
As a workaround it is possible to use the regular vendor supplied VMware Tools following these instructions:

1. Install prerequisites in the VM console:
When running on a 64 bit installation, run:

pkg_add -r compat6x-amd64

When running on a 32 bit installation, run:

pkg_add -r compat6x-i386

(Note for 32 bit: not tested but this should be it)

2. Install Perl:

pkg_add -r perl

3. Link the newly installed libraries so the VMware Tools installer can find it:

ln -s /usr/local/lib/compat/libm.so.4 /lib
ln -s /usr/local/lib/compat/libc.so.6 /lib
ln -s /usr/local/lib/compat/libthr.so.2 /lib

4. Install VMware Tools:
Select the "Install/Upgrade VMware Tools" option on the VM in the vSphere Client.
Back in the VM console type:

cd /mnt
mkdir cdrom
mount –t cd9660 /dev/acd0 /mnt/cdrom
cd /tmp
tar –zxvf /mnt/cdrom/vmware-freebsd-tools.tar.gz
cd vmware-tools-distrib
./vmware-install.pl

(Perl script, this is why Perl is installed)
Accept all defaults (press enter 8 times)

5. To clean up:

cd ..
rm -r vmware-tools-distrib

All credits go to OnHEL.
Original post linked.

pfSense.User.1138

Since ESXi 5.0 it is now possible to use VMXNET3 NIC's in pfSense using the vendor surplied VM-Tools.
Unfortunately the VMXNET3 NIC also seems to be missing the VLAN tagging and ALTQ features on FreeBSD.
I know the VMXNET3 NIC does support VLAN tagging in other OS's, odds is it must be an issue with the FreeBSD driver.

photonman

@pfSense.User.1138:

Since ESXi 5.0 it is now possible to use VMXNET3 NIC's in pfSense using the vendor surplied VM-Tools.
Unfortunately the VMXNET3 NIC also seems to be missing the VLAN tagging and ALTQ features on FreeBSD.
I know the VMXNET3 NIC does support VLAN tagging in other OS's, odds is it must be an issue with the FreeBSD driver.

I am getting ready to build a ESXi 5 pfSense vm. If I do not need those missing features, is the VMXNET3 going to perform better then the E1000?

pfSense.User.1138

@photonman:

I am getting ready to build a ESXi 5 pfSense vm. If I do not need those missing features, is the VMXNET3 going to perform better then the E1000?

It sure is.

photonman

I downloaded the packages manually from ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8-stable/Latest/ but now I am stumped as to how to load them in pfsense?

[UPDATE]…I got it loaded... used

pkg_add -rv ftp://ftp.freebsd.org/pubFreeBSD/ports/i386/packages-8-stable/Latest/

and assigning the interfaces was a tricky thing because it gave the valid interfaces as interface name + MAC address so for example:

vmx3f000:12:34:56:78:90:bb
vmx3f100:12:34:56:78:90:hh

You have to realize the interface names are vmx3f0 and vmx3f1

THANK YOU so much for the great instructions as I could have never done this without…

Notice that these files exist in /boot/modules and not in /boot/kernel like vmxnet.ko so is that a problem???

vmmemctl.ko
vmblock.ko
vmxnet3.ko

but I get this with kldstat which is good but no vmhgfs which could be a change in ESXi 5 vmtools?

$ kldstat
Id Refs Address Size Name
1 10 0xc0400000 11796f4 kernel
2 1 0xc157a000 5684 vmblock.ko
3 1 0xc1580000 3404 vmmemctl.ko
4 1 0xc1584000 4988 vmxnet.ko
5 1 0xc1589000 8608 vmxnet3.ko

but then when I add the vmxnet 3 nics to the vm and reboot I get this:

$ ps ax|grep vmware
19319 ?? S 0:00.00 sh -c ps ax|grep vmware
19733 ?? S 0:00.00 grep vmware

which means the tools are not running?

and $ find /* |grep vmware-guestd

gives me nothing???

so I reinstalled the package from the shell and now I get this so perhaps vmware-guestd is now vmtoolsd and in VMware client it actually says VMware Tools: Running (Current)

$ ps ax|grep vmware
12758 ?? S 0:00.00 sh -c ps ax|grep vmware
12846 ?? R 0:00.00 grep vmware
15633 ?? S 0:00.11 /usr/local/lib/vmware-tools/sbin/vmtoolsd

but as soon as I reboot, the VMware says VMware Tools:Not Running (Current) and grep vmware does not show vmtoolsd?

I am hoping that I am way ahead of the pack and the current documentation cannot explain ESXi 5 vmtools on a pfSense 2.0 machine.

biggsy

and assigning the interfaces was a tricky thing because it gave the valid interfaces as interface name + MAC address so for example:

I honestly can't remember where I first came across this script. It may have been on this forum. It certainly helps in figuring out which MAC address belongs to which interface when assigning them in pfSense.

EDIT:
Info - http://www.virtuallyghetto.com/2011/05/how-to-query-for-macs-on-internal.html
Download - http://vghetto.svn.sourceforge.net/viewvc/vghetto/other/vswitchInfo.sh?view=log

Biggsy

wagebox

Hello everyone,

I have managed to get the vmxnet3 working under pfsense 2.0 runing on shiny new esxi 5.0 host (build 474610).
I have installed the native vmware tools using guide in this thread. vmxnet3 interfaces are working fine so far, but I have come across one weird issue. According to vmware configuration maximums, it is possible to assign up to 10 virtual nics per vm. However, as soon as I assign four or more vmxnet3 adapters to pfsense vm, network gets cut off. Link is up, I can ping hosts from pfsense, but it is not possible to ping or connect to pfsense, it is not accepting any incomming connections or routing traffic. I have tried to reset pfsense to factory defaults, but that did not help. If I remove just one vmxnet3 adapter, everything is back to normal.
This issue is not manifested when using e1000 adapters.
I am by no means freebsd guru so I cannot do in depth troubleshooting, but if someone is interested I can provide additional data when instructed.

In the meantime, back to e1000 adapters :)

pfSense.User.1138

@wagebox:

However, as soon as I assign four or more vmxnet3 adapters to pfsense vm, network gets cut off. Link is up, I can ping hosts from pfsense, but it is not possible to ping or connect to pfsense, it is not accepting any incomming connections or routing traffic. I have tried to reset pfsense to factory defaults, but that did not help. If I remove just one vmxnet3 adapter, everything is back to normal.

That is strange. I use six vmxnet3 adapters with no problem at all.

pfSense.User.1138

I've just updated from 2.0 Release to 2.0.1 Release and found the vendor supplied VMware Tools failed to start on boot after the upgrade.
But fortunately its a quick and easy fix.

On the console open the shell ( press 8 )
Then type:

vmware-config-tools.pl

When vmware-config-tools.pl is done running type:

exit

To leave the shell again.

VMware Tools status for the pfSense VM in your vSphere Client should now be "Running".
As an extra check, try reboot the VM to verify that VMware Tools is running when pfSense is back up.

zachsaw

Hi guys,

I've managed to install vendor supplied vmware tools in pfsense 2.0.1.
However, upon changing e1000 to vmxnet3, my WebGUI slows down to a halt. All other traffic runs at full speed except for the WebGUI.

I also notice however that if I run "Assign Interfaces" from shell and go through the prompts again, WebGUI will become responsive. At least until the next reboot.

This sounds as though the vmxnet3.ko driver must be reloaded after VMWare Tools memory manager has been loaded.

I guess alternatively, if I could just do the last part of "Assign Interfaces" where it reloads interfaces, it would work as well.

So my question is, where can I find the script that does "Assign Interfaces"?

Or better yet, does anyone know what it does to reload interfaces after writing to configuration file?

EDIT: Found it – /etc/rc.reload_interfaces

johnpoz

Ok I was using the vmxnet3 drivers without any issue – I thought!

But had not had need in a while to get onto work vpn.. Last night got a call, and I could not get in.. Spent this evening troubleshooting it, and nothing was jumping out at me..

Even enabled oakley.log on the box -- nothing jumped out why it was just freaking hanging.. Its a BS custom client from the mother company in DE.. Logs are useless in the thing, etc. Anyway I went back to the e1000 and bam connected right in.. No issues at all.

If I get a chance I will try and figure it out - maybe this weekend, but for me had to go back to the e1000 drivers.

casper4242

I just found out: the vmxnet3 driver does support jumbo frames, vlan tagging etc… however due to the
somewhat odd naming "vmx3f0" etc the pfsense interface name parser fails (it tries to seperate the driver
from the unit on the first number, which fails here). When these regex are fixed, the interfaces can be
used as vlan parents, and from what I could test quickly, they do work this way.

Here's an ifconfig of such a vmnet3 interface:
vmx3f0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>ether 00:50:56:8d:06:39
inet 192.168.166.1 netmask 0xffffff00 broadcast 192.168.166.255
inet6 fe80::250:56ff:fe8d:639%vmx3f0 prefixlen 64 scopeid 0x1
nd6 options=3 <performnud,accept_rtadv>media: Ethernet 10Gbase-T
status: active

Here's one as a VLAN sub-interface (just used with tcpdump so far):
vmx3f2_vlan1101: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=303 <rxcsum,txcsum,tso4,tso6>ether 00:50:56:8d:06:3b
inet6 fe80::250:56ff:fe8d:639%vmx3f2_vlan1101 prefixlen 64 scopeid 0xa
nd6 options=3 <performnud,accept_rtadv>media: Ethernet 10Gbase-T
status: active
vlan: 1101 parent interface: vmx3f2

And this is all on ESXi4 with ESXi5-vmware-tools installed. Looking good so far :)

Cheers,
Markus</performnud,accept_rtadv></rxcsum,txcsum,tso4,tso6></up,broadcast,running,simplex,multicast></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></up,broadcast,running,simplex,multicast>

hoostine

http://redmine.pfsense.org/issues/2010

there's a link to patches that seem to solve the issue for me.

note: i only modified globals.inc and interfaces.inc, they're located in /etc/inc, in case anyone is lazy like me. ;D

iFloris

Thanks for finding that patch for us hoostine, will give this a shot sometime next week.
Seeing that this patch is six months old makes me wonder if this code is already in 2.1 dev somewhere.

Comexans

@pfSense.User.1138:

I've just updated from 2.0 Release to 2.0.1 Release and found the vendor supplied VMware Tools failed to start on boot after the upgrade.
But fortunately its a quick and easy fix.

On the console open the shell ( press 8 )
Then type:
vmware-config-tools.pl
When vmware-config-tools.pl is done running type:
exit
To leave the shell again.

VMware Tools status for the pfSense VM in your vSphere Client should now be "Running".
As an extra check, try reboot the VM to verify that VMware Tools is running when pfSense is back up.

Hi,

I started a fresh install from the pfSense 2.0.1 amd64 liveCD on an ESX 5.1 host
The VM was initially configured with 2 E1000 NICs the second NIC is connected to a 802.1Q trunk vswitch (VLAN ID 4095).
I have been able to set the first NIC as the WAN interface (no VLAN) and the 5 VLANs on the second NIC as LAN and OPT1 to OPT4 interfaces.
The routing and firewalls seems to work correctly.

After reading this posts, I decided to benefit of vmxnet3 NICs as they are supposed to improve network throughput. I wanted also to be able to manage te VM with VMTools.

So I have installed the "native" VM Tools, by following this thread and also this tutorial :
http://www.howtoforge.com/how-to-install-vmware-tools-on-pfsense-freebsd
but I had to specify othe locations for PACKAGESITE variable (in order to dowload perl5 and compat6x-amd64 packages):```
setenv PACKAGESITE "ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8-stable/Latest/"

I had also to patch 2 following files according to http://redmine.pfsense.org/issues/2010 (to support VLAN tagging on vmxnet3):

*   /etc/inc/interfaces.inc

*   /etc/inc/globals.inc

I switched the 2 existing NICs to VMXNET 3, and resetup the interfaces and VLANs. It works !
But I still get an issue : vSphere Client reports that the VMTools are not installed (and not running) and I can't start it using vmware-config-tools.pl script :

vmware-config-tools.pl
Initializing...

Making sure services for VMware Tools are stopped.

Stopping VMware Tools services in the virtual machine:
Guest operating system daemon: done
Guest memory manager: done

The vmblock enables dragging or copying files between host and guest in a
Fusion or Workstation virtual environment. Do you wish to enable this feature?
[no]

No X install found.

Starting VMware Tools services in the virtual machine:
Switching to guest configuration: done
Guest memory manager: done
Guest operating system daemon: failed
Unable to start services for VMware Tools

Execution aborted.


Is anybody can help me to make this running ?

jacob81286

To fix your Error "guest operating system daemon: failed" while installing the vmware tools for esxi 5.1 issue the following commands


ln -s /usr/local/lib/compat/libm.so.4 /usr/local/lib
ln -s /usr/local/lib/compat/libc.so.6 /usr/local/lib
ln -s /usr/local/lib/compat/libthr.so.2 /usr/local/lib
ln -s /usr/local/lib/compat/libkvm.so.3 /usr/local/lib

asterix

Hi,
I am trying to get my VM working on VMXNET3 but pfSense won't detect the NICs. VMXNET2 NICs are identified and I can use them but I want to switch to VMXNET3.

I am running a AMD64 2.0.1 pfSense version on a VM with 8GB RAM.

The guide in this thread works but for VMXNET2 only. Are there an special 64-bit drivers for VMXNET3?

johnpoz

so you installed the native tools from vmware? This would have the drivers - Mine sees the nics, but I am running 32bit of pfsense.

vmx3f0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>ether 00:0c:29:1e:18:90
media: Ethernet 10Gbase-T
status: active
vmx3f1: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
options=403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso>ether 00:0c:29:1e:18:9a
media: Ethernet 10Gbase-T

I just don't currently have them enabled because I don't see any point to using them and had issues with vpn outbound from client inbound when they were being used.</rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></broadcast,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso></broadcast,simplex,multicast>