Performance tests of pfSense 2.1 vs. 2.2 and different NIC types on VMware ESXi

Supermule

This is 2.1.5 on E1000….

cmb

pfSense 2.2 with FreeBSD builtin vmxnet3 driver:
Down 406 Mbit/s, Up 1.3 Mbit/s (WTF?!)

Clearly that is not right.. I can tell you for fact that is wrong.

Yeah there's something seriously wrong there that has nothing to do with the type of NIC being used. Most of our dev/test setup is on vmxnet3. One quick test through one of those VMs:
http://www.speedtest.net/my-result/4106023142

Hard to get great results when you have a gigabit connection in a datacenter, as speed test servers vary in performance so much at that level. But there's real Internet traffic that's way beyond that "ideal circumstance" with local-only traffic.

Supermule

And you are running 100gbit vswitches??? :D

@cogumel0:

Just for completeness, I took the 2 Ubuntu VMs I had and put them on the same LAN and repeated the iPerf test:

15.8GB/s .. ..

Didn't quite expect to get these speed with pfSense in the middle, even with pf disabled, but it shows what the server can handle.

Just FYI it is a Dell DL380 G6 2x Xeon 6 core 72GB ram

FauxShow

Getting the official VMWare Tools to work on 2.2 was a pain, but that offered the best speeds for me; I'm able to get 2.5Gb/s through pfSense using the VMXNet3 drivers.
Using Open-VM-Tools I was able to get just 2Gb/s

Using vSphere 5.5 U2 on Enterprise Plus licensing on a HP c7000 blade chassis with two hosts running 10GbE FlexNET adapters.

KOM

Getting the official VMWare Tools to work on 2.2 was a pain, but that offered the best speeds for me

Considering how pfSense 2.2 is built on FreeBSD 10.1 which has direct support for vmxnet3 NICs, I'm wondering what the real VMware Tools install is giving you. One of the regulars here, johnpoz, was testing just yesterday and saw problems when using VMware Tools under 2.2.

https://forum.pfsense.org/index.php?topic=90535.msg505761#msg505761

johnpoz

I was not able to get them to work.. I could ping stuff but could not access anything using the native tools.. Following the great instructions here..

http://www.v-front.de/2015/01/pfsense-22-was-released-how-to-install.html

He also had problems with the native tools. The native tools work other than the driver for vmx3… I would be very interested how you got the native tools to work..

I can try disabling the checksumming

rx-checksumming: on
tx-checksumming: on

Which I show is still on, at the esxi level. But I have offloading off.

[root@esxi:~] ethtool –show-pause vmnic2
Pause parameters for vmnic2:
Autonegotiate: on
RX: off
TX: off

I can play around with them a bit more.. So if you say they are working I would love to know how you got them working.

Here are the nics I have in my esxi box

[root@esxi:~] esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:04:00.0 tg3 Up 1000Mbps Full 2c:76:8a:ad:f6:56 1500 Broadcom Corporation NetXtreme BCM5723 Gigabit Ethernet
vmnic1 0000:02:00.0 e1000e Up 1000Mbps Full 00:1f:29:54:17:14 1500 Intel Corporation 82571EB Gigabit Ethernet Controller
vmnic2 0000:02:00.1 e1000e Up 1000Mbps Full 00:1f:29:54:17:15 1500 Intel Corporation 82571EB Gigabit Ethernet Controller
vmnic3 0000:03:00.0 e1000e Up 1000Mbps Full 68:05:ca:01:b3:26 1500 Intel Corporation 82574L Gigabit Network Connection

FauxShow

You were exactly correct about turning options off on the NICs to get pfSense 2.2 with VMWare Tools to work on vSphere 5.5U2+.

I noticed by monitoring the console that if I changed the link speed of a VMXNET3 adapter that it changed the enabled options from what you show here as RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO to just VLAN_MTU,VLAN_HWTAGGING

So I made a script in /usr/local/etc/rc.d that has a "ifconfig vmx3f0 -rxcsum -txcsum -tso4" for each adapter, and that gets it to work upon startup. So far it works and iperf between two RHEL VMs shows an average of 2.5 or so Gb/s through VMXNET3 pfSense interfaces.

Don't know if it's worth it though, as it seems to be about 100Mb/s difference from the native drivers to Open VM Tools and then up to VMWare Tools.

johnpoz

Well I can validate that it in does seem to work.. As soon as turned that stuff off, traffic flows through pfsense.

This should be added to the docs, I will do that sometime this weekend I hope. Want to play around with it a bit more, etc. I want to check the performance between local network segments, etc.

Thanks! While I agree that native driver with the openvm tools is much easier method.. Unless there some drastic difference in performance just stick with that method.. Nice thing about the native drivers, etc is very easy to get pfsense up and running on esxi vs having to install tools before it works, etc. Guess I could put up a ova for others with the tools already installed. So your saying this does not survice reboot?

FauxShow

Correct, the adapters need to be snapped into action every time. I tried to do "ifconfig vmx3f0 media autoselect" but that just gave an error code and "ifconfig vmx3f0 media 10Gbase-T" didn't wake them up. If you use the webgui to select the link speed then that does work though, and you can monitor the console to see.

dburkland

Just wanted to say thanks for this post as this helped me get past this connectivity issue with PfSense 2.2.3 + vSphere 6.0. I created the following RC script (not pretty but it gets the job done) based on the example provided which seems to do the trick:


#!/bin/sh

for vnic in $(/sbin/ifconfig | grep "vmx3f[0-9]:" | awk -F ':' '{ print $1 }'); do
	/usr/bin/logger -t vmxnetfix.sh "Disabling checksumming on $vnic"
	/sbin/ifconfig $vnic -rxcsum -txcsum -tso4
	/sbin/ifconfig $vnic down
	/sbin/ifconfig $vnic up
	/usr/bin/logger -t vmxnetfix.sh "Checksumming has been properly disabled on $vnic"
done

Cheers,

Dan