Performance tests of pfSense 2.1 vs. 2.2 and different NIC types on VMware ESXi
-
Me either and I have no need to.
-
Ok, so here are my test results.
These were carried out on ESXi 5.5U2.
The setup is as follows: 3 VMs, all on the same host, 2 running Ubuntu 14.04, 1 running pfSense.
pfSense VM has 1 WAN and 2 LANs, 2 CPUs with 2 cores each and 8GB ram.
Ubuntu1 has 4 CPUs, 1 NIC on LAN1 and 4GB ram, running from LiveCD
Ubuntu2 has 4 CPUs, 1 NIC on LAN2 and 4GB ram, running from LiveCD.
The only changes made in pfSense were to enable DHCP on OPT1 and setup any-to-any IPv4 and IPv6 rules on OPT1 (to mirror those on LAN1).
Tests were done using iPerf and always from Ubuntu1 to Ubuntu2 (LAN1 to OPT1 in pfSense).
There is only one other VM running on this physical host (a live pfSense box but traffic was pretty much non-existent during the test and no packages installed so its affects on the test results are negligible)
The test results are the average of running iPerf 5x on each configuration.
Test results:
1.546GB - pfSense 2.1.5 E1000 pf enabled
2.512GB - pfsense 2.1.5 E1000 pf disabled
1.91GB - pfSense 2.1.5 VMX3 pf enabled
2.554GB - pfSense 2.1.5 VMX3 pf disabled1.474GB - pfSense 2.2 E1000 pf enabled
2.334GB - pfSense 2.2 E1000 pf disabled
1.818GB - pfSense 2.2 VMX3 pf enabled
2.732GB - pfSense 2.2 VMX3 pf disabledBased on these results, I think it's fair to draw the following conclusions:
- Performance with pf enabled is always slower in pfSense 2.2
- Bigger gains are achieved in pfSense 2.2 with pf disabled
- While there is a small decrease in speed with VMX3 in pfSense 2.2 in comparison to pfSense 2.1.5 with pf enabled, I don't think this is to do with the driver as much as it is to do with pfSense itself, as the same speed decrease was noticed when using E1000.
In short, I don't think that VMware Tools VMX3 driver will provide any better performance increase vs the one from FreeBSD 10.1 (though it's impossible to say without testing it, you never know…) but an interesting thing to point out is that out of the box pfSense 2.2 performs worse than pfSense 2.1.5 with pf enabled (though better with pf disabled when using VMX3).
Are there any configuration changes I can do to attempt to increase the performance in 2.2?
-
so basically there is "only' a 300-400 mbit increase with the vmxnet drivers.
i was hoping bsd10.1 with the new pf would handle lots more …. in my tests, increasing virtual cores didn't make much of a difference .
Perhaps it needs serious tuning to get to speeds between 5-8gbit ??
-
Don't suppose you could do the same test on bare hardware?
-
Don't suppose you could do the same test on bare hardware?
Well, not on a server, but I could put pfSense on a desktop. But how would you then want me to carry out the tests? The speed would be capped by the NIC at 1GB…
-
Thats ok - Just a simple test that shows throughput and cpu load would still be informative.
Better if you had a 10bg LAN of course but still should be nice to see.
-
Just for completeness, I took the 2 Ubuntu VMs I had and put them on the same LAN and repeated the iPerf test:
15.8GB/s .. ..
Didn't quite expect to get these speed with pfSense in the middle, even with pf disabled, but it shows what the server can handle.
Just FYI it is a Dell DL380 G6 2x Xeon 6 core 72GB ram
-
well - It the two ubuntu machines were on the same LAN (same switch) then it would do an end run around pfsense completely. (I'm sure you already know)
I'm just stating the obvious for the few people it might not be obvious to.
-
Correct, I just added that to show what the server can handle, and what the difference between having the traffic go through pfSense and not is.
Still… I expected pfSense to not slow things as much, specially with pf disabled... It's a 600% decrease in performance by having the traffic going through pfSense. I'm sure that the ESXi host has something to do with it too since it's two different virtual switches involved but...
-
This is 2.1.5 on E1000….
-
pfSense 2.2 with FreeBSD builtin vmxnet3 driver:
Down 406 Mbit/s, Up 1.3 Mbit/s (WTF?!)Clearly that is not right.. I can tell you for fact that is wrong.
Yeah there's something seriously wrong there that has nothing to do with the type of NIC being used. Most of our dev/test setup is on vmxnet3. One quick test through one of those VMs:
http://www.speedtest.net/my-result/4106023142Hard to get great results when you have a gigabit connection in a datacenter, as speed test servers vary in performance so much at that level. But there's real Internet traffic that's way beyond that "ideal circumstance" with local-only traffic.
-
And you are running 100gbit vswitches??? :D
Just for completeness, I took the 2 Ubuntu VMs I had and put them on the same LAN and repeated the iPerf test:
15.8GB/s .. ..
Didn't quite expect to get these speed with pfSense in the middle, even with pf disabled, but it shows what the server can handle.
Just FYI it is a Dell DL380 G6 2x Xeon 6 core 72GB ram
-
Getting the official VMWare Tools to work on 2.2 was a pain, but that offered the best speeds for me; I'm able to get 2.5Gb/s through pfSense using the VMXNet3 drivers.
Using Open-VM-Tools I was able to get just 2Gb/sUsing vSphere 5.5 U2 on Enterprise Plus licensing on a HP c7000 blade chassis with two hosts running 10GbE FlexNET adapters.
-
Getting the official VMWare Tools to work on 2.2 was a pain, but that offered the best speeds for me
Considering how pfSense 2.2 is built on FreeBSD 10.1 which has direct support for vmxnet3 NICs, I'm wondering what the real VMware Tools install is giving you. One of the regulars here, johnpoz, was testing just yesterday and saw problems when using VMware Tools under 2.2.
https://forum.pfsense.org/index.php?topic=90535.msg505761#msg505761
-
I was not able to get them to work.. I could ping stuff but could not access anything using the native tools.. Following the great instructions here..
http://www.v-front.de/2015/01/pfsense-22-was-released-how-to-install.html
He also had problems with the native tools. The native tools work other than the driver for vmx3… I would be very interested how you got the native tools to work..
I can try disabling the checksumming
rx-checksumming: on
tx-checksumming: onWhich I show is still on, at the esxi level. But I have offloading off.
[root@esxi:~] ethtool –show-pause vmnic2
Pause parameters for vmnic2:
Autonegotiate: on
RX: off
TX: offI can play around with them a bit more.. So if you say they are working I would love to know how you got them working.
Here are the nics I have in my esxi box
[root@esxi:~] esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:04:00.0 tg3 Up 1000Mbps Full 2c:76:8a:ad:f6:56 1500 Broadcom Corporation NetXtreme BCM5723 Gigabit Ethernet
vmnic1 0000:02:00.0 e1000e Up 1000Mbps Full 00:1f:29:54:17:14 1500 Intel Corporation 82571EB Gigabit Ethernet Controller
vmnic2 0000:02:00.1 e1000e Up 1000Mbps Full 00:1f:29:54:17:15 1500 Intel Corporation 82571EB Gigabit Ethernet Controller
vmnic3 0000:03:00.0 e1000e Up 1000Mbps Full 68:05:ca:01:b3:26 1500 Intel Corporation 82574L Gigabit Network Connection -
You were exactly correct about turning options off on the NICs to get pfSense 2.2 with VMWare Tools to work on vSphere 5.5U2+.
I noticed by monitoring the console that if I changed the link speed of a VMXNET3 adapter that it changed the enabled options from what you show here as RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO to just VLAN_MTU,VLAN_HWTAGGING
So I made a script in /usr/local/etc/rc.d that has a "ifconfig vmx3f0 -rxcsum -txcsum -tso4" for each adapter, and that gets it to work upon startup. So far it works and iperf between two RHEL VMs shows an average of 2.5 or so Gb/s through VMXNET3 pfSense interfaces.
Don't know if it's worth it though, as it seems to be about 100Mb/s difference from the native drivers to Open VM Tools and then up to VMWare Tools.
-
Well I can validate that it in does seem to work.. As soon as turned that stuff off, traffic flows through pfsense.
This should be added to the docs, I will do that sometime this weekend I hope. Want to play around with it a bit more, etc. I want to check the performance between local network segments, etc.
Thanks! While I agree that native driver with the openvm tools is much easier method.. Unless there some drastic difference in performance just stick with that method.. Nice thing about the native drivers, etc is very easy to get pfsense up and running on esxi vs having to install tools before it works, etc. Guess I could put up a ova for others with the tools already installed. So your saying this does not survice reboot?
-
Correct, the adapters need to be snapped into action every time. I tried to do "ifconfig vmx3f0 media autoselect" but that just gave an error code and "ifconfig vmx3f0 media 10Gbase-T" didn't wake them up. If you use the webgui to select the link speed then that does work though, and you can monitor the console to see.
-
Just wanted to say thanks for this post as this helped me get past this connectivity issue with PfSense 2.2.3 + vSphere 6.0. I created the following RC script (not pretty but it gets the job done) based on the example provided which seems to do the trick:
#!/bin/sh for vnic in $(/sbin/ifconfig | grep "vmx3f[0-9]:" | awk -F ':' '{ print $1 }'); do /usr/bin/logger -t vmxnetfix.sh "Disabling checksumming on $vnic" /sbin/ifconfig $vnic -rxcsum -txcsum -tso4 /sbin/ifconfig $vnic down /sbin/ifconfig $vnic up /usr/bin/logger -t vmxnetfix.sh "Checksumming has been properly disabled on $vnic" doneCheers,
Dan
