Vmware vmxnet3 nic vs. e1000 vs. hardware-install - throughput performance

miloman

So… I've been seeing all these posts saying the vmxnet3 nic driver in vmware will give you better performance over the e1000 driver.
But you would get the best performance by installing pfsense directly onto your hardware.

Tonight i set out to test these claims.

My test setup:

A shuttle XG41 with dual NICS. Intel Core 2 Duo E6400, 4096Mb Ram. And two laptops with lots of CPU power and gigabit nics. The laptops are running knoppix, and all bandwidth tests was done using iperf.

I used Pfsense 2.1 beta0 in my test. (I've done the same test with pfsense 2.0.1, the results are the same.)

The fist test i did was to connect the laptops with a crossover cable and test their maximum speed. Result 939Mbit

I then proceeded to install vmware esxi 5.0 on the shuttle, and in this i made a virtual 64bit freebsd machine with 2vcpu's and 1024mb ram and 2xe1000 nics. The wan interface got an ip called 10.0.0.15/24 and the lan 192.168.1.1/24. The laptops were then configured with an IP in their respective ranges, and a any any rule was created.

In the tests below laptop1 was connected straight to nic1 of the shuttle and laptop2 was connected straight to nic2.

Pfsense with E1000 nics. Result 850mbit cpu at 100%

I then installed the vmware drivers by using the guide from this post: http://forum.pfsense.org/index.php/topic,34043.0.htm (the post fom pfSense.User.1138)

Pfsense with vmxnet3 nics. Result 852mbit cpu at 100%

Then i took a live-cd with pfsense 2.1 image, put it on a USB-key and booted the shuttle on it, I then ran another test. Result 939mbit cpu at 30% according to the dashboard

The main reason i'm writing this post, is because i've had trouble finding information like this in the forums. Hopefully this will help someone else.
I've attached some screenshots on CPU usage in vmware when the iperf test was done, with e1000 and vmxnet3 drivers.

DSC06289.JPG_thumb

miloman

some screenshots

cpu_e1000.png_thumb

cpu_vmxnet3.png_thumb

interfaces_vmx.png_thumb

pfsense_vmware_console.png_thumb

podilarius

Very nice. Thanks for doing that.

dhatz

Interesting test, thanks for sharing.

I wonder what the numbers would be with pf disabled (no nat, no packet filtering).

podilarius

Are you talking about a VM setup with that disabled?

dhatz

Yes, although it's pretty safe to assume that the bottleneck is due to VM Ethernet driver …

stephenw10

Great post.
Like an oasis of fact in a desert of speculation! :)

Steve

heper

would the results be different when the hardware supports virtualization technology? (intel VT, AMD-v).
I wonder if the vmxnet drivers would benefit from them techs.

Thanks for the tests by the way :)

gesshoku

Hi,

i've recently started using pfsense again and it's running as a VM on my NAS.

i am curious to know, if one would get the same results using VT-d. i can pass the NICs directly to the pfsense VM. The reason why i haven't done this yet is because i have another VM that is a heavy downloader (WAN-speed is 128 Mbit). My thoughts were: with both VMs using the same controller, the traffic would stay within the hypervisor. If i dedicate the NICs to the pfsense VM only, i assume that traffic would have to leave the ESXi-Host and travel back through the switch.

Am i guessing correctly? Would that extra traffic be negligible compared to the stress i save the CPU?

Thanks.

miloman

@dhatz:

Interesting test, thanks for sharing.

I wonder what the numbers would be with pf disabled (no nat, no packet filtering).

Allright, so i disabled pf under "system - advanced - firewall/nat". I then ran the test using the e1000 driver, and the vmxnet3 driver. The results are similar. 100% cpu in vmware graphs, 850mbit throughput.

miloman

I've also tried enabling/disabling TSO, powerD, fast tcp forwarding etc… But so far i haven't been able to get above the 850mbit marker.

louis-m

notice your media is set at 10gbaseT. i'm using open vmtools and my all my intel gigabit cards with vmxnet3 are only recorded as 1000baseT

miloman

@louis-m:

notice your media is set at 10gbaseT. i'm using open vmtools and my all my intel gigabit cards with vmxnet3 are only recorded as 1000baseT

I'm using the vendor supplied vmtools, and 10Gbit is their default speed. Source:http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1013083

johnpoz

Thanks for doing this testing.. Could you see what cpu usage you get when your not maxing out the pipe.

Put a switch between, and on the 1 box set interface to 100full and then on pfsense you got Gig – so max your going to see is 1/10 of what nic can do.. Is the cpu usage less in this mode on the vmxnet3?

This would be more of a setup you might see in normal usage -- isp is not always a gig connection, and you rarely see 100% saturation of the line etc.

louis-m

ah, i knew there was a reason why i didn't pursue this further at the time. i was using vlans and the vmxnet driver didn't support that at the time. there is a patch but i don't want to apply it at the moment on 2.1_x64 and it doesn't appear that the nics are detected as i've added a new one with the vmxnet3 driver and pfsense isn't seeing it at the moment.
miloman, seeing as your interfaces have been found under the vmxnet3 driver, could you try and see if you can add a vlan for that interface? if so, i might push it a little further to get them working.

miloman

@johnpoz:

Thanks for doing this testing.. Could you see what cpu usage you get when your not maxing out the pipe.

Put a switch between, and on the 1 box set interface to 100full and then on pfsense you got Gig – so max your going to see is 1/10 of what nic can do.. Is the cpu usage less in this mode on the vmxnet3?

This would be more of a setup you might see in normal usage -- isp is not always a gig connection, and you rarely see 100% saturation of the line etc.

I see where you're going. I'll be doing this test later today.

miloman

@louis-m:

ah, i knew there was a reason why i didn't pursue this further at the time. i was using vlans and the vmxnet driver didn't support that at the time. there is a patch but i don't want to apply it at the moment on 2.1_x64 and it doesn't appear that the nics are detected as i've added a new one with the vmxnet3 driver and pfsense isn't seeing it at the moment.
miloman, seeing as your interfaces have been found under the vmxnet3 driver, could you try and see if you can add a vlan for that interface? if so, i might push it a little further to get them working.

Vlans are indeed supported under PfSense 2.1_beta0 with the vmxnet3nic. You don't need to use the patch.

For me the vmxnet3 NIC was essential for getting better performance/througput. But it was useless in a production setup seeing vlan tagging wasn't supported. After my tests, i don't see why i should bother installing the driver and introducing a potential vmware tools/driver crash when the performance of the e1000 is pretty much the same.

louis-m

thanks for the testing. does make me wonder why my vmxnet3 interfaces ain't showing. i'm using 2.1_x64 beta0 and when setting the driver to vmxnet3, pfsense doesn't see the additional interfaces.
vmxnet.ko is loaded and has corrected permissions but still doesn't show.

johnpoz

@miloman:

I see where you're going. I'll be doing this test later today.

So for example my internet connection is about 16MBps sustained - sure it boosts to like 25, but on say a sustained download it levels off at about 16MBps – so maybe in this scenario e1000 causes 40% cpu while vmxnet3 only uses 30% ?

miloman

Here ya go…

Throughput capped to 100mbit using a switch.

Test with computers connected to each other only by using a switch = 96.5Mbit (this number is used for reference as to which speeds are possible without any firewalling)
Test with firewall in between doing the routing/firewalling = 94.5mbit

You can see the CPU usage in the screenshot i've attached. In this test the vmxnet3 driver uses a bit less cpu than the e1000. But i'm not impressed.

100mbit-cap_cpu_vmxnet3_e1000.jpg_thumb