More than 1 Gbps using VMWare ESXi with VMXNET3?

  • Hello, I'm not sure what forum this should go in, so I figure this is a good start.

    I'm utilizing pfSense 2.4.3-RELEASE (amd64) built on Mon Mar 26 18:02:04 CDT 2018 FreeBSD 11.1-RELEASE-p7.
    It is running ontop of a VMWare ESXi host running 6.0.0, Build 3620759.

    It is also running Open-VM-Tools 10.1.0,1.

    When the Firewall was set up initially it was configured with E1000 VMNIC's. The problem with this is that all of the network interfaces are limited to 1 Gbps, but the physical server this firewall is running on has 10 Gbps NIC's.

    From my understanding, to get a 10 Gbps link, I'd need to reconfigure the interfaces as VMXNET3 VMNIC's.

    I'd like to know, what would be the best way to do this? The firewall is only a single firewall to the entire site, but I do have a "backdoor" way to get in if connectivity fails.

    Are there any caveats I should know about?

  • You will have to do this onsite. No way you're going to add & remove NICs and reconfigure everything remotely, unless you have another way in to the network. The way that I've seen others do this is:

    • make a config.xml backup via Diagostics - Backup & Restore
    • power down the vm
    • in vmware, remove all the e1000 NICs
    • in vmware, add your vmx3 NICs
    • edit your config.xml file and replace all instances of em0, em1, em2 etc with vmx0, vmx1, vmx2 etc.
    • install pfSense fresh
    • restore your config.xml

  • OK, So I've changed over all of the NIC's to VMXNET3, but I'm still not seeing the throughput I want.

    The physical NIC's on the server are reporting back 10000 Mbps and they are on a LAG on a virtual distributed portgroup, so in theory, bandwidth should be 20 Gbps.

    When I take a peek in the shell of pfSense, an ifconfig | grep media just reports back "Ethernet autoselect" for all interfaces.

    I'm honestly not sure how else I can test. The server is running a Xeon E5-2650 @ 2.3 Ghz and it has 40 Logical processors, so that's probably not an issue. The VM itself has 2 vCPU with 1 socket/2 cores. CPU load never goes above 70 percent.

    What else can I tweak to get more throughput?

  • LAYER 8 Global Moderator

    And what throughput are you seeing?

  • @johnpoz When I go to and run a test against our ISP (Who is on the same network) I get about 700 Mbps, which is what I would expect for a 1 Gbps connection since there's other traffic going across it.

    I think it won't hurt to also have a chat with our Data Centre provider as well to ensure that they're also not throttling us.

    I'll update the forums as soon as I find out.

  • right .... you are having too many unknown factors.

    1. you won't get 10Gbe from (or any other similar that i know)
    2. 10Gbe over the internet will currently be hard to reach due to latency
    3. pfsense does not do 10Gbe on small sized packets .... on minimum sized packets its maxes out at around 2.7gbit/s
    4. you can have a trillion cores ... it doesn't matter. ghz has a bigger impact then huge core numbers.

  • @bbennett That's a poor way to test throughput. Use iperf on each side of your WAN. Testing a 10G connection over the Internet is crazy.

Log in to reply