More than 1 Gbps using VMWare ESXi with VMXNET3?

bbennett

Hello, I'm not sure what forum this should go in, so I figure this is a good start.

I'm utilizing pfSense 2.4.3-RELEASE (amd64) built on Mon Mar 26 18:02:04 CDT 2018 FreeBSD 11.1-RELEASE-p7.
It is running ontop of a VMWare ESXi host running 6.0.0, Build 3620759.

It is also running Open-VM-Tools 10.1.0,1.

When the Firewall was set up initially it was configured with E1000 VMNIC's. The problem with this is that all of the network interfaces are limited to 1 Gbps, but the physical server this firewall is running on has 10 Gbps NIC's.

From my understanding, to get a 10 Gbps link, I'd need to reconfigure the interfaces as VMXNET3 VMNIC's.

I'd like to know, what would be the best way to do this? The firewall is only a single firewall to the entire site, but I do have a "backdoor" way to get in if connectivity fails.

Are there any caveats I should know about?

KOM

You will have to do this onsite. No way you're going to add & remove NICs and reconfigure everything remotely, unless you have another way in to the network. The way that I've seen others do this is:

make a config.xml backup via Diagostics - Backup & Restore
power down the vm
in vmware, remove all the e1000 NICs
in vmware, add your vmx3 NICs
edit your config.xml file and replace all instances of em0, em1, em2 etc with vmx0, vmx1, vmx2 etc.
install pfSense fresh
restore your config.xml

bbennett

OK, So I've changed over all of the NIC's to VMXNET3, but I'm still not seeing the throughput I want.

The physical NIC's on the server are reporting back 10000 Mbps and they are on a LAG on a virtual distributed portgroup, so in theory, bandwidth should be 20 Gbps.

When I take a peek in the shell of pfSense, an ifconfig | grep media just reports back "Ethernet autoselect" for all interfaces.

I'm honestly not sure how else I can test. The server is running a Xeon E5-2650 @ 2.3 Ghz and it has 40 Logical processors, so that's probably not an issue. The VM itself has 2 vCPU with 1 socket/2 cores. CPU load never goes above 70 percent.

What else can I tweak to get more throughput?

johnpoz

And what throughput are you seeing?

bbennett

@johnpoz When I go to speedtest.net and run a test against our ISP (Who is on the same network) I get about 700 Mbps, which is what I would expect for a 1 Gbps connection since there's other traffic going across it.

I think it won't hurt to also have a chat with our Data Centre provider as well to ensure that they're also not throttling us.

I'll update the forums as soon as I find out.

heper

right .... you are having too many unknown factors.

you won't get 10Gbe from speedtest.net (or any other similar that i know)
10Gbe over the internet will currently be hard to reach due to latency
pfsense does not do 10Gbe on small sized packets .... on minimum sized packets its maxes out at around 2.7gbit/s
you can have a trillion cores ... it doesn't matter. ghz has a bigger impact then huge core numbers.

KOM

@bbennett That's a poor way to test throughput. Use iperf on each side of your WAN. Testing a 10G connection over the Internet is crazy.