Slow internet access within VM

[punchy]

I'm running into an issue that I can't really explain and I'm hoping someone here can help point me in a direction to investigate as to what's going on. I have a pretty basic setup and I'll list the important bits below:

• I have a server box running VMWare ESXi 6.5 where pfSense 2.6 is running in its own VM and another VM running Ubuntu server
• pfSense VM is configured with 2-core CPU, 2GB of RAM, 16GB hard drive and 2 network adapters (WAN/LAN) both using the VMXNET 3 driver
• internet modem (providing 1.5Gbps) is in bridge mode and connected to one of the physical network ports of the server (gigabit, builtin to the motherboard)
• dumb 8-port switch connected to the other physical network port on the server (also gigabit, builtin to the motherboard)

What I'm seeing is as follows:

• laptop connected straight to the modem using ethernet I get close to 1gbps (high 900s) when using speedtest
• laptop connected to the dumb switch mentioned above also gets close 1gbps using speedtest
• running the speedtest cli python package on both the pfsense VM and ubuntu VM results in speeds of 200-400 mbps; this is confirmed by trying to download anything on the Ubuntu VM maxes out to 30-40MBps
• I ran iperf tests between my laptop (connected with ethernet to the dumb switch) and the ubuntu and pfSense VMs and I get 1gbps; iperf tests between the pfSense and ubuntu VMs also results in 1gbps
• cpu never gets close to even 50% on the pfSense vm

The iperf test leads me to believe that the network infrastructure on my LAN is capable of 1gbps. What I really don't understand is how can I get close to 1gbps to the internet using my laptop connected to the switch on my LAN but nowhere close to that when running straight from either of my VMs to the internet. Everything should be routed through the pfSense VM so I can't explain how the connection from my laptop to the internet isn't affected.

Any help here would be really appreciated.

[NollipfSense]

@punchy said in Slow internet access within VM:

laptop connected straight to the modem using ethernet I get close to 1gbps (high 900s) when using speedtest

It seems like you should be getting more here...like 1450gbps. Never used ESXI for pfSense but I know for Proxmox/pfSense VM, it's important for WAN/LAN to passthrough. Is the switch, when testing, connected directly to pfSense LAN?

[punchy]

@NollipfSense thanks for the reply. All my network gear maxes out at 1gbps, so I don't expect to be getting anywhere close to the 1.5gbps that my internet connection provides (though fast.com did report 1.2-1.3gbps).

I only have two network ports on my server, both on the motherboard. They're configured in VMWare without any pass through, the VMs are getting virtual interfaces. The modem is connected to one of the network ports, and the switch is connected to the other. The switch is what everything else in my network connects to (that is outside the VMs). This is the configuration I used during the test and daily as well. The only time I had a different configuration is when I connected my laptop directly to the modem, but otherwise it's the same test setup.

[johnpoz]

@punchy said in Slow internet access within VM:

cli python package on both the pfsense VM and ubuntu VM results in speeds of 200-400 mbps

This while a test, really isn't a valid way to test what the pfsense vm can route/firewall - you need to route through this vm with your speedtest client to validate it can route/firewall at the speed your isp provides.

[punchy]

@johnpoz said in Slow internet access within VM:

@punchy said in Slow internet access within VM:

cli python package on both the pfsense VM and ubuntu VM results in speeds of 200-400 mbps

This while a test, really isn't a valid way to test what the pfsense vm can route/firewall - you need to route through this vm with your speedtest client to validate it can route/firewall at the speed your isp provides.

Would the following test cover what you're proposing: laptop connected with ethernet to the same switch that my server's LAN port is connected to? If that's the case, then I've done that test at I'm getting close to gigabit speeds using speedtest in the browser.

I do wonder though, what sort of issues do you see with using the speedtest cli command on one of the VMs to test my internet throughput? I'd expect that to be bottlenecked only by virtual network infrastructure, but for some reason the internet speeds are almost a third of what I'd be expecting.

[johnpoz]

@punchy said in Slow internet access within VM:

switch that my server's LAN port is connected to? If that's the case

No - you need to test actually routed through pfsense..

laptop -switch - lan (pfsense) wan -- modem -- internet

You need to test what pfsense can actually route.

Pfsense is not meant to be a client in a speed test.. its a "router" if you want to test what speed you get then route through it.. Don't use it as some client.. If you have a beefy enough pfsense box or slow enough internet, its possible you can see your full internet speed running the cli.. But its also possible you won't see what it can do via routing the packets vs trying to actually process them..

While its fine to validate your seeing traffic, or once you know what is full speed for the cli after you have validating your routing at full speed, etc..

But I would not take cli running on pfsense as same as you would see when routing.

[stephenw10]

You will need to set the tunable to get multiqueue on the vmx NICs:
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#vmware-vmx-4-interfaces

I would still expect to have seen better than 320Mbps though unless the per core speed is low.

Steve

[punchy]

@stephenw10 said in Slow internet access within VM:

You will need to set the tunable to get multiqueue on the vmx NICs:
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#vmware-vmx-4-interfaces

I would still expect to have seen better than 320Mbps though unless the per core speed is low.

Steve

Thank you so much for that link, massive improvement just from that. I'm now seeing 600-800 Mbit/s with speedtest running on both the pfsense VM and also my ubuntu VM. That's much closer to what I'd expect to be getting.

My server is really old and it has an i7 950 @ 3.07GHz but thought it should be good enough for this purpose. The pfsense VM is currently configured with two virtual cpus; I'm not sure if giving it more CPUs would make a difference after making this change, but I can experiment with that.

[stephenw10]

Mmm, that is old but still. One 3GHz core would usually be enough. If it's actually running at 3GHz....