9000 MTU WAN and slow internet in VMs

AndyRH

@itestandroid You did not say if you did, but try changing the MTU to 1500. Your ISP may do jumbo frames, but I am pretty certain the Internet does not. Me and many others get 1Gb with 1500 MTU.

johnpoz

@andyrh I wonder if he thinks he "needs" jumbo to do gig?

There is no freaking way his isp is using jumbo his connection.. I don't buy it..

It sure is not "needed" by any shape of the imagination to do gig or even higher.. Here doing 2+ gig guess what 1500 mtu..

$ iperf3.exe -c 192.168.10.10
Connecting to host 192.168.10.10, port 5201
[  5] local 192.168.10.9 port 57345 connected to 192.168.10.10 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   286 MBytes  2.40 Gbits/sec
[  5]   1.00-2.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   2.00-3.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   3.00-4.00   sec   282 MBytes  2.37 Gbits/sec
[  5]   4.00-5.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   5.00-6.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   6.00-7.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   7.00-8.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   8.00-9.00   sec   282 MBytes  2.36 Gbits/sec
[  5]   9.00-10.00  sec   283 MBytes  2.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  2.76 GBytes  2.37 Gbits/sec                  sender
[  5]   0.00-10.01  sec  2.76 GBytes  2.37 Gbits/sec                  receiver

iperf Done.

Here are the 2 different boxes and interfaces doing that - 1500 MTU..

iTestAndroid

@andyrh @johnpoz Well, when I setup everything initially, my download speeds were around 500mbps and no matter what we did, we couldn't improve it, so someone suggested MTU 9000 and we went with that. My end and ISP's router side, we both set WAN to 9000.

I don't think that's relevant to my problem. My problem is inside Proxmox and inside VMs that I have in Proxmox, speed is not good. What can I check and fix?

For instance, if I'm doing PCIE passthrough

Should I leave these unchecked or checked since my pfSense is a VM guest?

johnpoz

@itestandroid said in 9000 MTU WAN and slow internet in VMs:

so someone suggested MTU 9000

Yeah don't listen to that guy going forward.. ;)

Put a hardware box on it.. Do you get gig with that.. Validate your isp is giving you the speed your paying for - before you throw something as variable as Virtual into it..

Could be as simple as plugging in a laptop to the router - do you see gig.. There you go your sure the problem is in your VM hardware or setup..

edit: this has zero to do with routing or multiple wan - moving to virtual area.

iTestAndroid

@johnpoz Ok, so should I contact the ISP and ask them to return the MTU to 1500 ?

As for laptop test, since they are giving me SFP+ and fiber I couldn't easily do (this is in a datacenter, not home)

I bought one SFP+ to RJ45 10Gb adapter from Amazon (https://www.amazon.com/gp/product/B07MWSSN8K/)

But they said its probably this device and it can't handle the traffic.

What do you suggest for my pfSense settings and optimization and tuning, maybe I can make it better in settings?

AndyRH

@johnpoz said in 9000 MTU WAN and slow internet in VMs:

Yeah don't listen to that guy going forward.. ;)

Seconded.

AndyRH

Set your MTU to 1500, smaller works just fine.

Verify the speed, then vanilla pfSense on the Proxmox system, test speed.
1 step at a time until you stop performing, then analyze that step.
Too many people run virtual pfSense for it to be hard to configure.

johnpoz

@itestandroid said in 9000 MTU WAN and slow internet in VMs:

me SFP+ and fiber I couldn't easily do (this is in a datacenter, not home)

Your converter hands off to ethernet - plug something into that.. Maybe its junk? True.. But you need to validate your speed before you throw any virtual anything into the mix or your just going to be chasing your tail..

How do you know its something in virtual settings or pfsense vm setup if you don't even know if your getting gig from the wire to specific hardware that you know can do gig... Maybe they are right and that media converter you bought is junk.. Ok validate that first before you start looking at what offload settings or system tunables to try and tweak. Or mtus to change.

edit: just noticed this

Too many people run virtual pfSense for it to be hard to configure.

hahah - ROFL good one!!!

stephenw10

@itestandroid said in 9000 MTU WAN and slow internet in VMs:

Should I leave these unchecked or checked since my pfSense is a VM guest?

I would disable all the hardware off loading options unless you have a good reason not to. They don't help much and could be causing problems.
You don't need to have AltQ for hn NICs enabled outside Hyper-V/Azure.

What's the host hardware? Is is maxing out any CPU cores when you test?

Steve

iTestAndroid

@stephenw10 Yeah, I think my ISP is no good, its not my hardware or my setup.

My VM host is crazy fast, pfSense guest have like 12 core (3.4GHz) and 48GB RAM. So its 100% not my VM or my hardware. I'll check the boxes for hardware offloading and AltQ. I'm going to contact my ISP, they are messing this up, not my setup or my hardware. Thanks for your help and giving me confidence its not me

iTestAndroid

@johnpoz We just tested with 1500 MTU, everything slowed down. I don't know what's weird about 9000 MTU, but honestly everything was faster, not perfect, still 500mbps-ish, but faster than 1500 MTU. We changed it on my side and ISP side, 9000 MTU acted better. No idea why

stephenw10

Hmm, higher throughput with higher MTU like that implies something in that local link it hitting a PPS limit. Mismatched MTU will cause horrible throughput issues generally.

But, yeah as mentioned, the first thing I would be doing here connecting something basic and baremetal to the connection to prove it can do 1Gbps. That really shouldn't be very demanding though.

Steve