Download speed bottomed out after switching providers

WN1X

@Symlink What NIC are you using for the WAN? Have you tried disabling hardware offloaded checksums?

Symlink

@stephenw10 I'm using the web interface in the browser on the windows pc that is connected to the vpn. Disconnect from vpn, 2Mbps down. Connect to vpn, 300Mbps down. The normal speedtest.net site.

Symlink

@WN1X Nic is a 4 port intel I350. One port is comcast, one is conexon, and another is lan. The 4th port is not used in pfsense. I've also moved ports and cables between comcast and conexon with no differences. TX checksums are off and have been since I set this up last year. The conexon port was unused before switching to fiber.

Symlink

Since this is a vm I spun up another one with the same specs, basic install on another network range. Assigned it the same nic ports for conexon and lan as the other install. Changed my desktop to the new ip range. Now I get 400Mbps down and 900Mbps up on the desktop that was getting 2Mbps down before. I'm looking to see what differences could be causing this.

JonathanLee

Check the MTU the isp uses

stephenw10

Are you passing the hardware through to the VM or using virtual switches and vtnet NICs?

Bad TCP throughput but good UDP (assuming the VPN is using UDP) looks like a hardware offloading issue IMO.

Symlink

@stephenw10 I'm not passing any hardware through to this vm. The vpn is using UDP. I'm almost positive it's a setting or config in pfSense as a new basic install with no other differences gives the expected download and upload speeds.

stephenw10

What NIC type is the VM using?

Symlink

@stephenw10 Realtek RTL8139

stephenw10

Hmm, it doesn't have a virtualised NIC type? I was expecting xn(4).

Symlink

@stephenw10

xn0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: WAN
	options=3<RXCSUM,TXCSUM>
	ether ea:b2:72:44:1c:3e
	media: Ethernet manual
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
xn1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: LAN
	options=3<RXCSUM,TXCSUM>
	ether 72:c8:eb:c2:6d:68
	media: Ethernet manual
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
xn2: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: FIBER
	options=3<RXCSUM,TXCSUM>
	ether d6:0c:d4:16:41:17
	media: Ethernet manual
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Ip address lines redacted.

I switched the nic type to e1000 and I'm getting 450 down/300 up currently on fiber. I disabled all nic offloading for those 3 interfaces with no changes as well.

xe vif-param-set uuid=7703da34-7578-1b0c-5420-738b769bcc36 other-config:ethtool-gso=off
xe vif-param-set uuid=7703da34-7578-1b0c-5420-738b769bcc36 other-config:ethtool-ufo=off
xe vif-param-set uuid=7703da34-7578-1b0c-5420-738b769bcc36 other-config:ethtool-tso=off
xe vif-param-set uuid=7703da34-7578-1b0c-5420-738b769bcc36 other-config:ethtool-sg=off
xe vif-param-set uuid=7703da34-7578-1b0c-5420-738b769bcc36 other-config:ethtool-tx=off
xe vif-param-set uuid=7703da34-7578-1b0c-5420-738b769bcc36 other-config:ethtool-rx=off

Symlink

Changing the nic type and opting out of the CGNAT network address space seems to have corrected whatever issue I was having.