Download speed bottomed out after switching providers

Symlink

Hi all,

I'm hoping someone here can point me in the right direction.

I'm in the process of switching from Comcast cable to Conexon fiber. I get 900Mbps+ down and 25Mbps up on comcast with a bridged modem. Plug the same pfsense router into the ONT for fiber, reboot everything, and I get 2Mbps down and 900Mbps up. Switch back to cable, same speeds I have had before. Speedtest from the router is 900+Mbps up and down, just everything on the inside has almost no download speed at all. I have no limiters in place in pfSense. pfSense version is 2.7.2-RELEASE and is in a vm in XCP-NG 8.2.1 with 16 cpus and plenty of ram. I've checked MTU sizes, TX checksumming is off. If I connect though a PIA vpn on my desktop I get 300Mbps up and down.

Is there some setting that's fiber specific that I'm missing?

stephenw10

@Symlink said in Download speed bottomed out after switching providers:

If I connect though a PIA vpn on my desktop I get 300Mbps up and down.

Through the same pfSense setup that only sees 2Mbps down?

How exactly are you testing?

WN1X

@Symlink What NIC are you using for the WAN? Have you tried disabling hardware offloaded checksums?

Symlink

@stephenw10 I'm using the web interface in the browser on the windows pc that is connected to the vpn. Disconnect from vpn, 2Mbps down. Connect to vpn, 300Mbps down. The normal speedtest.net site.

Symlink

@WN1X Nic is a 4 port intel I350. One port is comcast, one is conexon, and another is lan. The 4th port is not used in pfsense. I've also moved ports and cables between comcast and conexon with no differences. TX checksums are off and have been since I set this up last year. The conexon port was unused before switching to fiber.

Symlink

Since this is a vm I spun up another one with the same specs, basic install on another network range. Assigned it the same nic ports for conexon and lan as the other install. Changed my desktop to the new ip range. Now I get 400Mbps down and 900Mbps up on the desktop that was getting 2Mbps down before. I'm looking to see what differences could be causing this.

JonathanLee

Check the MTU the isp uses

stephenw10

Are you passing the hardware through to the VM or using virtual switches and vtnet NICs?

Bad TCP throughput but good UDP (assuming the VPN is using UDP) looks like a hardware offloading issue IMO.

Symlink

@stephenw10 I'm not passing any hardware through to this vm. The vpn is using UDP. I'm almost positive it's a setting or config in pfSense as a new basic install with no other differences gives the expected download and upload speeds.

stephenw10

What NIC type is the VM using?

Symlink

@stephenw10 Realtek RTL8139

stephenw10

Hmm, it doesn't have a virtualised NIC type? I was expecting xn(4).

Symlink

@stephenw10

xn0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: WAN
	options=3<RXCSUM,TXCSUM>
	ether ea:b2:72:44:1c:3e
	media: Ethernet manual
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
xn1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: LAN
	options=3<RXCSUM,TXCSUM>
	ether 72:c8:eb:c2:6d:68
	media: Ethernet manual
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
xn2: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: FIBER
	options=3<RXCSUM,TXCSUM>
	ether d6:0c:d4:16:41:17
	media: Ethernet manual
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Ip address lines redacted.

I switched the nic type to e1000 and I'm getting 450 down/300 up currently on fiber. I disabled all nic offloading for those 3 interfaces with no changes as well.

xe vif-param-set uuid=7703da34-7578-1b0c-5420-738b769bcc36 other-config:ethtool-gso=off
xe vif-param-set uuid=7703da34-7578-1b0c-5420-738b769bcc36 other-config:ethtool-ufo=off
xe vif-param-set uuid=7703da34-7578-1b0c-5420-738b769bcc36 other-config:ethtool-tso=off
xe vif-param-set uuid=7703da34-7578-1b0c-5420-738b769bcc36 other-config:ethtool-sg=off
xe vif-param-set uuid=7703da34-7578-1b0c-5420-738b769bcc36 other-config:ethtool-tx=off
xe vif-param-set uuid=7703da34-7578-1b0c-5420-738b769bcc36 other-config:ethtool-rx=off

Symlink

Changing the nic type and opting out of the CGNAT network address space seems to have corrected whatever issue I was having.