Strange Speed Issue with 5gbit AT&T Fiber Upload

coold8

Hi,

Recently got AT&T 5gbit Fiber installed at house. Been having a really strange issue with my NIC and upload speeds.

Network Setup:

Speedtest.net AT&T speedtest when NIC card is negotiated at 5gbps/5gbps, tests out at 4500-4700mbps downstream, 1000mbps upstream.

Speedtest.net AT&T Speedtest when NIC card is negotiated at 2.5gbps/2.5gbps, tests out at 2500mbps downstream, 2500mbps upstream.

Nic card is a TP Link TX401 PCIe 3.0 x4 10gbit card (updated to most recent AQtion Drive, 3.1.6 and firmware 3.1.121. I have also tried downgraded firmware. Running Windows 11.

Upstream switch is a Netgear MSTX510UP

Computer is connected via 5G Copper Port. PFSense is connected with 10gbit transceiver to Netgear switch. AT&T is connected to a 10gb copper port to ONT.

If I do an iperf directly to pfsense, at first it will be limited to 500-1000mbps. On a repeated test (and not over time, have to end test and start it again), I am able to test the 5gbit upload using iperf. I don't have any 5gbit NICs on other machines to test speed test.

When I run Speedtest-cli on the pfsense box itself, the numbers are just 1890.39mbps/342.86 mbits, so I can't test there.

I don't have any other 5gbit devices.

So far I have tried the following fixes:
Turn off Offloading on both Pfsense and Windows 11.
Update TP-Link Drivers and Firmware
Update Netgear MSTX510UP switch to latest firmware
Bypass the MSTX510UP

Nothing seems to be working to achieve the 5gbit speed symmetrically.

Was thinking about getting another card but there aren't a whole lot of 10gbit cards that also are PCIe x4 (which is the only slot I have open on my Mobo.

Any ideas as to whats causing this strange speed issue?

Thanks!

stephenw10

How are you changing the negotiated link speed? What does it link at by default?

Can you put that NIC in some other device and test using a different OS?

Or maybe boot, say, Ubuntu on the pfSense box and test directly that way?

Steve

ljr

@coold8 are you getting retransmissions on the WAN when uploading? Confirm with a packet capture. I'll bet you just need some egress flow control. Set up a limiter from pfsense on your WAN interface at close to but below 5000mbps, it should help you not flood the ONT on transmit

stephenw10

Good point it could just be a flow control mismatch on the link.

coold8

@stephenw10

Hi guys,

So I am getting these:

10.0.1.1.443 > 10.0.1.13.51477: Flags [.], cksum 0x1628 (incorrect -> 0x045f), seq 849691, ack 4122, win 514, length 0
16:53:09.329017 IP (tos 0x0, ttl 128, id 53498, offset 0, flags [DF], proto TCP (6), length 122)
    10.0.1.13.51477 > 10.0.1.1.443: Flags [P.], cksum 0x8281 (correct), seq 4122:4204, ack 833054, win 8195, length 82
16:53:09.329020 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40, bad cksum 0 (->24c3)!)
    10.0.1.1.443 > 10.0.1.13.51477: Flags [.], cksum 0x1628 (incorrect -> 0x040d), seq 849691, ack 4204, win 514, length 0
16:53:09.329108 IP (tos 0x0, ttl 128, id 53499, offset 0, flags [DF], proto TCP (6), length 40)
    10.0.1.13.51477 > 10.0.1.1.443: Flags [.], cksum 0xe60b (correct), seq 4204, ack 849691, win 8195, length 0
16:53:09.329112 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500, bad cksum 0 (->1f0f)!)
    10.0.1.1.443 > 10.0.1.13.51477: Flags [.], cksum 0x1614 (incorrect -> 0xd470), seq 849691:851151, ack 4204, win 514, length 1460
16:53:09.329236 IP (tos 0x0, ttl 128, id 53500, offset 0, flags [DF], proto TCP (6), length 40)
    10.0.1.13.51477 > 10.0.1.1.443: Flags [.], cksum 0x9be7 (correct), seq 4204, ack 868671, win 8195, length 0
16:53:09.329241 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500, bad cksum 0 (->1f0f)!)
    10.0.1.1.443 > 10.0.1.13.51477: Flags [.], cksum 0x1614 (incorrect -> 0xebcf), seq 868671:870131, ack 4204, win 514, length 1460
16:53:09.329352 IP (tos 0x0, ttl 128, id 53501, offset 0, flags [DF], proto TCP (6), length 40)
    10.0.1.13.51477 > 10.0.1.1.443: Flags [.], cksum 0x5777 (correct), seq 4204, ack 886191, win 8195, length 0
16:53:09.329357 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500, bad cksum 0 (->1f0f)!)
    10.0.1.1.443 > 10.0.1.13.51477: Flags [P.], cksum 0x1614 (incorrect -> 0xfff5), seq 890571:892031, ack 4204, win 514, length 1460
16:53:09.329633 IP (tos 0x0, ttl 128, id 53502, offset 0, flags [DF], proto TCP (6), length 117)
    10.0.1.13.51477 > 10.0.1.1.443: Flags [P.], cksum 0x68f7 (correct), seq 4204:4281, ack 886191, win 8195, length 77
16:53:09.329636 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40, bad cksum 0 (->24c3)!)
    10.0.1.1.443 > 10.0.1.13.51477: Flags [.], cksum 0x1628 (incorrect -> 0x2d52), seq 904584, ack 4281, win 514, length 0
16:53:09.329637 IP (tos 0x0, ttl 128, id 53503, offset 0, flags [DF], proto TCP (6), length 124)
    10.0.1.13.51477 > 10.0.1.1.443: Flags [P.], cksum 0xe337 (correct), seq 4281:4365, ack 886191, win 8195, length 84
16:53:09.329640 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40, bad cksum 0 (->24c3)!)
    10.0.1.1.443 > 10.0.1.13.51477: Flags [.], cksum 0x1628 (incorrect -> 0x2cfe), seq 904584, ack 4365, win 514, length 0
16:53:09.329679 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500, bad cksum 0 (->1f0f)!)
    10.0.1.1.443 > 10.0.1.13.51477: Flags [P.], cksum 0x1614 (incorrect -> 0x9d81), seq 904584:906044, ack 4365, win 514, length 1460
16:53:09.329698 IP (tos 0x0, ttl 128, id 53504, offset 0, flags [DF], proto TCP (6), length 40)
    10.0.1.13.51477 > 10.0.1.1.443: Flags [.], cksum 0x0efd (correct), seq 4365, ack 904584, win 8195, length 0
16:53:09.329724 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500, bad cksum 0 (->1f0f)!)
    10.0.1.1.443 > 10.0.1.13.51477: Flags [P.], cksum 0x1614 (incorrect -> 0x1292), seq 908670:910130, ack 4365, win 514, length 1460
16:53:09.330042 IP (tos 0x0, ttl 128, id 53505, offset 0, flags [DF], proto TCP (6), length 40)

I tried turning the traffic shaper on and setting it to 4650mbps CBQ on WAN/LAN for ATT. When I do that, my speeds decrease on the download to 3800mbps and increase the upload to 2000mbps. I set no shaper rules beyond the 1st step in the shaper multi wan wizard.

Anything else you suggest I try? Seems like you are on to something with the flooding.

ljr

@coold8 well it's a large pipe, you might have to play with the queue lengths and bucket sizes. A limiter by interface (first tab in traffic shaping, ie. ALTQ limiters) acts on egress traffic from the point of view of the firewall, so limiter on WAN would shape your tx from firewall to gateway which is where your upload woes are originating (WAN egress). A limiter on the LAN would limit your download speed from the point of view of your workstation (LAN egress from firewall). That being said, I've heard FreeBSD 13 will bring BBR congestion control out of the box, I've heard that helps with slight mismatches, might be worth waiting for pfsense 2.7 / + 23.01

I would also disable hw offload for checksums, TSO and LSO

stephenw10

I would definitely test enabling (or disabling) flow control at the link level on the NIC.

Some connections absolutely require that.