WAN2 high latency, slower speeds

avidaviator

Hello all,

Long time reader, first time poster.

I've been tearing my hair out over an issue I am having with my WAN2 connection. I'm hoping someone here can help.

My setup:

Router is an AMD Ryzen 2200G-based system w/ 6GB RAM
Onboard NIC: Realtek (Yes, I know it's bad, but in this case it's not the issue, I promise!)
Add-in NIC: Mellanox ConnectX-3 2x SFP+
WAN 1: On Realtek NIC from Spectrum-provided modem
WAN 2: On Mellanox NIC via SFP+ to RJ45 converter connected to 5G Gateway with T-Mobile Internet (the Gateway is not T-Mobile's, it is the Chester Tech Repairs Cheetah V2). The gateway has a DMZ enabled to my PF box's interface IP so as to reduce any issues surrounding double-NAT as possible
LAN/VLANs: On Mellanox NIC via SFP DAC to TP-Link 8-port agg switch
Client machine: W10 w/ Mellanox NIC using fiber txcvr to same TP-Link agg switch

The issue:

My WAN connection (Charter/Spectrum on Realtek NIC) is currently rock-steady, providing my client machine with a solid 650/35 speedtest (I pay for 600/30) and a decent 20ms unloaded, 60-90ms loaded latency. My WAN2 connection (5G T Mobile) yields an easy 700-800/100 and all-around 35ms latency when I direct-connect a laptop to the gateway and run a speedtest. When I use a firewall rule to force any protocol from my client machine's IP to any destination with WAN2 as the forced gateway, then do a speedtest after the firewall is reloaded, I get a really inconsistent download that starts in the 30s then slowly claws its way up to the low 600s, and an upload that starts at 75 but quickly tapers off to 16. All the while, my unloaded latency is at around 90 with loaded latencies easily hitting 300+.

So to recap, My WAN connection is good and solid. My WAN2 connection is inconsistent in speed and very high latency, despite bypassing the PFSense box yielding a reliably solid connection in both speed and latency.

The only difference between the two WAN configurations as far as PF goes is that WAN2 is set to Static IP whereas WAN is set to DHCP.

Troubleshooting steps I've taken so far:

Upgrade to 2.7.2 from 2.5.0
Disable hardware offloading
Perform iperf test from client(s) to router to ensure I can reach at or near linespeed, result minimum 7Gb of 10Gb linespeed
Swap WAN connections between NICs to rule out SFP+ to RJ45 transceiver issue - result: Issue remained with the gateway, not the NIC
Swap WAN assignments in PF (so WAN2 was now on the Realtek NIC instead of the Mellanox NIC) to rule out config issue - result: same same
Constantly and tirelessly direct-plug my laptop into the 5G gateway to perform tests immediately before or after I do them on the client machine that's routing through PF
Swap cables between 5G gateway and PF with known-good cable
Forced 1000 FDX link between PF and 5G gateway
Run speedtest via speedtest-cli in PF and iperf in PF to public iperf servers with no improvement over client machines

I hope someone can help me with this. I see no logical (or illogical) reason this can't just work the way I would expect it to.

stephenw10

Hmm, curious indeed. You seem to have done most of the tests I would have suggested.

Just to confirm though with the Realtek NIC assigned as WAN2 you still saw the same behaviour when testing it?

Have you tested with WAN2 set as the default gateway? With WAN1 disconnected entirely?

Steve

avidaviator

@stephenw10 Yes, in fact the Realtek NIC is presently assigned as WAN2 as I am doing some more brute-force testing as I like to call it.

To be clear on your second question, are you asking me to go to System -> Routing -> Default Gateway IPv4 to WAN2 then unplug WAN1?

stephenw10

Yes exactly. To rule out any sort of routing issue.

avidaviator

@stephenw10 Finally had a second to do this. I did so and there's no improvement.

avidaviator

Alright, I'm at a complete loss for words.

I just...start plugging stuff in and unplugging it. I was screwing around a bit with the 5G gateway, switching ports on it, unplugging and replugging WANs, and suddenly...it works? I'm going to end up rebooting everything a few times just to be sure that it stays working, but right now, it's all working. I haven't the foggiest clue what may or may not have fixed it...I hadn't made any configuration changes since I did the default gateway and then set it back the way it was...

Solved...for now?

stephenw10

Hmm, odd. Potentially some MAC address lock-in perhaps? We have seen devices that will pass traffic for other devices but only at some basic connection profile. Rebooting the modem would usually clear it though.

avidaviator

@stephenw10 I'm back to the same 'ol problems following a gateway reboot this morning. I'm starting to wonder if I need to reset my PF config?

stephenw10

I doubt this is a config issue. However if you back it up you easily restore it so testing a default config would at least rule that out.