mixed networks 1G/10G/2.5G

armstrtb

TL;DR I switched LAN from mixed 10G/2.5G to just 2.5G and problem resolved but I'd like to understand more of the reason why that might be. Any thoughts?

I have recently moved and built a new home network with the below gear.
Netgate 6100
Unifi switch flex utility 2.5G POE
Two U7 pro XG access points

WAN 1G/200M on Ethernet
LAN 10G SFP+ to unifi switch
APs both 2.5G Ethernet to unifi switch
Desktop PC 1G to unifi switch

Limiters:
Fq_Codel on the WAN interface.

I found a significant amount of latency issues when playing video games and my wife opening instagram or there streaming services. I don't think it was bufferbloat because tests showed A+ ratings once limiters in place.

Tried switching backbone to just copper:
WAN 1G/200M on Ethernet
LAN 2.5G Ethernet to unifi switch
APs both 2.5G Ethernet to unifi switch
Desktop PC 1G to unifi switch

And the latency spikes have resolved with a few ms increase in overall latency to he desktop PC (no big deal).

My question is why would this have happened? Did I just need to tune things on pfsense to get it to manage packet queues better?

luckman212

@armstrtb Since you mentioned Unifi, have you tried experimenting with the Flow Control setting? I don't use it myself (also have a mixed 1G/2.5G/10G network) but it at least purports to solve the issue you're describing.

b40cce73-e195-4271-9271-24c2898277fa-CleanShot 2026-07-13 at 14.07.27.png

I assume the "Desktop PC" is where you're doing the gaming? So, 1G LAN link into the rest of the network which theoretically has devices capable of using >1G of bandwidth (2.5G WiFi AP, other 2.5G LAN devices?) so yes there could be some congestion there.

Enabling and analyzing the Packet Flows could be helpful.

armstrtb

@luckman212

My IT consultant (Gemini) suggested the same thing which I tried last week but it didn't resolve the issue.

I didn't run packet flow control on pfsense. Maybe that would have resolved. Seems quite complicated to setup over multiple vlans.

My desktop does have a 2.5g port that I just tested this afternoon and the issue did not return either so I'm pretty sure it's on the 10g uplink on SFP+ where the issue was arising. Not sure which hardware is causing the issue.

I'll keep it a 2.5g backbone unless someone has some suggestions for other things to try.

stephenw10

Yeah I would also be looking at flow control given those symptoms. Though I would have expected a 1G WAN to be the limit in the link there. If it was more than 2.5G (or al least more than 1G) then you can get buffers filling where the speed changes. So at the switch for downloads.

armstrtb

@stephenw10

Ill spend some time looking into it and see if there's something simple to do to test if that could resolve it. Might save me a few MS in latency if I can get it to work!

Thanks for the suggestions everyone.

tgl

I've also got a mixed-speed network and had to fight with this. The rule of thumb that has worked for me is to enable flow control on any switch port running at faster than 1G. I have not had to fool with client-machine settings --- they all seemed to enable flow control by default. But my Cisco switches did not.

armstrtb

@tgl did you need to enable anything in pfsense? I looked into the documentation above and it seems to just be good to thanks capture and export not a toggle to enable follow control.

tgl

@armstrtb said in mixed networks 1G/10G/2.5G:

@tgl did you need to enable anything in pfsense?

I did not touch that, no. My router is a Netgate 4200 which does have 2.5G ports, but the ISP connection is 1G (Verizon FiOS). So I'd expect outbound traffic from a 2.5G client to require flow control at the switch -> router connection. The fact that I don't see problems when running a speedtest that way shows that pfSense does enable flow control on the 4200's LAN port.

luckman212

@tgl you can see if flow control is enabled on the dashboard, under interfaces. If you see rxpause,txpause next to your LAN/WAN interface, flow control is on.

You can also check the following kernel parameters controlling flow control from the commandline

sysctl -a | grep fc:

0 indicates flow control disabled
3 means it (should be) enabled

eg from my 6100 it appears to enable FC by default on all interfaces:

# sysctl -a | grep fc:
dev.ix.3.fc: 3
dev.ix.2.fc: 3
dev.ix.1.fc: 3
dev.ix.0.fc: 3
dev.igc.3.fc: 3
dev.igc.2.fc: 3
dev.igc.1.fc: 3
dev.igc.0.fc: 3

There appears to be some debate over whether you can override these per-interface via system tunables or putting something like dev.ix.0.fc=0 in /boot/loader.conf.local so definitely needs to be tested before deciding if it needs to be changed.

This System Tunables docs page says to use

hw.ix.flow_control="0"

... to disable FC globally for all ix Intel NICs.

tgl

@luckman212 Hm, I can't find the display you mention anywhere in the GUI. But sysctl does confirm that the "fc" setting is 3 for all four ports. Again, I'm pretty sure that's the default because my notes don't mention having changed it.

luckman212

@tgl you might not have added the Interfaces widget to your Dashboard. The info is also visible under Status > Interfaces.

MarinSNB

@luckman212

For me for some reason, this has never worked under the System Tunables on several pfSense boxes I have owned including a few Netgate ones.

It only works under the File Editor as explained in the link you included after creating the following file in the boot folder:

/boot/loader.conf.local

and entering the line you mentioned to turn off flow control (for SFP+ interfaces):

hw.ix.flow_control=0

After saving and a reboot, all the tx/rx pauses are gone.

I have also added another line there (see below) - I don't remember where I got this but it was a long time ago and at a time when I was trying to improve the compatibility of the SFP transducer I was using at that time with a Netgate 6100. Not sure if this makes a difference anymore and/or if people still use it.

hw.ix.unsupported_sfp=1

stephenw10

Generally hw.ix.* are loader values and must be set in a loader file. dev.ix.* are sysctls that can be set in System Tunables and changed at run time.

armstrtb

I did some additional troubleshooting and found something quite interesting.

My cable modem was setup in bridge mode. I noticed my wan had rx/tx pauses on the interface so I thought I'd check my modem again. I had mistakenly connected to the 1G port instead of a 2.5G port on the modem. Now the entire backbone to he modem is on 2.5G.

My bufferbloat tests just improved from north of 25ms to sub 17-19ms and A+ ratings on waveform.

This was a pretty different result then I've ever seen for testing this network. I wonder if the flow control had some interaction with the modem. Maybe the 2.5g has direct connection to the coaxial line without switching? Anyways sharing this incase others run into these issues.

My solution was to make entire backbone (modem to PC/wifi) at the same connection speeds. This coupled with traffic shaping on WAN for managing bufferbloat results in A+ waveform tests. Many guides on fqcodel with taildrop are available. My settings are the same as how guides.

Thanks everyone for your help!