Netgate 4100 slower at 2.5gbe than 1gbe

stephenw10

You might try disabling flow control as a test: sysctl dev.igc.2.fc=0

What pfSense version is that?

matt_m

I’ll give that a try and report back. It’s 24.03, I know I’m behind.

stephenw10

Ah, well I'd definitely try 24.11 then. I'm not aware of anything specific there that might address this but there have been a number of updates/fixes.

matt_m

@stephenw10
Yeah been meaning to do it but with two of us working from home and being a crazy busy time can’t even really afford weekend downtime. But I just kicked it off and will test changing flow control tomorrow.

matt_m

@stephenw10 Unfortunately neither the upgrade or changing flow control made any difference. Still get near 1gb when forced to 1gb, still get in the 300's when forced to 2.5gb.

stephenw10

How are you testing that exactly?

Can you grab a packet capture during the poor throughput?

matt_m

@stephenw10 I use a docker container called Speedtest Tracker which uses Speedtest.net. Running in docker and taking the browser out of the picture it seems to provide pretty consistent results. I'll have to look into capturing packets, should be doable, just have never had the need.

stephenw10

Hmm, the issues could be on the LAN side and are just hidden by limiting the throughput to 1G on the WAN.

Do you see any errors on the LAN interface?

You might try running an iperf3 test to or from pfSense itself and a LAN side client. Or running a a speedtest client test from pfSense directly so the LAN side is not involved.

matt_m

@stephenw10

Might be getting somewhere. I installed the speedtest widget from https://github.com/LeonStraathof/pfsense-speedtest-widget and get above the full provisioned speed as I would typically expect.

So I guess I don't understand why having more available bandwidth on the WAN interface would slow down the LAN? Don't see any errors but the interrupts seems high?

dennypage

@matt_m said in Netgate 4100 slower at 2.5gbe than 1gbe:

So I guess I don't understand why having more available bandwidth on the WAN interface would slow down the LAN?

Lack of flow control on either interface can cause this. I have 2.5Gb link on WAN (provisioned @ 2Gb down), and 10Gb link on the LAN interface. If I turn off flow control on the LAN, speed tests go very, very badly.

matt_m

I did some more testing.

If I set the LAN interface to 1000Gb I get most of the speed back. Tests in the high 800s, still notably lower than normal.
I tried all combinations of flow control on the LAN and WAN interfaces, doesn't seem to have any impact.
I enabled flow control on my Unifi network. At 2.5gb it doesn't seem to have any impact or maybe made things worse.
Flow control enabled everywhere, LAN at 2.5, WAN at 1, it seemed to be staying in the 800s
Back to flow control disabled in Unifi, enabled everywhere in pfSense, LAN at 2.5, WAN at 1, its back to the mid 900's.

Something about the router or my configuration of the router is an issue.

matt_m

One additional test, to rule out the SFP to RJ45 module, I have a small Netgear unmanaged 2.5gb switch sitting unused. I connected the LAN interface and the Proxmox host running my docker server directly to the 2.5gb switch and set the WAN back to 2.5 and didn't do any better than through all the Unifi equipment. I'll also add that I recently upgraded my proxmox host and saw these issues with the old host so its not that.

stephenw10

Yeah check the sysctl outputs for the LAN side NIC.

Try an iperf test between pfSense and a LAN side client, make sure you can actually pass >1G there.

matt_m

@stephenw10 had trouble getting iperf installed—just seemed to stall out. Figure a reboot was in order but couldn’t take down the network since we were working then headed out of town.

I have been intending to upgrade my switches to 2.5gb and picked up a Unifi Pro Max 24 during our trip. That will eliminate the GBIC to RJ45 adapter I've been using. I don’t expect that to resolve the issue but it’s something that’s been on my list for a while and will give me the ability to set up a 2.5gb connection from the router to my Docker machine for testing and will let a couple other machines take advantage of higher speeds. That also means I need to resolve this one way or another.

I’ve stumbled upon 1-2 other posts in dusty corners of the internet that might be describing the same issue but none with any resolution.

stephenw10

What switch do you have on the LAN side currently? Is it linked at 2.5G to the pfSense LAN and 1G to the clients?

Try setting the LAN side link to 1G when the WAN at 2.5G so the bandwidth change is in pfSense not the switch.

If it's in the switch you pretty much need flowcontrol active to avoid buffer overruns.