Connection Speeds Drop Randomly

swinn

I've been using pfSense for many years without any issues but recently I'm having an issue I cannot seem to resolve. I switched from cable Internet (400/2) to fiber (symmetrical gig). I expected to just unplug the ethernet from the cable modem and plug in the ethernet from the ONT and it would be fine. Most of the time, it is fine. I get 940Mbps up and down. The issue is that seemingly randomly, my speeds will drop to 300-500Mbps down and 50-70Mbps up.

My original setup was running pfSense virtualized under Hyper-V. I had been doing this for years. The Hyper-V host has an Intel 10Gbps SFP+ interface for the LAN and an Intel 1Gbps interface for WAN. I have several VLANs trunked on the LAN interface and pfSense does all the routing. This is when I first noticed the issue.

I could not get the issue to happen by bypassing pfSense and connecting a computer directly to the ONT. Speeds were always ~940 up and down. This points towards my network.

I moved pfSense to a Dell R220 with Intel (ix) 10Gbps SPF+ interfaces for LAN and the built-in Broadcom (bge) interface for WAN. The issue persisted.

Things I have tried:

Replaced the ONT.
Moved the WAN connection to the second ix interface.
Put a switch in between the ONT and WAN interface. I did some packet captures and see tons of DUP ACK.
Disconnecting the LAN and connecting a computer directly to an interface on pfSense (so only one client computer directly attached to the router).
Tried various settings in loader.conf and system tunables.
Installed pfSense from scratch and only setup the interfaces and VLANs, so no packages or other settings could affect it.
Tried reloading filters or clearing states to see if it would be fixed at that moment.

Things I have noticed:

While the issue is happening, LAN routing is not affected. I can route at 9Gbps.
The most curious thing is when the issue is occurring, if I pull the ethernet cable from the WAN interface and plug it back in, the issue is gone (until it happens again). I can do this in the middle of a speed test and watch it go from 70Mbps to 900Mbps.

I'm looking for any help of what to do next.

verizu

I would suggest as a first move installing some monitoring tools with historical data, for example telegraf (influxdb + grafana on separate machine) will show dropped packets, pf metrics, system ....