Interface starts losing connection after 2-3 months
-
Hi all,
This has been a bit of a head scratcher for me. I'm using an Intel Xeon D-1718T based system running the latest version of pfSense Plus (24.03). Among others, the system has two SFP28 ports and the underlying controller I believe is the Intel 823-L, which uses the FreeBSD ICE driver:
https://man.freebsd.org/cgi/man.cgi?ice(4)
One of the SFP28 ports is used as the WAN interface and connects to a fiber ONT RJ45 port via a plugged in SFP+ to RJ45 adapter (the ONT has a 10Gbit connection for multi-gig service). This has generally worked fine, but after 2-3 months the interface will lose the connection for about 10-15 seconds. It will then be stable again for a few days, before losing the connection again briefly for 10-15 seconds. The first time this occurred I thought it was a fluke and just rebooted the firewall. Everything was fine again until recently after about 75 days of uptime when the interface dropped again for 10-15 seconds.
I'm curious if anyone has ever seen this behavior before? Could it be a FreeBSD driver issue? I figure if it was an issue with the cabling it would re-occur immediately vs. after about 2-3 months of uptime. I also realize that using SFP+ to RJ45 adapters can be a bit hit or miss, perhaps especially in this case when plugged into a SFP28 cage. For now I have removed the SFP+ to RJ45 adapter and installed a separate SFP+ to RJ45 media converter instead. Plugged into the SFP28 port is now a standard SFP+ transceiver, which connects to the media converter on one side, and the other side of the media converter is connected to the ONT via RJ45 (via standard ethernet cabling). I'm going to monitor this setup and see if it causes same issues after a couple months, or is more stable.
Thanks in advance for any insight / suggestions you might have as to what might be causing these interface drops.
-
What's logged when that happens?
Anything in the sysctl stat output?
-
@stephenw10 said in Interface starts losing connection after 2-3 months:
What's logged when that happens?
Anything in the sysctl stat output?
Hi @stephenw10 - unfortunately I didn't save out the logs after the last time this occurred. How far back is log history stored in pfSense? Maybe I can still retrieve it. Thanks again.
-
Depends on the log settings and the amount that you have being logged. But there's a good chance the archived system log files still exist. I have logs reaching a lot further back than that.
-
@stephenw10 said in Interface starts losing connection after 2-3 months:
Depends on the log settings and the amount that you have being logged. But there's a good chance the archived system log files still exist. I have logs reaching a lot further back than that.
Hi @stephenw10 - apologies for the delay in responding. I just rebooted the system in the process of upgrading from 24.03 to 24.11, but will keep eye on it to see if the issue reoccurs. So far so good. Thanks again for your help.
-
Thought I would follow up on this topic: I ended up deciding to add a 10Gbit RJ45 card to the firewall so that the connection between ONT and firewall is copper to copper. Unfortunately, I still saw the disconnection issue described in the original post, which essentially rules out that it was a problem with SFP28 cages and/or transceiver. My next thought was that maybe it's issue with auto-negotiation of the link speed between firewall and ONT. I decided to force the speed on the firewall side to 10Gbit vs. Default and so far so good (no further connection drops since making the change).
