Erratic Packet Loss Loop (maybe a bug, hard to describe/title)
-
I've been using pfSense for ages now and have come across this odd behavior 3 or 4 times in total but have yet to find a solution or good reason for it.
Symptoms:
- Seemingly randomly, packet loss will start happening in an almost cyclical fashion
- Pinging a host, any host on the internet, will result in 1/20 (roughly) pings timing out, the rest are all exactly inline with normal times
- This happens to all devices, no matter where they are on the LAN
- This happens to all IPs on the public internet and results in TCP breakdown causing websites to sometimes just not load and gaming becomes nearly unplayable
- The only fix I have found is rebooting the firewall
What makes this particularly odd though, and why I think it's maybe a long standing bug, is that I've seen this in the corporate environments I manage with pfSense as well. In total I've seen it on 2 different Netgate 1541's and my Netgate 6100, the symptoms are identical, almost like port flapping but it's too rapid for it to be that.
Network architectures are entirely different in the environments I've seen this behavior too; for example I use SFP adapters at home, but at a work site it's RJ45 straight from the 1541, so it's not like some odd SFP compatibility issue.
I've had this happen on pfSense 25.07 and now 26.03 as well, and possible a version before that though I don't recall the exact version number.
Clearing the state table and other things like that don't make a difference, pfTop will look normal, no excessive states or anything like that. CPU usage will be normal, etc...
I'm not looking for an answer, but more so where to start digging on something like this, it's one of the weirder network issues I've encountered in my career.
-
If it's losing 1 in 20 packets in that state it should be relatively easy to see it in a packet capture. So the first thing I would do is run a pcap to see if those pings are actually leaving. And if they are is there something different about them.
-
@stephenw10 yeah I should've done more troubleshooting this time since it was in my home setup. The other few times it's happened it was in a customers site with an HA cluster so it was basically just get it back up and fast as possible lol.
My annoyance is that it's so incredibly rare it's hard to troubleshoot cuz it needs to happen in a non critical environment and when I actually have time to dig deep on it.
I haven't managed to find any specific changes causing it to happen either, though I slightly suspect a pfBlocker issue but that's just a gut feeling.
Appreciate the response here.
-
How do you know it is directly related to the pfSense node? Could be a hop in your ISP that has some transient routing issues? That's something I have experienced myself, and I finally pinpointed the problem to the ISP box having hardware issues. A long lived traceroute (TCP one) could also help.
-
@e3prom Rebooting pfSense has had a 100% success rate in resolving the issue, which is why I think it's pfSense related.
It's also important to note this has happened across a lot of different hardware and network configurations, the only consistent piece has been pfSense itself.
-
Yes, a faulty SFP or cable can also create packet loss from time to time. I would also check the speed, if you interface is stuck at 10 Mbps Half-duplex or something (both side).
