Packet Loss on 2.4.5p1 Similar to what was Reported on 2.4.5

[etep15]

I'm on 2.4.5p1 and have been having issues with packet loss for months. I believe it has been ever since I upgraded to 2.4.5. All the forum posts I read point to 2.4.5 being an issue. People talking about firewall maximum table entries, disabling bogons on the interfaces, and disabling multi processors. Nothing is working and of course finding 2.4.4p3 is next to impossible now.

What I have found that can trigger the packet loss for sure is a large amount of uploads. So when I conduct a speed test I notice that ping times increase briefly but nothing too bad during the download. It's during the upload phase that things go sideways. When the upload on the speed-test starts I get about 5-10 dropped packets then things return to normal. That's been the only way I've been able to replicate the issue. It does suffer from the intermittent, mass packet loss, that seems to be identical to the issues that 2.4.5 had that people were talking about. But those are hard to replicate. They just happen and they are bad (100s of dropped packets).

I'm dropping streams, voip calls, and timing out downloads. My internet is unusable.

And just to be complete, I have tested running a PC directly connected to my ISP's cable modem. When I get the mass packet drops through pfsense, I'm not seeing any drops on the PC directly connected. So it's pfsense for sure. I've spent weeks with my ISP getting my internet connection tip top. There are issues but it's no longer them now. It's pretty clear its my pfsense.

Also the only packages I have installed is FreeRadius.

I have PFSense installed on a Qotom Q190G4 box.

I do have a number of VLANS etc but I have tried disabling all kinds of things. No luck. I've also disabled the eee (energy savings?) on all 4 nics. Still no love.

Here's a graph of my WAN quality...

You'll notice that things seem fine until I got up today and got online.

I assume posting my config xml is a bad thing to do. Let me know if I can grab snippets of that to help.

[Cool_Corona]

Try setting the MTU manually to 1440 and MSS to 1400 on all interfaces and reboot pfsense.

[stephenw10]

Seeing some packet loss when you have saturated your upload bandwidth is not unusual.

If you are seeing the issue the existed on 2.4.5 simply start a ping then reload the ruleset. (Status > Filter Reload). You would see huge latency while it loads. But you will not be seeing that exact thing as it doesn't exist in 2.4.5p1. Unless you have managed to still be running the 2.4.5 kernel or something similar.

Where is the gateway monitoring? Does changing the target make any difference?

Steve

[etep15]

@Cool_Corona Tried the MTU/MSS change.. No love there.

@stephenw10 that makes sense. However I'm just using that to try to "recreate the issue" perhaps I'm not. Otherwise I just have to sit there and wait for it to happen again. It seems to be when I just am doing typical internet usage. Such as VOIP calls (I work remotely these days) or any other kind of streaming etc.

I have a ping setup with a 0.1s timeout so I can see any hiccups that a standard ping may miss. However, in both cases I will get 10-120 second long packet drops. Just nothing gets through. I have no idea what triggers it. I've also seen this while I did the testing where I had a laptop hooked up directly to my cable modem. So I could see the packet loss going through pfsense and none with the direct connection (2 different public IPs).

I did try the filter reload. No issues there so you are right, not the same. But the symptoms definitely seem similar. I do have my "Firewall Maximum Table Entries" set to 65,000 as someone suggested back during the 2.4.5 issue. But that doesn't help. I should probably set that back to default.

I also have dev.igb.0.eee_disabled set to 1 on all my igb devices (igb 0 through 3). No love there either.

@stephenw10 You asked "Where is the gateway monitor?" I assume you mean what IP am I monitoring? I'm monitoring my ISP's gateway. I've also tried other IPs such as 4.4.4.4 and 8.8.4.4. I'm pinging my ISP's gateway from one of my internal machines as mentioned above at 0.1s.

I've also tried setting up a second connection to my ISP through pfsense and that gives a different gateway. Same issue monitoring that new gateway.

Finally, I've tried using a whole different ISP. Same issue. I'm fairly certain I've eliminated my ISP(s) as the issue. Even though I have gotten them to clean up all their stuff through multiple visits.

BTW, thank you for the responses. :) Appreciated.

[SteveITS]

@etep15 said in Packet Loss on 2.4.5p1 Similar to what was Reported on 2.4.5:

I've also seen this while I did the testing where I had a laptop hooked up directly to my cable modem. So I could see the packet loss going through pfsense and none with the direct connection

Just so I'm clear, if you ping the pfSense LAN IP you get no loss, and ping the ISP router LAN IP you get loss? There was a thread here within the last week or two where after some back and forth they changed a patch cable or actually I think it was a NIC, and the problems went away.

@etep15 said in Packet Loss on 2.4.5p1 Similar to what was Reported on 2.4.5:

I do have my "Firewall Maximum Table Entries" set to 65,000 as someone suggested back during the 2.4.5 issue

That issue was due to the processing of large tables, for instance when using pfBlocker to generate country lists or other large aliases.

[etep15]

@teamits Yes you are correct. My ping to the LAN port is fine. No drops while I'm getting drops pinging out to the WAN through the pfsense box.

(unless of course I do certain updates to the UI that can cause the UI to hang for a bit and timeout. Then I eventually can reconnect to it. Separate issue that I'm not concerned about.)

I originally had everything going through port 0 and had a VLAN setup to pass my WAN through it from my cisco 3750 switch. But I've since gone straight from my cable modem into port for the WAN and using a different port on the LAN. Same issue. I've tried different cables, and different ports on the QOTOM server that its running on. No luck there.

I will say I only changed the cable once. I'm now using a cable that I know works as that is what I used to test from my PC straight to the cable modem. I've put that now in place and will see if I get the sporadic massive packet loss. Will report back if I see anything.

Btw I'm still running the MTU and MSS settings mentioned in the first reply by @Cool_Corona .

[etep15]

@teamits Unfortunately, the changing of the cable did not help. I just had a massive blast of packet loss that lasted about 20 seconds.

[stephenw10]

In 2.4.5p1 you can leave the max tables value at the default, which is now 400000.

120seconds with no traffic is something significant. I would expect to see something logged. Or at least something in the monitoring graphs, maybe for CPU usage.

Try a packet capture on WAN when that's happening. Is anything leaving? Anything coming back?

Steve