Periodic packet loss at multiples of 15 minutes

[nazar-pc]

I have 3 WAN interfaces, and only on one of them (the first) pfSense detects periodic packet loss:

The very suspicious thing is that it happens almost exclusively on multiples of 15 minutes: 0/15/30/45 minutes.
The most often interval is 45 minutes between detected packet loss, but sometimes it is 30 or even 15.
Rarely I see something at 5 minute multiple and basicaly nothing outside of that.

This seems to indicate some software issue where something is happening based on a periodic cron job or something like that.
Nothing like that is running on the machine where pfSense is though.

Since this packet loss is detected by pinging gateway every 500ms, I tried to run ping the gateway on both pfSense and my workstation from CLI and after many hours have not detected a single packet lost, no substantial delay either.
I was extremely lucky with 1s interval vs 500ms or there is something substantially different in a way pfSense monitoring does it.

I checked logs and do not see anything particularly suspicious happening at specific times when packet loss is detected, often nothing related to this WAN connection at all.

Any suggestions on how I can debug this further?

[TheNarc]

@nazar-pc If it's on the WAN (and specifically only 1 of 3) I think it's fair to say that it's likely an external issue. I experienced something similar - though not as severe - where for a period of weeks I would get brief and sometimes complete packet loss at the same time exactly every 24 hours. Despite all my efforts, I could not determine an internal cause, and eventually it stopped happening. Since it didn't correlate with any changes I had made, I wrote it off as something that must have been an external issue that was resolved.

That said, the obvious things that occur to me to try would be:

1. Try configuring a different monitor IP for that WAN to ensure that it's not an issue with packet loss on the end of the current monitoring IP.
2. Allow external ICMP to that WAN and ping it from some other remote host to see whether the packet loss is observed that way as well.
3. More of a pain and perhaps not viable, but if you could temporarily connect the offending WAN to a different router or host and run a simple persistent ping to see whether you get the loss there as well.

Of course, unless changing the monitor IP reveals that it was just a problem with whatever the current monitor IP is, none of these will solve the problem, but should at least get some more data points that may be useful in determining where the problem really exists.

[nazar-pc]

@TheNarc I should have thought of trying a different monitoring IP, thanks!
Changed it to 8.8.8.8 and zero loss in 5.5 hours.

Not sure what to make of it though since pining 8.8.8.8 will go through the same gateway that wasn't responding to ICMP packets from time to time
I guess I have something to chat about with my ISP.

[Gertjan]

@nazar-pc said in Periodic packet loss at multiples of 15 minuts:

Not sure what to make of it though since pining 8.8.8.8 will go through the same gateway that wasn't responding to ICMP packets from time to time

That gateway is a router.
It gets a packet from an interface, and copies the content into another interface.
Hardware is purely optimized for doing just this.
Of course, a router has at least two interfaces and probably more, otherwise there is nothing to 'route'.
A router is a guy with a list in his hand on an intersection. For every car that approaches the intersection, our guy knows (he has a list ^^) where which car has to go.

Now, you send a special packet ...euh, sorry : car to the intersection : this car has as an end destination : the guy at the intersection itself. Now, the guy has to sop handling incoming cars, and deal with the car directed to himself. Let's say he has to go park it.
A router takes the packet, hands it over to the local OS, or system that controls the router (the list, and more) and shortly after this, the local OS will inject a ping reply into the same intersection : this new packet (car) that will get send back to the original sender : this is the answer on our original ping request packet.
Routers are made and optimized for packet handling. It doesn't really (very little) act upon them. If it has tro do so, this will take a way local performance.
That's why the ping (= ICMP) protocol has a low priority : if the router has a lot to do, it can even decide to drop the ICMP packet. Replying on a ICMP is optional.

Btw : be ware : this story is also valid for '8.8.8.8'. The very moment 8.8.8.8 gets swamped with DNS requests, it will start to dropping (= not answering) ICMP requested send to it. It's main job is doing DNS, as that is what makes the money, not replying to ICMP.

And yes, the day 8.8.8.8 stops replying to ICMP, this will take down the connection of millions of devices like our pfSense, as they use 8.8.8.8 as there 'end destination'.
So, I'm just saying : maybe using 8.8.8.8 for this ping test isn't a great idea after all ^^

The ping monitoring that pfSense is using is just an indication. If ping replies stop coming back, this doesn't imply that your ISP uplink connection is bad. It means what it means : ICMP packet don't come back - and that's it.
For most of use, this :

is really an option that you could/should consider.
The monitoring still takes place, and all it does is filling up the monitoring stats.

@nazar-pc said in Periodic packet loss at multiples of 15 minuts:

I guess I have something to chat about with my ISP

If you're lucky, you get the same story back.
Maybe they will promise you that they will invest in a more powerful router : they'll ditch the 10k$ router and put a 100k$ version in place ....
But do you really want that ? Who is going to pay for the upgrade ?

[nazar-pc]

@Gertjan It doesn't sound like a great idea to disable monitoring completely. How would pfSense know to stop using the connection and switch to other ISPs when the connection is down?
I only used 8.8.8.8 as a temporary test and I am yet to see it dropping packets under load.

[Gertjan]

@nazar-pc said in Periodic packet loss at multiples of 15 minuts:

It doesn't sound like a great idea to disable monitoring completely. How would pfSense know to stop using the connection and switch to other ISPs when the connection is down?

And you're totally right.
'Something' is always better as 'nothing'.
I just wanted to highlight that monitoring - sending and receiving the pings to one specific device, an upstream gateway is just a (small) part the story.

Btw : pfSense, and you, will know that the upstream link stopped 'flowing'.
TCP traffic is a constant stream of 'data send' going upstream, and 'acknowledge' coming back. When these 'ack's' stop to come back, the upstream network stack will fill up, and when the local buffers are full, the entire system will stall. The end user starts to see "turning icons" on his browser screen. Other communicating software will just stall, and after a time out, show an error.
pfSense might pull the plug : take the WAN interface down for a moment, and put it back on, as this will regenerate a DHCP sequence, or restart a pppoe negotiation upstream. If the uplink was 'broken' like ruptured, pfSense won't be able to do anything.