default gateway packet loss - no system logs
-
Had a brief internet outage and wanted to correleate some alerts to ensure that things are working properly but o noticed that for my gateways, my default gateway did not generate any system logs. The other configured gateways have.
For example, searching 'gateway alarm'
Thinking it was a configuration error i searched the edit gateway options and its set up the same as the other gateways with the difference being that the one not reporting is set as default.
-
@michmoor
Is your gateway monitoring even configured to check a public IP?
If it only pings the ISP gateway or the local router it would not determine an outage behind these IPs. -
@viragomann yep. it was configured for 1.1.1.1 as the monitor IP.
-
@michmoor
So the outage should have been detected.Is the also nothing in Status > System Logs > System > Gateways?
And is the status of the concerned gateway "online" Status > Gateways?
-
@viragomann Forgot about that section there. Confirmed that there was no alert there.
So to get my understanding here. I have 4x gateways.
Each gateway is monitoriong either an public IP (Quad9) or an IPsec endpoint. All gateways except one detected packet loss today around the time of the outage with the exception of my WAN_DHCP default gateway.Picture of the gateway showing loss as noted below. I can overlay the other gateways showing loss across them at the same time.
-
@michmoor The monitoring averages results over 1 minute by default. With 20% (or, mathematically, a much higher %) loss it's possible enough of those packets succeeded that the average was not over the threshold.
-
@steveits I see what you're saying. Is it fair to say that if it was above the threshold during the brief outage there wouldve been an alert in the logs right? Thats the part i want to be sure about.
-
@michmoor It only logs the "alarm" after it reaches the threshold and triggers. It won't log individual ping failures.
Looking at your log entries, those are 7 hours apart...
-
@steveits Yeah meant to capture another alert.
The alerts under Gateways
Alerts under general
So whats not seen here is an alarm for my default gateway.
-
@michmoor said in default gateway packet loss - no system logs:
So whats not seen here is an alarm for my default gateway.
So if the monitoring is up and working, I would expect that there was not any outage.
-
@viragomann said in default gateway packet loss - no system logs:
So if the monitoring is up and working, I would expect that there was not any outage.
Yeah thats the concern. Clients on the LAN saw a brief interruption and dpinger saw packetloss to other gateways. Definitely outside of the network. Just wondering why the default gateway didnt see anything. Perhaps its a timing thing. Recovery happened which was quick so the gateway threshold for failure was never reached.
-
Seeing packet loss across a VPN but not on the WAN is not that unusual. I'm not seeing a problem here. Those VPN connections saw some packet loss. At different times. If there had been packet loss on the WAN it would have affected both the WAN monitoring and any VPNs using the WAN.
Any clients that had traffic routed across those VPNs would have seen an interruption at that point.Steve
-
@stephenw10 said in default gateway packet loss - no system logs:
Any clients that had traffic routed across those VPNs would have seen an interruption at that point.
MS Teams is the main application that saw the interruption. Not routed over any VPN.
It could have been just a coincidence that all VPN gateways saw a problem. Also likely that there was an upstream routing commonality that saw the drops and application issue.
Either way, i agree with everyone's viewpoint here which is at the end of the day the WAN gateway with a monitor IP thats outside the ISPs infra, did not see a problem as indicated by the logs.EDIT: Ok something @SteveITS mentioned about thresholds..So i checked what the thresholds are for all my gateways as they are all configured the same.
Packet Loss thresholds is set for default 10/20.
Looking at the Interactive Graph for the WAN gateway, the packet loss at the time of the incident was at 3.25%
The VPN gateways have a packet loss of 100% Thats the reason i believe why the VPN gateways alerted and the default gateway didnt.
So yes there was definitely a path issue out on the internet. -
Because those VPN interfaces have gateways defined pfSense sees them as WANs which means when they get marked down a bunch of scripts are run. Depending on exactly what's running on your firewall that can be an expensive operation for the CPU. You might see some interruption on other gateways if so.
-
@stephenw10 I think you misunderstand me
Another viewpoint - Grafana
All gateways saw a loss but i believe the threshold for failure was not met for the WAN_DHCP hence no alarms were generated in the system logs. The loss percentage was too low where as for the VPNs it was at a 100%
-
Ah OK, that could also be true. Depending on where you're monitoring the VPNs could well see greater packet loss at that time.
-
@stephenw10 any reason for more packet loss over a vpn then when not using one ?
-
Longer route, more hops. Generally more chances to lose packets.
1.1.1.1 is an anycast address so you see replies from whatever is logically closest to you.
Steve
