Gateway monitor down
-
@kevindd992002 said in Gateway monitor down:
I don't really care if the the DHCP server restarts every now and then because of DHCP registrations. I accept the fact that it does this.
No, no the pfSense DHCP server. It's far worse.
When the pfSense DHCP server gave an IP lease to a LAN based device, it will :Sending HUP signal to dns daemonThis means : it will restart unbound, the DNS resolver.
Ok if it does so ones in a while.
Not every minute or so, as you will be loosing your DNS cache every time it restarts.
The DNS functionality on your LAN will be not available during restart.
And that's bad .... -
@gertjan said in Gateway monitor down:
@kevindd992002 said in Gateway monitor down:
I don't really care if the the DHCP server restarts every now and then because of DHCP registrations. I accept the fact that it does this.
No, no the pfSense DHCP server. It's far worse.
When the pfSense DHCP server gave an IP lease to a LAN based device, it will :Sending HUP signal to dns daemonThis means : it will restart unbound, the DNS resolver.
Ok if it does so ones in a while.
Not every minute or so, as you will be loosing your DNS cache every time it restarts.
The DNS functionality on your LAN will be not available during restart.
And that's bad ....Ohhh, you're right. Yeah, then I should probably disable that if it deletes the cache every single time :) Even though I have my own DNS server (adguard home), it is still pointed to pfsense's unbound for faster resolution.
-
@kevindd992002 said in Gateway monitor down:
Also, why am I seeing frequent "renewal in 1800 seconds" messages? Does that mean the DHCP lease is just every 30 minutes?
The dhcp client will typically renew at half the lease time to prevent the lease ever expiring. So it looks like the ISP is handing you a 1 hour lease.
Steve
-
@stephenw10 said in Gateway monitor down:
@kevindd992002 said in Gateway monitor down:
Also, why am I seeing frequent "renewal in 1800 seconds" messages? Does that mean the DHCP lease is just every 30 minutes?
The dhcp client will typically renew at half the lease time to prevent the lease ever expiring. So it looks like the ISP is handing you a 1 hour lease.
Steve
Right. That makes sense. Checking the logs again, it looks like most of the times the lease get renewed properly but there are random times that the client just sends out a Unicast DHCPREQUEST multiple times until it gets a DHCPNAK like I showed above. Do you think this is an ISP DHCP server issue? If so, do you have any tips on what I should tell them?
-
It happened again just this very moment and the logs show the exact same thing.
-
Does it eventually switch back to broadcast and then get a reply from a different server?
I have seen ISPs with badly configured redundant DHCP servers that can behave like that.
You can set the WAN dhcp client to requests a different lease time. The server can just ignore that though.
Steve
-
@stephenw10 said in Gateway monitor down:
Does it eventually switch back to broadcast and then get a reply from a different server?
I have seen ISPs with badly configured redundant DHCP servers that can behave like that.
You can set the WAN dhcp client to requests a different lease time. The server can just ignore that though.
Steve
No, it doesn't. Though I'm reading that it should do broadcast after several tries. Not sure if there has been any update to pfsense about this causing the behavior to change. And from the logs, it's always talking to the same DHCP server IP.
What it does is that the client sends multiple (no exact number) unicast DHCPREQUESTs to the ISP DHCP server and the server responds with a DHCPNAK eventually. As expected, when the client receives a NAK, it starts the whole DORA process. At this point, the DISCOVER will be a broadcast and it gets completed until the clients gets an ACK from the server.
But then, like I said, the usual unicast process works "most of the time". So that tells me that it's not a case of unicast or broadcast but I don't know what's causing it.
And yes, changing the lease time would probably be ignored by the server. I think it's one of the most basic security mechanisms of DHCP.
-
The DHCP server may have limits set that it ignores requests outside of but it may well accept requests inside that. I have seen similar situations where the DHCP server was handing out a lease that was far too long resolved by doing that. That doesn't fit what you're seeing here exactly though.
Steve
-
@stephenw10 said in Gateway monitor down:
The DHCP server may have limits set that it ignores requests outside of but it may well accept requests inside that. I have seen similar situations where the DHCP server was handing out a lease that was far too long resolved by doing that. That doesn't fit what you're seeing here exactly though.
Steve
I see. But what will increasing the lease time do though?
-
For example it may be something rejecting too frequent requests. Though that seems unlikely at 1800s.
-
So it does look like that they fixed the DHCP lease issue. However, I'm still having issues with gateway monitoring and ping latency in general.
Look how crappy my gateway montioring graph is. It started increasing in latency since Dec. 16:
When I try pinging even just the WAN gateway (a public router IP on my ISP's network), it's very unstable too. It's very hard to explain this to the ISP support agents because they simply don't understand.
-
Looks like the graph didn't upload.
-
@stephenw10 said in Gateway monitor down:
Looks like the graph didn't upload.
Sorry. I edited my post above to fix this. Here's another tracert result that also shows the problem:
-
Here's the latency problem that is evident even when I have my router ping the WAN interface IP (first hop from my router):
PING 112.205.32.1 (112.205.32.1) from {my router's WAN interface IP}: 56 data bytes 64 bytes from 112.205.32.1: icmp_seq=0 ttl=255 time=1242.815 ms 64 bytes from 112.205.32.1: icmp_seq=1 ttl=255 time=1310.078 ms 64 bytes from 112.205.32.1: icmp_seq=2 ttl=255 time=1457.912 ms 64 bytes from 112.205.32.1: icmp_seq=3 ttl=255 time=473.654 ms 64 bytes from 112.205.32.1: icmp_seq=4 ttl=255 time=2.773 ms 64 bytes from 112.205.32.1: icmp_seq=5 ttl=255 time=2.146 ms 64 bytes from 112.205.32.1: icmp_seq=6 ttl=255 time=1.822 ms 64 bytes from 112.205.32.1: icmp_seq=7 ttl=255 time=4.379 ms 64 bytes from 112.205.32.1: icmp_seq=8 ttl=255 time=455.918 ms 64 bytes from 112.205.32.1: icmp_seq=9 ttl=255 time=424.541 ms --- 112.205.32.1 ping statistics --- 10 packets transmitted, 10 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 1.822/537.604/1457.912/557.557 msThat's a a ping from the same WAN subnet and should be just less than 1ms or maybe even 2/3ms.
-
Mmm, that's catastrophically bad!
If that's the first hop, and it's not just the gateway not responding to ping, there's not much that pfSense can do about it. I assume you were not saturating the link at that time? -
@stephenw10 said in Gateway monitor down:
Mmm, that's catastrophically bad!
If that's the first hop, and it's not just the gateway not responding to ping, there's not much that pfSense can do about it. I assume you were not saturating the link at that time?Exactly! Yes, the gateway/first hop does respond respond to ping but the latency is very unstable as you see in my last post. No saturation at all.
That's also why I'm very convinced that it's an ISP issue. I just don't know how to dumb it down for them to understand. They all base their "knowledge" on the results of www.speedtest.net. When I do a test there, I do see a 2ms latency which I think is just one of those ping results that's normal. But what's more important is the average of continous ping latency results which is what pfsense does.
-
The gateway itself does not have to respond to ping so results against it directly are not necessarily indicative of an issue.
I would try running smokeping or MTR against a number of external targets and see how that varies.
Steve
-
@stephenw10 said in Gateway monitor down:
The gateway itself does not have to respond to ping so results against it directly are not necessarily indicative of an issue.
I would try running smokeping or MTR against a number of external targets and see how that varies.
Steve
Yeah it doesn't need to respond to ping but shouldn't that be a clear cut yes or no scenario? Since we're seeing that it responds to ping, doesn't that tell us that it is setup to respond to ping?
I do run smokeping and it's also seeing the issue:
-
Ouch, yeah that's pretty bad!
The gateway might respond to ping but it may not be prioritised at all. When the gateway is loaded it might drop ping packets or respond with high latency and that's acceptable if traffic through it is passed as expected. That's not happening here though.
-
@stephenw10 said in Gateway monitor down:
Ouch, yeah that's pretty bad!
Right? And now I'm getting intermittent packet losses with my pfsense gateway monitor. I disable gateway monitoring action for now in pfsense but because there are packet losses I can definitely "feel" how sluggish my Internet browsing is. I probably need to ask my neighbors if they experience the same.
The gateway might respond to ping but it may not be prioritised at all. When the gateway is loaded it might drop ping packets or respond with high latency and that's acceptable if traffic through it is passed as expected. That's not happening here though.
Ahh, that makes sense.
