Gateway monitor down
-
@gertjan said in Gateway monitor down:
@kevindd992002 said in Gateway monitor down:
I'm seeing DHCPNAK
Looks like the upstream DHCP server send a DHCPNAK. This was with your ISP device in bridge mode ? So it was the DHCP server from the ISP ... ?
" I'm seeing DHCPNAK" => The ISP is seeing your DHCPDISCOVERS and didn't expect them ? It tells the pfSEnse DHCP client 'to shut up'.Correct. That is the upstream DHCP server from the ISP because it is in bridge mode.
You mean the ISP is seeing DHCPREQUESTs and not DHCPDISCOVERs, right? I'm seeing multiple DHCPREQUESTs that aren't being answered.
@kevindd992002 said in Gateway monitor down:
seeing frequent "renewal in 1800 seconds" messages? Does that mean the DHCP lease is just every 30 minutes?
This part :
Dec 8 12:11:44 dhclient 26504 bound to {WAN IP} -- renewal in 1800 seconds. Dec 8 12:11:43 dhclient 24813 Creating resolv.conf Dec 8 12:11:43 dhclient 24678 RENEW Dec 8 12:11:43 dhclient 26504 unknown dhcp option value 0x52 Dec 8 12:11:43 dhclient 26504 DHCPACK from {DHCP server/WAN interface gateway} Dec 8 12:11:43 dhclient 26504 DHCPREQUEST on igb0 to {DHCP server/WAN interface gateway} port 67 Dec 8 11:45:06 dhcpd 21997 DHCPACK on 192.168.20.253 to 0a:d6:94:12:78:5c via igb1 Dec 8 11:45:06 dhcpd 21997 DHCPREQUEST for 192.168.20.253 from 0a:d6:94:12:78:5c via igb1 Dec 8 11:45:06 dhcpd 21997 reuse_lease: lease age 20117 (secs) under 25% threshold, reply with unaltered, existing lease for 192.168.20.253You're getting a TFC1918 = 192.168.20.253 is an IP from your ISP device in router mode ....
Your ISP would not give ypu a RFC1918 it the device was in bridge mode. It shouldn't.Please ignore the dhcpd events on the 1st set of logs that I posted today. Those are for pfsense acting as the DHCP server to my "LAN clients" which is why you see RFC1918 addresses in the logs. This is why I posted a 2nd set of logs that only shows the dhclient entries which is what's important for my WAN DHCP lease renewal issue.
Btw : the dhcp (pfSense) client receives a option 0x52 = 82 decimal = "Relay Agent Information" and the client doesn't know what that means / isn't aware of that option / doesn't know what to do with it.
Yes, I saw that too. So that means that the ISP DHCP server is using DHCP relay which is why this whole issue could be related to this, no?
You saw several
dhcpleases xxxxxSending HUP signal to dns daemoGo to the Services > DNS Resolver > General Settings and un check "DHCP Registration". You deal with that issue later (it's a very known : evey time a (any) device on your LAN asks for a new IP by DHCO, or renews, the Resolver gets restarted. If you have many devices, or a device that likes to ask a new IP every xx seconds, the resolver (unbound) passes more time with restating as doing its actual job = handling your DNS).
Just un check "DHCP Registration" and have this stopped.I am totally aware of this and this only affects the DHCP server service (dhcpd) in pfsense, not the dhclient. I don't really care if the the DHCP server restarts every now and then because of DHCP registrations. I accept the fact that it does this.
-
I've edited - add another part ot my reply above.
-
@gertjan said in Gateway monitor down:
edit :
What about telling the dhcp pfSense client to wait for a minute or two when a WAN UP/DOWN event is detected ?
Check this one :
look up the meaning of the several time out values here https://www.freebsd.org/cgi/man.cgi?query=dhclient.conf&sektion=5&n=1
I'm looking into this too but I don't want to be breaking any RFC rules that aren't supposed to be broken. Not sure if the problem is in the client side or the ISP DHCP server side.
You could also enter the IP (RFC1918) of your ISP device to be rejected :
Read :
To have the DHCP client reject offers from specific DHCP servers, enter their IP addresses here (separate multiple entries with a comma). This is useful for rejecting leases from cable modems that offer private IP addresses when they lose upstream sync.So :
if you don't want to accept an IP from your ISP device - it's internal DHCP server (when it is in bridge mode).
The IP of my ISP's DHCP server is a public IP which is expected. So not sure if this has some effect.
-
@kevindd992002 said in Gateway monitor down:
I don't really care if the the DHCP server restarts every now and then because of DHCP registrations. I accept the fact that it does this.
No, no the pfSense DHCP server. It's far worse.
When the pfSense DHCP server gave an IP lease to a LAN based device, it will :Sending HUP signal to dns daemonThis means : it will restart unbound, the DNS resolver.
Ok if it does so ones in a while.
Not every minute or so, as you will be loosing your DNS cache every time it restarts.
The DNS functionality on your LAN will be not available during restart.
And that's bad .... -
@gertjan said in Gateway monitor down:
@kevindd992002 said in Gateway monitor down:
I don't really care if the the DHCP server restarts every now and then because of DHCP registrations. I accept the fact that it does this.
No, no the pfSense DHCP server. It's far worse.
When the pfSense DHCP server gave an IP lease to a LAN based device, it will :Sending HUP signal to dns daemonThis means : it will restart unbound, the DNS resolver.
Ok if it does so ones in a while.
Not every minute or so, as you will be loosing your DNS cache every time it restarts.
The DNS functionality on your LAN will be not available during restart.
And that's bad ....Ohhh, you're right. Yeah, then I should probably disable that if it deletes the cache every single time :) Even though I have my own DNS server (adguard home), it is still pointed to pfsense's unbound for faster resolution.
-
@kevindd992002 said in Gateway monitor down:
Also, why am I seeing frequent "renewal in 1800 seconds" messages? Does that mean the DHCP lease is just every 30 minutes?
The dhcp client will typically renew at half the lease time to prevent the lease ever expiring. So it looks like the ISP is handing you a 1 hour lease.
Steve
-
@stephenw10 said in Gateway monitor down:
@kevindd992002 said in Gateway monitor down:
Also, why am I seeing frequent "renewal in 1800 seconds" messages? Does that mean the DHCP lease is just every 30 minutes?
The dhcp client will typically renew at half the lease time to prevent the lease ever expiring. So it looks like the ISP is handing you a 1 hour lease.
Steve
Right. That makes sense. Checking the logs again, it looks like most of the times the lease get renewed properly but there are random times that the client just sends out a Unicast DHCPREQUEST multiple times until it gets a DHCPNAK like I showed above. Do you think this is an ISP DHCP server issue? If so, do you have any tips on what I should tell them?
-
It happened again just this very moment and the logs show the exact same thing.
-
Does it eventually switch back to broadcast and then get a reply from a different server?
I have seen ISPs with badly configured redundant DHCP servers that can behave like that.
You can set the WAN dhcp client to requests a different lease time. The server can just ignore that though.
Steve
-
@stephenw10 said in Gateway monitor down:
Does it eventually switch back to broadcast and then get a reply from a different server?
I have seen ISPs with badly configured redundant DHCP servers that can behave like that.
You can set the WAN dhcp client to requests a different lease time. The server can just ignore that though.
Steve
No, it doesn't. Though I'm reading that it should do broadcast after several tries. Not sure if there has been any update to pfsense about this causing the behavior to change. And from the logs, it's always talking to the same DHCP server IP.
What it does is that the client sends multiple (no exact number) unicast DHCPREQUESTs to the ISP DHCP server and the server responds with a DHCPNAK eventually. As expected, when the client receives a NAK, it starts the whole DORA process. At this point, the DISCOVER will be a broadcast and it gets completed until the clients gets an ACK from the server.
But then, like I said, the usual unicast process works "most of the time". So that tells me that it's not a case of unicast or broadcast but I don't know what's causing it.
And yes, changing the lease time would probably be ignored by the server. I think it's one of the most basic security mechanisms of DHCP.
-
The DHCP server may have limits set that it ignores requests outside of but it may well accept requests inside that. I have seen similar situations where the DHCP server was handing out a lease that was far too long resolved by doing that. That doesn't fit what you're seeing here exactly though.
Steve
-
@stephenw10 said in Gateway monitor down:
The DHCP server may have limits set that it ignores requests outside of but it may well accept requests inside that. I have seen similar situations where the DHCP server was handing out a lease that was far too long resolved by doing that. That doesn't fit what you're seeing here exactly though.
Steve
I see. But what will increasing the lease time do though?
-
For example it may be something rejecting too frequent requests. Though that seems unlikely at 1800s.
-
So it does look like that they fixed the DHCP lease issue. However, I'm still having issues with gateway monitoring and ping latency in general.
Look how crappy my gateway montioring graph is. It started increasing in latency since Dec. 16:
When I try pinging even just the WAN gateway (a public router IP on my ISP's network), it's very unstable too. It's very hard to explain this to the ISP support agents because they simply don't understand.
-
Looks like the graph didn't upload.
-
@stephenw10 said in Gateway monitor down:
Looks like the graph didn't upload.
Sorry. I edited my post above to fix this. Here's another tracert result that also shows the problem:
-
Here's the latency problem that is evident even when I have my router ping the WAN interface IP (first hop from my router):
PING 112.205.32.1 (112.205.32.1) from {my router's WAN interface IP}: 56 data bytes 64 bytes from 112.205.32.1: icmp_seq=0 ttl=255 time=1242.815 ms 64 bytes from 112.205.32.1: icmp_seq=1 ttl=255 time=1310.078 ms 64 bytes from 112.205.32.1: icmp_seq=2 ttl=255 time=1457.912 ms 64 bytes from 112.205.32.1: icmp_seq=3 ttl=255 time=473.654 ms 64 bytes from 112.205.32.1: icmp_seq=4 ttl=255 time=2.773 ms 64 bytes from 112.205.32.1: icmp_seq=5 ttl=255 time=2.146 ms 64 bytes from 112.205.32.1: icmp_seq=6 ttl=255 time=1.822 ms 64 bytes from 112.205.32.1: icmp_seq=7 ttl=255 time=4.379 ms 64 bytes from 112.205.32.1: icmp_seq=8 ttl=255 time=455.918 ms 64 bytes from 112.205.32.1: icmp_seq=9 ttl=255 time=424.541 ms --- 112.205.32.1 ping statistics --- 10 packets transmitted, 10 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 1.822/537.604/1457.912/557.557 msThat's a a ping from the same WAN subnet and should be just less than 1ms or maybe even 2/3ms.
-
Mmm, that's catastrophically bad!
If that's the first hop, and it's not just the gateway not responding to ping, there's not much that pfSense can do about it. I assume you were not saturating the link at that time? -
@stephenw10 said in Gateway monitor down:
Mmm, that's catastrophically bad!
If that's the first hop, and it's not just the gateway not responding to ping, there's not much that pfSense can do about it. I assume you were not saturating the link at that time?Exactly! Yes, the gateway/first hop does respond respond to ping but the latency is very unstable as you see in my last post. No saturation at all.
That's also why I'm very convinced that it's an ISP issue. I just don't know how to dumb it down for them to understand. They all base their "knowledge" on the results of www.speedtest.net. When I do a test there, I do see a 2ms latency which I think is just one of those ping results that's normal. But what's more important is the average of continous ping latency results which is what pfsense does.
-
The gateway itself does not have to respond to ping so results against it directly are not necessarily indicative of an issue.
I would try running smokeping or MTR against a number of external targets and see how that varies.
Steve
