Auto-renew DHCP after outage

e4ch

The LAN-side of the firewall is quite complicated (double NAT with additional router, local DNS, etc.), but that is not the problem. The pfSense is acting just as a firewall.

Yes, the WAN is connected to public Internet and would have an IPv4 Address when it's working. When it's not working, there is no IPv4 address on the WAN Interface. So this is not related to DNS or anything on the LAN-side. When it's not working, I still have the correct WAN Gateway IPv4 address though and that doesn't change after a Release+Renew.

You're saying that it'll try to renew periodically. Maybe this period is just too long? What's the default value? I would expect that this doesn't take several minutes. I think I even waited for half an hour or so, but maybe it does resolve after an hour or more. I think if there's no IPv4 address, there's no need to wait for many minutes; it could immediately request a new one. But maybe at that point my modem isn't up yet (after the power outage) and instead of trying again every few seconds, it just waits progressively longer, causing the issue.

Any further ideas?

JKnott

You're saying that it'll try to renew periodically. Maybe this period is just too long? What's the default value?

There is no default value. It depends on the lease time provided by the DHCP server. Typically, the client tries to renew the lease partway (1/2 -2/3) the lease time and will retry if it fails. Try capturing the DHCP traffic on the WAN port. You can use the pfSense Packet Capture to do that. You'll want to download the capture file and open it with Wireshark, so you can see all the relevant info.

e4ch

Ah, if that is what you meant, then that's not good. I get a new IP every 20 days I think, so if we're talking about that timeout, I certainly don't want to wait two weeks after a power outage just to automatically get a new IP.

But if the WAN IP is not set (= "n/a") I would assume that pfSense doesn't have to wait for such a timeout.

Grimson

@e4ch:

But if the WAN IP is not set (= "n/a") I would assume that pfSense doesn't have to wait for such a timeout.

Check the advanced settings for the DHCP client on your WAN interface:

dhcp-advanced.png_thumb

e4ch

Thanks, this seems helpful. I have never configured anything there. All the timing fields were empty and the Presets selection was on "Saved Cfg", probably meaning that nothing is configured. I can see that the defaults for FreeBSD and pfSense are quite different. The values are in seconds? The pfSense values are much smaller, but the values for Reboot and Backoff cutoff are not defined. I think I'll try the FreeBSD ones as you suggested. I'll report back after the next power outage in a few days then, as I don't want to try this now.

e4ch

Actually it still doesn't work. If I power down the Netgate SG-4860 pfSense (version 2.4.3-RELEASE) and also the cable modem by cutting off power and re-connecting the power, it still doesn't work. Please see attached screenshots. The DHCP advanced config is now set to standard FreeBSD values. On the status page for the interface, it shows Status and DHCP both as "up", but no packets get sent in or out and no DNS is accessible. Only on the dashboard the problem is visible: for WAN Interface, the IP is listed as "n/a". I can fix this on the Interface page by clicking the Release button and then the Renew button. I want that after a power outage pfSense reconnects automatically.

pfsense_config.png_thumb

pfsense_problem_status_interfaces2.png_thumb
pfsense_problem_status_dashboard_censored2.png_thumb

e4ch

This actually not only happens after a power outage, but also when I disconnect the antenna cable from the modem/router for a few minutes. This means the modem gets a new external IP afterwards, but pfSense doesn't know anything about it and it happily continues to try to send from the wrong old IP and that gets ignored by the modem. Maybe I need some external connection test and if failed automatically do a release/renew or something like that (if pfSense cannot handle this).

JKnott

^^^^
Normally, DHCP should run whenever the cable to the router is connected, but pfSense has no way to tell if a cable beyond it had disconnected/reconnected. If the modem is your DHCP server, I'd suspect a problem with it. As always, the way to find a problem is isolating where it happens. Can you try connecting a computer directly to the modem and see if it fails when you disconnect/reconnect the modem antenna? If the same problem happens, then the fault is with the modem. Also, as long as the modem gets the same subnet, it shouldn't be a problem, if it's address changes.

jahonix

Your gateway monitoring isn't working correctly.
From: System | Routing | Gateways configure a Monitor-IP so the system detects it's not online.

JKnott

@jahonix:

Your gateway monitoring isn't working correctly.
From: System | Routing | Gateways configure a Monitor-IP so the system detects it's not online.

On my system, gateway monitoring is disabled, but I don't have a problem with things failing. My cable modem is configured in bridge mode and pfSense has no problem restoring connection after a power failure, as happened last Friday. As discussed earlier, a device "owns" a DHCP address for the duration of the lease time. That means that even if the modem fails, for any reason, and then comes back up, the DHCP address should still be valid and work. Something else is going on with the OP, which is why I asked about the subnet after the failure. We need more info from the OP, including packet captures to really know what's happening.

e4ch

The modem is the DHCP server (I'd assume). That modem is provided by the Internet provider (UPC Cablecom Switzerland) and I cannot change it or use another model. It has tons of functionality, like IP telephony, WLAN, guest WLAN and many other things, but I've disabled all that and switched it to a modem/router-only mode, where nothing can be configured and WLAN is turned off.
I've tried with a standalone notebook connected only to that modem and the same problem occurs. But as I cannot change it and the problem is fixed after a release/renew, I'd like to configure that somehow.
It's worth mentioning that after a release/renew, when it works again, I often get the same IP address. But before the release/renew I couldn't ping 8.8.8.8 and afterwards I do get an answer, so there's definitely something broken that gets fixes with a release/renew.
And pfSense does recognize this problem somehow as the dashboard shows "n/a" in the IP on the WAN connection. So I'd like to automate that somehow; whenever there's a "n/a" on the dashboard for WAN, a release/renew should get issued.
Any idea how to achive this?
I might also complain to the Internet provider, but I suspect that won't get me anywhere.
Regarding the Monitoring, there's a default entry in System/Routing/Gateways, with Name=WAN_DHCP (default) with Monitor IP=external IP. When I click on Edit, I get Gateway=dynamic (Gateway IP address), the Default Gateway checkbox is checked. The monitor IP does respond to ping.

JKnott

I don't see anything that can be done to restore a connection when the failure is detected. One thing you could do is write a shell script that pings the gateway address and if it fails do something like restart the dhcp client.

ernieb44

I've got the same issue as OP. My ISP is Ziggo (which basically is UPC..) and cable modem in bridge mode.
I guess I have to start looking for a shell script which renews the IP when a ping to an outside address fails..

@OP, what kind of cable modem do you have? Perhaps it's related to one specific type..

Derelict

Another case of an ISP device that should know it has to do something like down/up the downstream link on an upstream address change.

e4ch

@Derelict:

Another case of an ISP device that should know it has to do something like down/up the downstream link on an upstream address change.

Well, in the case of the antenna cable disconnect, after a DHCP release/renew, I get the same IP as before. So it's not only the address change.

e4ch

@ernieb44:

I've got the same issue as OP. My ISP is Ziggo (which basically is UPC..) and cable modem in bridge mode.
I guess I have to start looking for a shell script which renews the IP when a ping to an outside address fails..

@OP, what kind of cable modem do you have? Perhaps it's related to one specific type..

UPC Cablecom Switzerland uses the so called "Connect Box". Googling for it shows that it seems to be used in other countries as well, including Ziggo.
This is the modem in question:
https://www.broadbandtvnews.com/2015/11/12/upc-cablecom-rolls-out-libertys-new-wi-fi-gateway/
I got the modem exchanged in the meantime, but of course it didn't fix the problem.
Please note that the box has many features like WLAN, routing, etc., but it's configured in modem-only mode (no settings at all in this mode).
If anyone wrote a script, please provide it here (including installation instructions if possible).

ernieb44

I have a different modem, the Technicolor TC7200. Then it must be our ISP which has it's own view on implementing technology instead of using worldwide standards.

In a different forum post I found someone who has created a script which pings certain external IP addresses and if all of them fails it resets the wan interface, then it pings again and when that fails it initiates a reboot
https://forum.pfsense.org/index.php?topic=51786.0

I will test with it asap..

stan-qaz

@e4ch I had to set pfSense to reject the DHCP info offered by my cable modem when it is not connected to the the Internet, that causes pfSense to wait to do a DHCP request until I'm on-line and getting DHCP information from my ISP instead of the internal modem server.

http://pfsense.home/interfaces.php?if=wan

Reject leases from
192.168.100.1
To have the DHCP client reject offers from specific DHCP servers, enter their IP addresses here (separate multiple entries with a comma). This is useful for rejecting leases from cable modems that offer private IP addresses when they lose upstream sync.

ernieb44

This it! Thanks now it's working fine! Thanks.

@stan-qaz said in Auto-renew DHCP after outage:

@e4ch I had to set pfSense to reject the DHCP info offered by my cable modem when it is not connected to the the Internet, that causes pfSense to wait to do a DHCP request until I'm on-line and getting DHCP information from my ISP instead of the internal modem server.

http://pfsense.home/interfaces.php?if=wan

Reject leases from
192.168.100.1
To have the DHCP client reject offers from specific DHCP servers, enter their IP addresses here (separate multiple entries with a comma). This is useful for rejecting leases from cable modems that offer private IP addresses when they lose upstream sync.

e4ch

I still haven't done anything yet and the problem persists. The reject setting did not solve the issue and I'm a bit reluctant to implement a script that is dependent on external sites - from the sample script, already two of the four sites no longer exist. Also, why should we check external sites, if the problem is somehow clearly detectable? When the dashboard shows "n/a" as WAN IP address, then we have a problem.
Let me show you the situation again:
(not sure how to attach images here, so let me give a description)
On the Dashboard, in Interfaces, the WAN shows as "up", but as IP shows "n/a" when the problem exists (with some network traffic on the chart). If there's no problem, it shows a correct external IP address instead.
In Interface Status, Status and DHCP both show as "up" (both in working / not working case). The IPv4 Address only shows in the working case. When it's not working it's not listed. The DNS servers there are shown as 127.0.0.1 and the four servers from my ISP in the working case. When it's not working because the modem rebooted, it only shows the 127.0.0.1. When it's not working because modem and pfSense rebooted both together, it shows all 5 DNS like in the working case.
I created a log file (I'm on latest version 2.4.4-RELEASE-p2(amd64)).
Here first the system log:

Mar 5 01:04:38  kernel   igb1: link state changed to DOWN  
Mar 5 01:04:38  check_reload_status   Linkup starting igb1  
Mar 5 01:04:39  php-fpm  37326  /rc.linkup: DEVD Ethernet detached event for wan  
Mar 5 01:04:41  php-fpm  37326  /rc.linkup: Shutting down Router Advertisment daemon cleanly  
Mar 5 01:04:41  check_reload_status   Reloading filter  
Mar 5 01:04:53  rc.gateway_alarm  8999  >>> Gateway alarm: WAN_DHCP (Addr:8X.XXX.XX.1 Alarm:1 RTT:10.948ms RTTsd:8.602ms Loss:21%)  
Mar 5 01:04:53  check_reload_status   updating dyndns WAN_DHCP  
Mar 5 01:04:53  check_reload_status   Restarting ipsec tunnels  
Mar 5 01:04:53  check_reload_status   Restarting OpenVPN tunnels/interfaces  
Mar 5 01:04:53  check_reload_status   Reloading filter  
Mar 5 01:04:54  php-fpm  37326  /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'  
Mar 5 01:04:54  php-fpm  37326  /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''  
Mar 5 01:04:54  php-fpm  37326  /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use WAN_DHCP.  
Mar 5 01:04:54  php-fpm  345  /rc.dyndns.update: Dynamic DNS: updatedns() starting  
Mar 5 01:04:54  php-fpm  345  /rc.dyndns.update: Dynamic DNS (): running get_failover_interface for wan. found igb1  
Mar 5 01:04:54  php-fpm  345  /rc.dyndns.update: Dynamic DNS () There was an error trying to determine the public IP for interface - wan (igb1 ).  
<<<Time b>>>
Mar 5 01:06:08  kernel   igb1: link state changed to UP  
Mar 5 01:06:08  check_reload_status   Linkup starting igb1  
Mar 5 01:06:09  php-fpm  20955  /rc.linkup: DEVD Ethernet attached event for wan  
Mar 5 01:06:09  php-fpm  20955  /rc.linkup: HOTPLUG: Configuring interface wan  
<<<Time c>>>
Mar 5 01:07:32  php-fpm  20955  /rc.linkup: calling interface_dhcpv6_configure.  
Mar 5 01:07:32  php-fpm  20955  /rc.linkup: Accept router advertisements on interface igb1  
Mar 5 01:07:32  php-fpm  20955  /rc.linkup: Starting rtsold process  
Mar 5 01:07:34  php-fpm  20955  /rc.linkup: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'  
Mar 5 01:07:34  php-fpm  20955  /rc.linkup: Default gateway setting Interface WAN_DHCP Gateway as default.  
Mar 5 01:07:34  php-fpm  20955  /rc.linkup: Gateway, none 'available' for inet6, use the first one configured. ''  
Mar 5 01:07:34  check_reload_status   Restarting ipsec tunnels  
Mar 5 01:07:35  rtsold  9729  <sendpacket> sendmsg on igb1: Permission denied  
Mar 5 01:07:37  check_reload_status   updating dyndns wan  
Mar 5 01:07:37  check_reload_status   Reloading filter  
Mar 5 01:07:38  php-fpm  344  /rc.dyndns.update: Dynamic DNS: updatedns() starting  
Mar 5 01:07:39  php-fpm  344  /rc.dyndns.update: Dynamic DNS (): running get_failover_interface for wan. found igb1  
Mar 5 01:07:39  php-fpm  344  /rc.dyndns.update: Dynamic DNS () There was an error trying to determine the public IP for interface - wan (igb1 ).  
Mar 5 01:07:39  rtsold  9729  <sendpacket> sendmsg on igb1: Permission denied  
Mar 5 01:07:43  rtsold  9729  <sendpacket> sendmsg on igb1: Permission denied

What I've done:
Starting with fully working pfSense
Turned off the ISP modem (logs until <<<Time b>>>)
Turned on the ISP modem again (logs until <<<Time c>>>); Dashboard showing IP 0.0.0.0 for WAN
After rest of the log: Dashboard now showing "n/a" for WAN

Here the DHCP log for the same period:

Mar 5 01:04:38  dhclient  36543  igb1 link state up -> down  
Mar 5 01:04:39  dhclient  32254  connection closed  
Mar 5 01:04:39  dhclient  32254  exiting.  
<<<Time b>>>
Mar 5 01:06:09  dhclient   PREINIT  
Mar 5 01:06:09  dhclient  73556  DHCPREQUEST on igb1 to 255.255.255.255 port 67  
<<<Time c>>>
Mar 5 01:06:11  dhclient  73556  DHCPREQUEST on igb1 to 255.255.255.255 port 67  
Mar 5 01:06:16  dhclient  73556  DHCPREQUEST on igb1 to 255.255.255.255 port 67  
Mar 5 01:06:29  dhclient  73556  DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2  
Mar 5 01:06:31  dhclient  73556  DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 3  
Mar 5 01:06:34  dhclient  73556  DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 8  
Mar 5 01:06:42  dhclient  73556  DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 13  
Mar 5 01:06:55  dhclient  73556  DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 20  
Mar 5 01:07:15  dhclient  73556  DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 13  
Mar 5 01:07:28  dhclient  73556  DHCPDISCOVER on igb1 to 255.255.255.255 port 67 interval 2  
Mar 5 01:07:30  dhclient  73556  No DHCPOFFERS received.  
Mar 5 01:07:30  dhclient  73556  Trying recorded lease 8X.XXX.XX.X11  
Mar 5 01:07:30  dhclient   TIMEOUT  
Mar 5 01:07:30  dhclient   Starting add_new_address()  
Mar 5 01:07:30  dhclient   ifconfig igb1 inet 8X.XXX.XX.X11 netmask 255.255.240.0 broadcast 255.255.255.255  
Mar 5 01:07:30  dhclient   New IP Address (igb1): 8X.XXX.XX.X11  
Mar 5 01:07:30  dhclient   New Subnet Mask (igb1): 255.255.240.0  
Mar 5 01:07:30  dhclient   New Broadcast Address (igb1): 255.255.255.255  
Mar 5 01:07:30  dhclient   New Routers (igb1): 8X.XXX.XX.1  
Mar 5 01:07:31  dhclient   New Routers (igb1): 8X.XXX.XX.1  
Mar 5 01:07:32  dhclient   Deleting old routes  
Mar 5 01:07:32  dhclient  73556  bound: renewal in 27196 seconds.

Any other ideas than implementing a cron script? And if I have to use a cron script, then how would I need to change it in order to detect this situation without relying on external sites? I'm lacking bash/Linux knowledge here.