pfSense unable to recover Internet access after power outage

fbrunken

@madbrain Hi and sorry for the delay. I made this very quick image here... Sorry, didn't take too long preparing it.
Untitled Diagram.png

Important is:

pfSense is your router/firewall into the internet, correct? In your case you have Comcast as ISP.
The IP of your router is dynamic, as of this log, 76.x.x.66, correct? (I would not leave public IPs here...)
I didn't take the time, but important is that you have a lot of HAOS and I suppose other devices in your network. Wired or wireless. Doesn't matter. Also, it doesn't matter if docker, VM instances, or physical devices, as long as they all need to go through pfSense to get to the internet.

So, I am not sure what the logs tell me. My problem was related to recovering from the outage without receiving a correct public IP. The CPE from my ISP, which I use in bridge mode, provided me with an IP that did not work... By forcing a DHCPREQUEST, I fixed the problem.

The log in my case that could clearly show that it wasn't getting fixed was the DHCP one. Yes, the DHCP log provides a log for when pfSense is the server as well as the client.

It is not clear to me, when did the problem finish or why. It started around 00:41, when were you able to get it going again? Do you have a monitor IP or it just uses the GW that comcast gives you, I suppose 76.x.x.1? Also, during the time that the problem was going on, did you check the status of your Gateway? In my dashboard I have Interfaces and Gateways always being shown. This way I can tell what IP address I have on my interface and if pfSense considers the Gateway online or offline.

madbrain

@fbrunken Thanks for your response.

yes, pfSense is my router firewall, and the ISP is Comcast
yes, it's a dynamic IP
yes, all my devices, wired or wireless, need to go through pfSense to reach the Internet

I don't believe I tried forcing a DHCPREQUEST to solve the issue.

I don't have a monitor IP. My WAN interface is just set to DHCP /DHCP6 to use whatever Comcast gives.
I don't recall the interfaces / gateways status at the time, unfortunately. Pretty sure I had to reboot at least the modem twice. Possibly pfSense too, but not certain.

I wish I could recreate the problem, but I couldn't. on tuesday.
However, as of yesterday, I have a second ISP, Sail internet, with a rooftop fixed wireless antenna. Maybe I can reproduce it with their modem.

I'm setting up dual WAN also. A bit complex with dynamic DNS and VPN in the picture. Sail internet only supports IPv4 also, not IPv6. Comcast does both. Hard to failover in that case. Anyway, I'm hoping to get rid of Comcast altogether as I hate their contract shenanigans.

fbrunken

@madbrain Dual WAN Rocks! Throughout my problems, I never really lost internet. A few things to look for/be mindful.

DNS, you can use the ISP DNS or use some free DNS. I use (System/General setup):
a. 1.1.1.1 - 1dot1dot1dot1.cloudflare-dns.com - WAN1
b. 1.0.0.1 - 1dot1dot1dot1.cloudflare-dns.com - WAN2
c. 208.67.222.222 - resolver1.opendns.com - WAN1
d. 208.67.220.220 - resolver2.opendns.com - WAN2
Important is to make sure that routing doesn't get mixed up.
I would put a monitor IP. Use as monitor IP the Comcast/Sail internet DNS IP to monitor each connection. It makes sure that a bit more than the GW is working...
Have 2 different VPNs. One thru Comcast one thru Sail. Don't mix the things... Yes, it can be done differently but I see an advantage in knowing which VPN I am using.
I would only use IPv4, unless you have a reason to use both... disable IPv6.

and yes, recreating the problem I had is not easy. Only saw it happening twice, in over 2 Years of operations.

dmatzen

I just came across this when I started a search for the same issue with my Internet connection after a power failure. It happened again to me when I was not home. I had the wife first try a power cycle of the PfSense firewall, but that did not help. So I had here power cycle the cable modem, and still no internet. One more power cycle of the firewall, and it came up.

My issue is that my ISP's equipment does not have much battery backup. When the power goes out, My UPS keeps things running here for at least 30 minutes, but typically, the internet drops in about 15 minutes.

My best guess is that when the power is restored, my equipment will be back online before the ISPs, and the firewall will be unable to get an address. Only by power cycling the modem can an address be obtained.

Next time this happens when I am home, I will get as much data as possible. I, too, would like to have an automated system in place to recover from this.

madbrain

@dmatzen Which ISP was it, if you don't mind ? I had the issue with Comcast. I have another ISP now, but haven't had another power outage recently.

dmatzen

My ISP is Spectrum and my modem is an Arris TM1602A.

madbrain

@dmatzen I see. It is still cable, though. My cable modem was an XB8, in bridge mode. I believe it was made by Technicolor.

This is a very difficult issue to fix without being able to reproduce it.
I believe I had already tried booting the modem with the cable disconnected, and pfSense at the same time, to simulate a home power outage, while the ISP hasn't come back. I was still unable to reproduce the problem. It may well come down to timing.

Maybe you could try a coax switch like this so you can try different timings for removing/restoring the cable.

https://www.amazon.com/Coaxial-A-B-Switch-1/dp/B0002ZPIQ4

Not sure if or how much it would degrade the signal.

I have not experienced the same issue with the ISP I still have. The new modem is on the roof and uses PoE, so there is no way to remove just the data signal except physically unplugging the Ethernet cable out of the PoE splitter from the pfSense box.
I'm not sure what the PoE adapter will do if to the Ethernet side if I only unplug the modem from the power side I'm guessing pfSense will show the interface as physically disconnected.

fbrunken

@madbrain @dmatzen I don't believe is ISP related but... My ISP is Telus. I live in the West Coast of Canada. My Internet is fiber, so, not cable related.

Important is to collect more data. In my case I am trying to increase the log of when the link goes down to see if I can catch something that makes sense. The problem is that without being able to reproduce, I just have to wait.

And also, in my case, the fix was to force a DHCPREQUEST. I dont remember at the moment if I just saved the interface or the Gateway. Yes, no need to reboot or anything, just save the information as it was and it came up again...

madbrain

@fbrunken If all you had to do was send a DHCP request, this seems like a different problem that I and others are having. The first thing tried was to reboot pfSense, and that didn't fix it. Rebooting should always force pfSense to issue a DHCP request. But that didn't suffice in fixing things. It required a modem reboot - or even two. But again, without a test case, it's really quite hard to say what the fix could be.

I'm thinking perhaps have an Ethernet smartplug, and some sort of script/package running on pfSense that sends a couple of commands over REST API to turn the modem off and back on. Then perhaps wait a bit, and if there is still no Internet connectivity, repeat. And after 2 failed attempts, also reboot pfSense ...

The trick is that Ethernet smartplugs are quite uncommon. The only one I could find is this one :

https://www.amazon.com/ezOutlet5-Internet-WiFi-Reboot-Switch/dp/B0861NX6H2/ref=cm_cr_arp_d_product_top?ie=UTF8&th=1

https://www.proxicast.com/shopping/ezoutlet5.html

Seems like it should be able to do the job. I probably wouldn't want to use the built-in function to reset router/modem, though. Woudln't trust it, especially if it depends on their cloud server being up, and that server goes away.

The device has unwanted Wifi and Bluetooth, smartphone apps and worst of all, cloud access.

It does feature an API, though. I just checked the doc. It is simple enough. Just feels a bit like it's still 1999, security wise - still using plaintext HTTP/1.1 and basic-auth. Guess they didn't want to maintain a TLS stack or the related admin functions to manage the cert/key. But they could still use something simpler like SSH.

The internal web "Sever" is not well documented. You probably would want to use a static IP for this so it survives router reboots.

The other thing you could use is this :
https://www.amazon.com/Iot-Relay-Enclosed-High-Power-Raspberry/dp/B00WV7GMA2
It requires a device such as a Raspberry Pi to connect it to. Or something with a trigger output
I use one in my home theater, with my receiver's 12V trigger out, to turn on the 2 front subwoofers. For the 2 rear subwoofers, I use 3 KP-125 smartplugs. 2 on the subs themselves, and 1 on the receiver, which measures the current, and Home Assistant detects when it gets turned on or off based on wattage, and turns the 2 rear subs on or off.
I could put 2 KP125 on the front 2 subs also, since I have several extra, and use the IOT Relay for something else, such as this modem/router reboot task.
I have several Raspberry Pi, and could dedicate one to the task also, but all of them are in cases that cover the GPIO pins.
I really dislike the type of connection used on the IoT relay & GPIO, though. They are very hard to setup for me with my declining vision. And very easily pulled, including by cats.

The Pi has Wifi onboard, and can act as a 2.4 GHz access point, though. I could use KP125 smartplugs and connect them to it, instead of using an Ethernet smartplug. Except the KP125 don't support static IPs, only DHCP, which is problematic if pfSense goes down.

My HAOS VM with Z-Wave solution is not reliable enough. I experienced a very bad bug today for the first time. After an HAOS upgrade, the VMWare hypervisor hung. The VM couldn't be rebooted. Or even shut down. I had to reboot the Windows host. And then it came back online. Never seen this before. I cannot even report the issue to Broadcom as I'm not a paying customer.

The other problem is a hardware one - when the host reboots, it sometimes fails to see a few of the 6 SSDs. Sometimes being, once in a blue moon after a power loss. It's never the SATA boot drive, but some of the 5 NVMe which are striped, and contain the HAOS VM.
And of course, in that case, it won't start, and cannot reboot any device.

I have a spare Z-Wave 700 series stick because I upgraded to the 800 series. I could install another instance of HAOS on one of the Pi, re-pair the 2 Z-wave smartplugs on the router & modem with it, and hopefully that would be a more reliable monitoring / modem & router reboot solution ...

dmatzen

I did not just try a DHCP request. Here is what I tried:

Disable / enable interface to force a DHCP request - Failed
Reboot pfSense firewall - failed
Reboot cable modem - failed
One more reboot of pfSense - Sucess

So, I think that because the modem powers up before the ISP can provide a DHCP address, it is left in a state where my firewall will not get an address until the modem has been reset.

As I said, hopefully, I will be home next time this happens so I can determine what the problem is and put something in place to auto-correct the problem in the future.

But some of your thoughts on how to power cycle the various devices are helpful.

Thanks