pfSense stopped routing after power failure
-
remove the cisco and directly connect a pc to em1 and see if it work
upload some screenshot of your gui configuration (routing table / firewall rules) -
@kiokoman there’s a lot of reconfig that would need to be done due to the VLAN tagging, but not sure what difference that would make since between the server and the pfSense I have no issues, it’s only when needing to do L3 routing I have issues which would be all internal to pfSense.
I will work on the screen shots.
-
You need to examine your system log and perhaps the
dmesg.boot
log for errors. You will find several service logs in /var/log on the firewall. The power failure very well could have resulted in a corrupted filesystem and perhaps some critical component is now not starting correctly.You should put your firewall (and perhaps also your switch) on a small UPS. $40 - $60 is cheap for the peace of mind it buys you by protecting your firewall from filesystem corruption.
-
@bmeeks did run fsck at reboot thinking of the corruption but don’t see any logs for it.
As for the logs, nothing jumped out at me but I will take a second look and will look at dmesg which I did t think of with this being FreeBSD and not Linux.
Didn’t really worry about a UPS because the power is really stable, this was self inflicted due to bad labeling of breakers but you bet I will be getting a UPS at this point.
-
The fact you have to restart the web configurator in order to gain GUI access indicates something is damaged in the pfSense configuration. Exactly what that may be, I don't know. Looking through those logs in /var/log may give you a clue.
-
@bmeeks ok so running sysctl -w net.inet.ip.forwarding=1 gets my internal connectivity restored. NAT is still not working but tells me something’s definitely corrupt on the file system.
So, not even sure where to set this on FreeBSD, and how to troubleshoot the NAT since iptables isn’t used.
-
@jlw52761 said in pfSense stopped routing after power failure:
@bmeeks ok so running sysctl -w net.inet.ip.forwarding=1 gets my internal connectivity restored. NAT is still not working but tells me something’s definitely corrupt on the file system.
So, not even sure where to set this on FreeBSD, and how to troubleshoot the NAT since iptables isn’t used.
It really would be easier to reinstall and restore your config. Check your config backups. Copy the most recent one off to some other location (for example, the PC you normally use to administer pfSense). You can then reinstall pfSense and restore the saved backup configuration. That will put back all of your VLANs and other setup information.
-
@bmeeks yeah, I think I’m at that point. Things seem to be really borked.
-
@bmeeks I ended up just reinstalling the OS and doing a restore. Things came right back up without a hitch.
-
@jlw52761 said in pfSense stopped routing after power failure:
@bmeeks I ended up just reinstalling the OS and doing a restore. Things came right back up without a hitch.
I know having to do that is a little painful and scary, but it's usually the best cure. Now get a UPS configured and install either the
nut
orapcupsd
package to monitor the UPS and gracefully shutdown pfSense when there is another power failure and the battery is near exhaustion. Installing a package is important as that lets the UPS notify the firewall that the AC mains are down and the battery is about to expire. The package code then shuts down pfSense gracefully. You can configured when that happens, but I think the default is when there are 5 minutes of battery life remaining.I had an incident in my neighborhood recently where the driver of a car ran off the road and knocked down a power pole. My house was without power for nearly 6 hours while repairs were made. I have a Netgate SG-5100 and my cable modem plugged into a APC BackUPS 650 ES. The UPS kept my firewall and cable modem running the entire duration of the power outage. Of course the same power pole also carried my cable Internet connection so I was dead in the water in terms of connectivity. I also have a UPS on all of my other computers including my ESXi servers. They all stayed up until their batteries neared exhaustion, then they each shutdown gracefully. Once power was restored they all came right back up just like nothing ever happened.
-
@bmeeks Got one 500VA UPS coming tomorrow for the fiber modem, pfSense, switch, and the two UniFi AP's. This will be USB cabled to the pfSense and it's an APC so if
nut
doesn't work then I will useapcupsd
.I have a second UPS also coming, in the range of 1800VA for my NAS, switch, and ESXi boxes.
As far as the restore, since I have backups and copy them off the appliance, it was stupid simple. I created the USB key, added the FAT32 "Recover" partition and copied the backup xml file and named it config.xml. I had to hook the firewall up to my TV as I have no VGA monitors in my house surprisingly, but it booted, installed, and on reboot applied config.xml and was up and going. Stupid simple DR in my mind and a huge bonus for pfSense in my book!
From now on, it's going to be a DR instead of hours of troubleshooting, its just too damned easy to recover.
Going to use a SIIG USB over IP device and a FTDI cable to have remote access to the console for any future needs.