Lights are on and no one is home after power outages with battery backup.
-
I apologize ahead of time as this is my first time troubleshooting pfSense - it has been great up until the last couple of months. I use it minimally, and I am dumb.
I have a Netgate SG-3100 running 22.01-RELEASE (arm), and recently when there has been a power outage to the house I am needing to pull power from the unit and then plug it back in to get it to operate normally.
Along with the Netgate I have several other devices being powered by the same UPS system: L3 switch, camera nvr, ubiquiti key, and a few servers, all of which behave flawlessly when there is either a brief brownout or a longer duration power outage. The UPS logs indicate flawless switch over between mains and battery with either one or none fluctuations of voltage.
I am not near the rack to see if the Netgate physically is powering up/down during the UPS switch overs, and better question would be why could it be when all the other devices are fine. I do see that it has power when I've checked on everything before, during, and after the switch over, and the Netgate has had lights on.
When after the fact I do then go to use the internets once power is restored, there is nothing. When I go to ping, https, ssh, or serial in to the Netgate, there is nothing. The lights are on, but it is doing nothing. I have to then pull power to it then power it back on, and cross fingers, so far it has come up fine each time after that.
Why is this happening? What can I check?
-
@SephYuyX you mention a few things there that are on the ups. But I didn't see you mention say the modem that provides internet to pfsense.
I would for sure check the logs on pfsense to make sure it didn't power cycle when your power went out.
But even if say the modem went down.. You should still be able to ping the pfsense lan IP, or ssh to the IP.. Possible if your using dns to do that - that dns might be down?
When its not responsive - I would console and and see if its stuck in a boot up mode or something.. Or if some other sort of crash happened. Oh I see you can not even console in?? Yeah that is not good.. hmmm.
i have my sg4860 on ups, and have never seen any sort of issues when goes on ups power.. Do you have your pfsense connected to your ups.. Is it possible its doing a halt when the ups goes on battery? I am not sure if the lights go out on a halt, I don't believe they do to be honest.
I would think if pfsense did a halt, that could explain your symptoms. I don't think console is available after a halt.. I don't see how it could be.. But the lights could still be on if there is still power from the ups.
-
@johnpoz Thanks for your reply.
Yes, the modem is on the UPS as well.
To clarify about not being able to console into it - I am able to console to it, or at least in the way that no errors are given in the attempt, but when in there it is just a black screen. Any sort of input appears to have no effect, it just stays on the black screen. Is this considered a halted state? When I initiate a power cycle while still consoled, I then immediately see its boot process occurring.
The Netgate pfSense is on the same UPS as everything else. I just don't understand how that could be much more sensitive to power fluctuations (and very minor ones if none at all). Also, prior to some months ago it would be fine during this power outages as well. Hardware starting to become faulty on the Netgate maybe? Just strange this is the only device having issues.
As for logs - which? Status > System Logs > System > General?
Or would there be another one to look at?
-
@SephYuyX What is doing the monitoring for the UPS? If pfSense, via NUT or Apcupsd, and pfSense is shutting down, then I believe the 3100 doesn't/can't power itself off. I know other Arm models are that way. But if it's a short outage one would think that would not apply.
Can you leave a console visible, to see if anything is output when this happens?
-
@SteveITS It is an APC UPS, and I have a network card on it that I can access and syslog off of to a server. I am strictly monitoring the UPS, and not doing any sort of powering off of other devices based on its status.
I at one point tried doing the UPS logging via pfSense, but I decided against it at one point and uninstalled those services.
I could suppose console into it and turn the breaker off to the UPS and see what happens, but I would have to do that next week.
-
Is it possible that you have the pfSense box accidentally plugged in to one of the surge protected only outlets on the UPS. I ask because I have done this myself on one of those days that I wake up dumber than a stump! DOH!!
Phizix
-
@Phizix I thought I was the only one that did that...
-
@Phizix Thanks, but it is on a PDU bar that has the router and one of the servers on it as well where that server still has an uptime. I'll try switching it to a different outlet to double check though.
-
@SephYuyX said in Lights are on and no one is home after power outages with battery backup.:
@SteveITS It is an APC UPS, and I have a network card on it that I can access and syslog off of to a server. I am strictly monitoring the UPS, and not doing any sort of powering off of other devices based on its status.
Did I understand correctly that you do not use NUT neither the APC package on pfSense ?
-
@Gertjan Correct.
-
When the power goes down, your 3100 wouldn't know about it.
Eventually, the UPS battery gets drained enough, and shut down the power : that's not good.
Ripping out the power is never a good idea - very bad for the file system.You 'have to use' NUT (or APC) package, with a direct USB to the UPS, or, have it connected to another remote NUT server.
With NUT (or the APC) package, your 3100 would go for a clean halt situation.
If at that moment the power comes back - and the 3100 was halted, then probably some one has to come over to 'remove the power', wait 5 sec and put the power back in : the 3100 would boot normally.
If the UPS really shut down the 230 V (110 ?), and it comes back, your 3100 should boot normally.If you use the recently introduced ZFS file system, you risk less, but the older ones can break more easier: your 3100 won't boot any more and you might be good for a re install.
See also How to Run a pfSense Software File System Check (5/2020) -
@Gertjan I'll look back into it.
So far I've never had an outage long enough to drain the batteries out (around 8 hours), but it will happen one day I am sure.
Still does not make sense why the Netgate appliance is having this issue when everything else is fine when switching back and forth between UPS and mains.
-
@SephYuyX How old is the UPS? I think you said it was an APC? What is it's sensitivity setting to switch from battery and mains?
It could be the power supply/power input to the 3100 is overly sensitive. -
@SephYuyX said in Lights are on and no one is home after power outages with battery backup.:
Still does not make sense why the Netgate appliance is having this issue
It's just a question of having some luck, or not.
DO not try to proof me wrong (or right), but :
Remove the power cable from your PC portable.
Power up the portable.
Remove the battery.
And try to reboot your MAC/Windows PC .... chances are big you (file system) have issues.Your : way easier to test : your Windows 10/11 desktop : rip out the power. After less then 10 times you're good for a re install as windows won't boot any more.
These issues don't happen often these days, as people are now trained to do a start menu click - click power down - or sleep mode.Using a more resilient file-system like ZFS helps
Using an UPS, and hook it up logically, so your system can do a clean shut down before the power really dies, is best. -
I don't think the 3100 can do ZFS per this thread - Netgate 3100, 22.01 and ZFS?. So that does not seem to be an option for the OP.
Phizix
-
@Phizix it can’t, 32 bit Arm cannot.
@SephYuyX if it’s an APC that big it’s presumably an online UPS…? So it would run off the battery 24x7 and not have any voltage fluctuations. Is there anything else plugged into the 3100 that isn’t on the UPS? Some other path for something to get to it?
-
I wasn't sure.
The older UFS ( ? ) file system is far less resilient against sudden power downs.
That's why the "How to Run a pfSense Software File System Check (5/2020)" video was published by Netgate, as many people do shut down properly their 'big' devices, but treat router/firewall devices as "rip out the power is just fine".
Well, that is not the case.
Shutting down by removing the power will leave "dents" in the file system. Some of them can stop the system from booting. -
My SG-5100 originally did not have ZFS until a power blip took out the config. Fortunately I had a recent enough backup that I updated it to ZFS from scratch and then restored the config.
It happened when one of the kids just turned off the UPS and I had not yet set up my RasPi 4 NUT server.
If I am home I shut things down manually before the NUT client shuts it down, but If I am not we will see. I don't think it has had a full real world test of the NUT environment.
What drives me crazy is if I am not home often my wife or kids will just turn off the UPSs without shutting down the attached devices first - DOH!
Phizix
-
@Phizix said in Lights are on and no one is home after power outages with battery backup.:
or kids will just turn off the UPSs without shutting down the attached devices first
Another to set up stuff "as it was meant to be done".
When they cut the UPS (it will start to beep beep), and at that moment your server farm, pfSense etc will get 'halted'.Me curious : but why would the cut the power from the UPS ?
Mine have labels : "don't touch - or be ready to assume ....".
-
Trust me, they did get a good bollocking from me when it corrupted the SG-5100 gateway.
Occasionally they will still do it and I get pretty cross at them.
Phizix
