pfSense auto reboot continuously itself after power was restored

Wepee

I followed the instructions as per shown on the Netgate Documentation.

The system rebooted.

I did not see anything being done on the screen. That is performing a system file check on C drive.

Or probably it ran the lines so fast I could have missed it.

kiokoman

you can also restart in single user mode and launch at the prompt

fsck -y /

do it like 5 or 6 times even if it tell you that the filesystem is clean

Wepee

@kiokoman said in pfSense auto reboot continuously itself after power was restored:

you can also restart in single user mode and launch at the prompt
fsck -y /
do it like 5 or 6 times even if it tell you that the filesystem is clean

Thank you for advice, but something need to be figured out on my own(as your advice was not giving enough
information to work with )

For those newbie who can benefit of understanding what is SINGLE USER MODE:
https://en.wikipedia.org/wiki/Single_user_mode

Below are the steps I took to get Manual File System Check working:

In order to get into Single User Mode.....

Go to the firewall console (plug in a monitor+keyboard)
Press 5 to reboot and then select the option by press: "S" (capital S)
The system will reboot....and then goes into Single User Mode. You see the message: pfSense rebooting into
Single User mode.
Then you see the text line: Enter full pathname of shell or RETURN for /bin/sh:
Press the RETURN key (press this key quickly or else the system will initiate a Random device blocking - system will
kick you out at a random time, if this happen, type: exit, the system will reboot and repeat above step 2 again.
Once you see the hash "#" type: /sbin/fsck -y/ and then press: Enter key, repeat this system a couple of times
even though, the system will display file system is clean. Once you have repeated this step for 4-6 times, type: exit
to leave Single User Mode and reboot the system. DONE

stephenw10

db:0:kdb.enter.default>  bt
Tracing pid 43731 tid 100161 td 0xfffff8002dc1e620
kdb_enter() at kdb_enter+0x3b/frame 0xfffffe0211f9b530
vpanic() at vpanic+0x194/frame 0xfffffe0211f9b590
panic() at panic+0x43/frame 0xfffffe0211f9b5f0
ffs_write() at ffs_write+0x539/frame 0xfffffe0211f9b690
VOP_WRITE_APV() at VOP_WRITE_APV+0x103/frame 0xfffffe0211f9b7a0
vn_write() at vn_write+0x1bc/frame 0xfffffe0211f9b820
vn_io_fault() at vn_io_fault+0x115/frame 0xfffffe0211f9b890
dofilewrite() at dofilewrite+0xc8/frame 0xfffffe0211f9b8e0
kern_writev() at kern_writev+0x68/frame 0xfffffe0211f9b930
sys_write() at sys_write+0x86/frame 0xfffffe0211f9b980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe0211f9bab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0211f9bab0
--- syscall (4, FreeBSD ELF64, sys_write), rip = 0x80221ed0a, rsp = 0x7fffffffc648, rbp = 0x7fffffffc680 ---
db:0:kdb.enter.default>  ps

The power outage caused a file system error that could not be recovered with the usual boot fsck.

As you found running it multiple times from single user mode usually resolves that.

Steve

Wepee

@stephenw10 Thank you for replying the finding.

I am just merely replying in the hope someone, who is a newbie like me, can find this information/tutorial helpful in solving this weird, out of the blue problem. This is my first encounter, in the past while experimenting with pfSense, I did not see this problem happen before.

Gertjan

@Wepee said in pfSense auto reboot continuously itself after power was restored:

I did not see this problem happen before

(DO NOT) Try this with your PC : Boot up .... sneek behind it and rip out the power cable.
You will have a big chance that the OS, - Windows, iOS, Linus, whatever, will complain at boot time that the system wasn't shut down properly.
Most damage to the file system can be auto repaired.
But you always have the chance that the error is non-recoverable, manual intervention will be needed.

Solution : never power down a system that uses a disk with a filesystem by riping out the cable. Use the GUI or commands to shut down.
UPS's are no luxury tools neither because it could also be the cat that rips the power, or a classic power grid outage.

bmeeks

A UPS is a very valuable addition for a firewall. Your firewall (at least a more capable one such as pfSense) is actually a full-fledged operating system that is somewhat constantly communicating with the disk drive. A sudden power loss can leave the disk file structure in an unknown state. A UPS is cheap these days, especially for a home network where you don't necessarily need industrial quality. Check Amazon and other sites for effective yet inexpensive UPS units.

This past Saturday morning a driver nodded off just down the road from my house, ran off the road and into a power pole knocking it down. My neighborhood was without power for almost 4 hours while the power company made repairs. My SG-5100 and cable modem both were kept up the entire time by a small UPS (APC BackUPS ES 650). The pole also carried my cable Internet provider's coax, so my Internet was down but my firewall stayed up. If the UPS had exhausted its battery (it got close), then the apcupsd package I am running on the SG-5100 would have sensed that and shut down the firewall in an orderly fashion preventing possible disk corruption.

I have a UPS on every PC in my house. They are relatively cheap insurance to prevent a corrupted disk due to a power failure. A side benefit is that you can leave your machines running 24 x 7 and let them download and install the Microsoft (or Apple) security updates during the night or very early morning hours while everyone is sleeping. You can also configure automated backups to run during those times to capture any data on the PC disk drive you want to protect.

Wepee

@Gertjan OMG I am sure these day no one should kill the PC like that...........

Wepee

@Gertjan said in pfSense auto reboot continuously itself after power was restored:

@Wepee said in pfSense auto reboot continuously itself after power was restored:

I did not see this problem happen before

(DO NOT) Try this with your PC : Boot up .... sneek behind it and rip out the power cable.
You will have a big chance that the OS, - Windows, iOS, Linus, whatever, will complain at boot time that the system wasn't shut down properly.
Most damage to the file system can be auto repaired.
But you always have the chance that the error is non-recoverable, manual intervention will be needed.

Solution : never power down a system that uses a disk with a filesystem by riping out the cable. Use the GUI or commands to shut down.
UPS's are no luxury tools neither because it could also be the cat that rips the power, or a classic power grid outage.

LOL only my great grand father would do this to a PC

Wepee

@bmeeks Thanks for the advice, may be I should invest a UPS, but I still think my internet connection is not super critical, as I am not running a business at home. APC batteries really sucks, they don't last long, after so many of cycles of recharging, the battery will die in about 1-2 years depends on how frequent do I have blackouts at my area.

bmeeks

@Wepee said in pfSense auto reboot continuously itself after power was restored:

@bmeeks Thanks for the advice, may be I should invest a UPS, but I still think my internet connection is not super critical, as I am not running a business at home. APC batteries really sucks, they don't last long, after so many of cycles of recharging, the battery will die in about 1-2 years depends on how frequent do I have blackouts at my area.

I usually get about 3 years out of a battery. I just buy a replacment when one expires. To each his own, but the price of the occasional replacement battery is relatively trivial to me when compared to the royal pain of corrupting your firewall disk (or any PC's disk, for that matter) and having to possibly rebuild the machine from scratch in order to recover. As an added bonus, the UPS is a dandy surge suppressor due to the additional level of isolation and offers a bit more protection from nearby lightning strikes than a plain surge protector does. With the UPS in place, I don't sweat the short outages or little power blips caused by the summer thunderstorms that are common where I live.

Wepee

@bmeeks Thanks for sharing. I have a couple of UPS lying around since the batteries have failed to hold power.
I am thinking of replacing it with my own batteries if I can get those DEEP CYCLE type rechargeable battaries,
which should last longer.

Does the APC package(plugin) downloadable from the package manager, works with other 3rd UPS?

Has anyone try it?

bmeeks

@Wepee said in pfSense auto reboot continuously itself after power was restored:

@bmeeks Thanks for sharing. I have a couple of UPS lying around since the batteries have failed to hold power.
I am thinking of replacing it with my own batteries if I can get those DEEP CYCLE type rechargeable battaries,
which should last longer.

Does the APC package(plugin) downloadable from the package manager, works with other 3rd UPS?

Has anyone try it?

Yes, the apcupsd package will work with 3rd party UPS equipment. It will work best with newer devices which have a USB signalling interface. There is also another package for pfSense called nut that works the same as apcupsd. If your particular UPS model does not work correctly with one package, you can try installing the other one.

Batteries in a UPS will naturally age out every few years. I seem to get about two or three years of life out of one. Depending on the size of the UPS (output power capability), it is sometimes about the same cost to just replace the entire UPS instead of changing just the battery. This is more true with the small 350 KVA or so units. I just look at the battery replacement as an expected expense and plan for it for all of my machines. I have a UPS on every PC, my two servers and the firewall in my home.