pfSense auto reboot continuously itself after power was restored



  • Hi all,

    My pfSense firewall did not have a backup power installed.

    The firewall device was not struck by lightning since there was no thunderstorm.

    The power grid contractors was fixing in the power substation near my home.

    A power failure occurred at my area, when the power was back on, the firewall device would
    normally auto restart during a power restore event (set in the bios).

    So it will boot up normally without having the user physically press the power button in order to power up the firewall.

    Under normal circumstance, the firewall will boot up as per usual........it would run........

    I could hear the beeping tune indicating all interfaces are up and running and all the services loaded.

    However, it would just crash( I just could not see on the screen when actually happened,
    it spat out a lot of the white text lines very fast) next...........

    Rebooted itself after running about 10 seconds.........repeatedly non-stop( in a continuous cycle)

    I switched off the power and unplugged the power cable, began troubleshooting.

    I took off the firewall device from my shelve, took out and checked the SSD health
    using Kingston SSD manager windows software, it reported to be good 95% and subsequently
    updated the lastest Kingston SSD firmware.

    Fortunately, I plugged back the SSD into the firewall device, booted up and it is running fine now- no more
    auto-rebooting problem.

    To find out, what was the cause of the problem. I have attached the crash reports (info.0 and textdump.tar.0)

    Can any of the developer interested in debugging what went wrong?

    Thank you in advance.

    info.0

    textdump.tar.0


  • LAYER 8

    probably a filesystem corruption
    you should restart and launch a filesystem check
    https://docs.netgate.com/pfsense/en/latest/hardware/forcing-a-filesystem-check.html



  • I followed the instructions as per shown on the Netgate Documentation.

    The system rebooted.

    I did not see anything being done on the screen. That is performing a system file check on C drive.

    Or probably it ran the lines so fast I could have missed it.


  • LAYER 8

    you can also restart in single user mode and launch at the prompt

    fsck -y /
    

    do it like 5 or 6 times even if it tell you that the filesystem is clean



  • @kiokoman said in pfSense auto reboot continuously itself after power was restored:

    you can also restart in single user mode and launch at the prompt

    fsck -y /
    

    do it like 5 or 6 times even if it tell you that the filesystem is clean

    Thank you for advice, but something need to be figured out on my own(as your advice was not giving enough
    information to work with☺ )

    For those newbie who can benefit of understanding what is SINGLE USER MODE:
    https://en.wikipedia.org/wiki/Single_user_mode

    Below are the steps I took to get Manual File System Check working:

    In order to get into Single User Mode.....

    1. Go to the firewall console (plug in a monitor+keyboard)
    2. Press 5 to reboot and then select the option by press: "S" (capital S)
    3. The system will reboot....and then goes into Single User Mode. You see the message: pfSense rebooting into
      Single User mode.
    4. Then you see the text line: Enter full pathname of shell or RETURN for /bin/sh:
    5. Press the RETURN key (press this key quickly or else the system will initiate a Random device blocking - system will
      kick you out at a random time, if this happen, type: exit, the system will reboot and repeat above step 2 again.
    6. Once you see the hash "#" type: /sbin/fsck -y/ and then press: Enter key, repeat this system a couple of times
      even though, the system will display file system is clean. Once you have repeated this step for 4-6 times, type: exit
      to leave Single User Mode and reboot the system. DONE

  • Netgate Administrator

    db:0:kdb.enter.default>  bt
    Tracing pid 43731 tid 100161 td 0xfffff8002dc1e620
    kdb_enter() at kdb_enter+0x3b/frame 0xfffffe0211f9b530
    vpanic() at vpanic+0x194/frame 0xfffffe0211f9b590
    panic() at panic+0x43/frame 0xfffffe0211f9b5f0
    ffs_write() at ffs_write+0x539/frame 0xfffffe0211f9b690
    VOP_WRITE_APV() at VOP_WRITE_APV+0x103/frame 0xfffffe0211f9b7a0
    vn_write() at vn_write+0x1bc/frame 0xfffffe0211f9b820
    vn_io_fault() at vn_io_fault+0x115/frame 0xfffffe0211f9b890
    dofilewrite() at dofilewrite+0xc8/frame 0xfffffe0211f9b8e0
    kern_writev() at kern_writev+0x68/frame 0xfffffe0211f9b930
    sys_write() at sys_write+0x86/frame 0xfffffe0211f9b980
    amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe0211f9bab0
    fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0211f9bab0
    --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x80221ed0a, rsp = 0x7fffffffc648, rbp = 0x7fffffffc680 ---
    db:0:kdb.enter.default>  ps
    

    The power outage caused a file system error that could not be recovered with the usual boot fsck.

    As you found running it multiple times from single user mode usually resolves that.

    Steve



  • @stephenw10 Thank you for replying the finding.

    I am just merely replying in the hope someone, who is a newbie like me, can find this information/tutorial helpful in solving this weird, out of the blue problem. This is my first encounter, in the past while experimenting with pfSense, I did not see this problem happen before. 👍 ☺



  • @Wepee said in pfSense auto reboot continuously itself after power was restored:

    I did not see this problem happen before

    (DO NOT) Try this with your PC : Boot up .... sneek behind it and rip out the power cable.
    You will have a big chance that the OS, - Windows, iOS, Linus, whatever, will complain at boot time that the system wasn't shut down properly.
    Most damage to the file system can be auto repaired.
    But you always have the chance that the error is non-recoverable, manual intervention will be needed.

    Solution : never power down a system that uses a disk with a filesystem by riping out the cable. Use the GUI or commands to shut down.
    UPS's are no luxury tools neither because it could also be the cat that rips the power, or a classic power grid outage.



  • A UPS is a very valuable addition for a firewall. Your firewall (at least a more capable one such as pfSense) is actually a full-fledged operating system that is somewhat constantly communicating with the disk drive. A sudden power loss can leave the disk file structure in an unknown state. A UPS is cheap these days, especially for a home network where you don't necessarily need industrial quality. Check Amazon and other sites for effective yet inexpensive UPS units.

    This past Saturday morning a driver nodded off just down the road from my house, ran off the road and into a power pole knocking it down. My neighborhood was without power for almost 4 hours while the power company made repairs. My SG-5100 and cable modem both were kept up the entire time by a small UPS (APC BackUPS ES 650). The pole also carried my cable Internet provider's coax, so my Internet was down but my firewall stayed up. If the UPS had exhausted its battery (it got close), then the apcupsd package I am running on the SG-5100 would have sensed that and shut down the firewall in an orderly fashion preventing possible disk corruption.

    I have a UPS on every PC in my house. They are relatively cheap insurance to prevent a corrupted disk due to a power failure. A side benefit is that you can leave your machines running 24 x 7 and let them download and install the Microsoft (or Apple) security updates during the night or very early morning hours while everyone is sleeping. You can also configure automated backups to run during those times to capture any data on the PC disk drive you want to protect.



  • @Gertjan OMG I am sure these day no one should kill the PC like that...........😀



  • @Gertjan said in pfSense auto reboot continuously itself after power was restored:

    @Wepee said in pfSense auto reboot continuously itself after power was restored:

    I did not see this problem happen before

    (DO NOT) Try this with your PC : Boot up .... sneek behind it and rip out the power cable.
    You will have a big chance that the OS, - Windows, iOS, Linus, whatever, will complain at boot time that the system wasn't shut down properly.
    Most damage to the file system can be auto repaired.
    But you always have the chance that the error is non-recoverable, manual intervention will be needed.

    Solution : never power down a system that uses a disk with a filesystem by riping out the cable. Use the GUI or commands to shut down.
    UPS's are no luxury tools neither because it could also be the cat that rips the power, or a classic power grid outage.

    LOL only my great grand father would do this to a PC 😁



  • @bmeeks Thanks for the advice, may be I should invest a UPS, but I still think my internet connection is not super critical, as I am not running a business at home. APC batteries really sucks, they don't last long, after so many of cycles of recharging, the battery will die in about 1-2 years depends on how frequent do I have blackouts at my area.



  • @Wepee said in pfSense auto reboot continuously itself after power was restored:

    @bmeeks Thanks for the advice, may be I should invest a UPS, but I still think my internet connection is not super critical, as I am not running a business at home. APC batteries really sucks, they don't last long, after so many of cycles of recharging, the battery will die in about 1-2 years depends on how frequent do I have blackouts at my area.

    I usually get about 3 years out of a battery. I just buy a replacment when one expires. To each his own, but the price of the occasional replacement battery is relatively trivial to me when compared to the royal pain of corrupting your firewall disk (or any PC's disk, for that matter) and having to possibly rebuild the machine from scratch in order to recover. As an added bonus, the UPS is a dandy surge suppressor due to the additional level of isolation and offers a bit more protection from nearby lightning strikes than a plain surge protector does. With the UPS in place, I don't sweat the short outages or little power blips caused by the summer thunderstorms that are common where I live.



  • @bmeeks Thanks for sharing. I have a couple of UPS lying around since the batteries have failed to hold power.
    I am thinking of replacing it with my own batteries if I can get those DEEP CYCLE type rechargeable battaries,
    which should last longer.

    Does the APC package(plugin) downloadable from the package manager, works with other 3rd UPS?

    Has anyone try it?



  • @Wepee said in pfSense auto reboot continuously itself after power was restored:

    @bmeeks Thanks for sharing. I have a couple of UPS lying around since the batteries have failed to hold power.
    I am thinking of replacing it with my own batteries if I can get those DEEP CYCLE type rechargeable battaries,
    which should last longer.

    Does the APC package(plugin) downloadable from the package manager, works with other 3rd UPS?

    Has anyone try it?

    Yes, the apcupsd package will work with 3rd party UPS equipment. It will work best with newer devices which have a USB signalling interface. There is also another package for pfSense called nut that works the same as apcupsd. If your particular UPS model does not work correctly with one package, you can try installing the other one.

    Batteries in a UPS will naturally age out every few years. I seem to get about two or three years of life out of one. Depending on the size of the UPS (output power capability), it is sometimes about the same cost to just replace the entire UPS instead of changing just the battery. This is more true with the small 350 KVA or so units. I just look at the battery replacement as an expected expense and plan for it for all of my machines. I have a UPS on every PC, my two servers and the firewall in my home.


Log in to reply