Boot Hangs After "Starting Cron...Done" w/High CPU and Unresponsive GUI
-
Hello All.
I am having an issue and here is a summary.Yesterday I had a PFblocker update notification within the dashboard in my stable and error free environment with version 2.7.2-Release that was running for nearly a month since last reboot with no issues. I clicked the update symbol and confirmed I wanted to perform the update.
Thinking nothing of it I went about my business assuming all would be well until this morning when I noticed the that the update screen was still working and the box was hot (turns out cpu maxed at 100%).
I refreshed and tried to log back in but the login which appeared successful hung forever.
I connected my monitor and keyboard to the box and did a reboot and thats when I started seeing that once starting cron...done was displayed nothing happened until 900 seconds later when I get a message "Starting package pfBlockerNG-devel...
Fatal error: Maximum execution time of 900 seconds exceeded in /etc/inc/util.inc on line 3631 PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 3631, Message: Maximum execution time of 900 seconds exceeded" displayed.Having tried everything I could think of (reinstalling known working backups, reboots, checking for updates, etc) I did a fresh install and the same thing still happens (starting cron....done and then nothing for 900 seconds).
As this is a fresh install I do not have the pfBlocker package installed as I cant get that far due to the system hanging. After I get the above message the Welcome screen is displayed in the DOS like window (sorry dont know the name of this)
After the fresh install I can login but I see cpu maxed out and when I try to do anything else like check for updates or go to another screen the GUI just hangs and I cant log back in until I do a reboot.
Also once the GUI hangs and I try to log back in by closing tab and opening another I get a Bad Gateway error. Not sure this info helps but figured I would note it.
Since I did a fresh install, which in theory should alleviate any issues with packages, custom configurations, scripts, etc that may be causing the issue I started to think possibly its hardware related but I dismissed that idea because the system boots, runs the application, connects to the internet and does everything a functioning PC should be doing with no hardware messages being displayed, also this is a new system (<6months old).
We can access the internet so at some level the firewall is working but obviously it is not working correctly.
I have exceeded my ability to troubleshoot this and am at a loss as what to do next and would really appreciate some suggestions.
-
The pkg has been updated to revert that problem. 3.2.0_17 is now good.
To upgrade to that you will need to kill the stuck process from the command line as shown here: https://forum.netgate.com/post/1185688
Steve
-
@stephenw10 Hey Stephen good to hear from you hope all is well. Thanks for the info.
I have looked at the link sent and I know I am a bit dense but hopefully you can help me.
I can see the command to install the new devel pkg (pkg-static install pfSense-pkg-pfBlockerNG) but am unclear on how to do the uninstall of current pkg or how to get the list of running processes with the PID, etc to see if anything is still hanging.
I used TOP but doesnt seem to be the same.
Also once I do this can I restore a backup which has pfBlocker and not run into the problem again?
PS.. The DOS window is now working properly so I can execute commands there.
-
Run the uninstall:
pkg-static remove pfSense-pkg-pfBlockerNG-develIt will hang.
Then in another terminal find the stuck process from:
ps -auxwwd
Kill that process.
In the other window the removal should complete.Then reboot and you should be able to install the package again from the webgui.
Restoring a config will always pull in the new version of the package which is now fixed.
-
@stephenw10 Ok so I cant get into the web gui unless I do a reboot and when I do once I get in the whole thing hangs. I executed the command via the DOS window to uninstall and like you said it has hung. I am not sure how to get into another window like you suggested given the GUI hangs and the DOS window has also hung. Sorry for being thick on this one. I am going to see if I can get SSH setup.
-
This post is deleted! -
@stephenw10 I configured SSH and was able to uninstall the package and the process did indeed hang so in the other window I found the process and killed it. I then did the install of the new pfb package 3.2.0_8 which seems to have been successful and when I logged into the GUI the CPU usage is now minimal so I think at this point I am good. I am now going to restore a prior backup and hope all goes well based on your comment that the new PFB package will be used instead of that contained in my backup. Will update with results. Fingers crossed.
-
Yup you should be good on the non-devel version. Though the devel package is now safe to upgrade to if you want.
-
@stephenw10 Thanks Stephen yes, woo hoo back up and running. As always you saved me and taught me something new and nudged me to enhance my ninja skills. I had to run out for a couple of hours which is why I am late with my update but I noticed that PFB package is now 3.2.0_17, there must have been an update while I was out. I wasnt aware that updates happened automatically, I usually had to initiate them. I need to do some tweaking as my backup is 2 months old but its all good and everything is green across the board. Thanks again for your help. You rock!
-
@LPD7 said in Boot Hangs After "Starting Cron...Done" w/High CPU and Unresponsive GUI:
I wasnt aware that updates happened automatically
They don't.
pfSense Packages like pfBlockerng(-devel) or pfSense itself, or even FreeBSD-pfSense-packages (you can't see these in the GUI, only on the console or SSH menu, option 13) are only installed when the 'admin' click 'update' in the GUI, and the Confirms = another click.
Or uses option 13 in the console menu, and then confirms with 'Y' or 'y' + enter.The manual method is mandatory, as you have to make your config backup before - inform connected users that the connection might go down, you have to prepare a coffee first, make sure no one is bothering you, and, depending on your familial situation, prepare the dog house outside as you might sleep in it this evening if things go bad.
-
@stephenw10
Thank you I was in the camp with LPD7. Recovered now.
