"PHP Fatal error: Maximum execution time of 900 seconds exceeded in /etc/inc/util.inc on line 3728" after installing latest pfBlockerNG 3.2.0_12
-
Just to add a data point.
Two boxes running CE 2.7.2, one "live" system, one hot spare.
Both were previously on PFBLocker-NG 3.2.0_15, both updated yesterday although not at exactly the same time, the "live" system updated to 3.2.0_17 seemingly without issue, (phew) while the spare system got stuck with the update never completing in the GUI, although after I refreshed the GUI things seemed OK at first and I didn't look any further as it is not a live system.
However I got an alert a while later which ironically was a CPU temperature alert... and logging into the GUI I saw a crash report which included "PHP Fatal error: Maximum execution time of 900 seconds exceeded in /etc/inc/util.inc on line...." although I can't currently get back to the crash log as the Web UI has hung.
Two processes - pfblockerng.php cron and pfSense-pkg-pfBlockerNG-devel POST-INSTALL were stuck on 100% each so fully utilising 2 of the 4 cores on the system. In the htop output below I had tried to go to the package manager in the GUI, this caused the web interface processes to also get stuck in a high CPU loop as well making the Web UI unresponsive:
4374 root 107 0 140M 51460 R 43.3 0.3 18:39.17 php-fpm: pool nginx 40181 root 106 0 140M 50880 R 42.1 0.3 4:11.17 php-fpm: pool nginx 22471 root 106 0 70596 48840 R 41.1 0.3 23h50:37 //usr/local/bin/php -f //etc/rc.packages pfSense-pkg-pfBlockerNG-devel POST-INSTALL 399 root 107 0 140M 54144 R 41.0 0.3 19:04.01 php-fpm: pool nginx 81451 root 106 0 140M 50040 R 40.9 0.3 3:30.04 php-fpm: pool nginx 89567 root 107 0 70340 47936 R 40.3 0.3 7h48:53 /usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php cron 46010 root 106 0 140M 50444 R 39.5 0.3 18:28.28 php-fpm: pool nginx 68826 root 106 0 140M 50920 R 39.1 0.3 18:40.88 php-fpm: pool nginx 21846 root 106 0 140M 52448 R 38.8 0.3 18:55.89 php-fpm: pool nginx@stephenw10 As this is not a live system I can afford to do further forensics on it if it's helpful to figure out what went wrong, before attempting a fix ? If no further info is required to help troubleshoot the cause I will attempt a manual fix.
-
As seen elsewhere :
kill 89567and then check if process 22471 continues finishing its job.
-
@Gertjan Killed. how long should I give it ?
The POST-INSTALL process is still on 100%.
However judging by my CPU graph, the cron process got stuck in a 100% loop AFTER the install process did... the install process was already sitting at 100% CPU for hours before that.
(Cron runs at 3am)
-
@DBMandrake Checking system.log I can see I started the update at 10:47am: (UK time)
Sep 24 10:46:58 pfSense3 lighttpd_pfb[4067]: [pfBlockerNG] DNSBL Webserver stopped Sep 24 10:46:58 pfSense3 tail_pfb[6074]: [pfBlockerNG] Firewall Filter Service stopped Sep 24 10:46:58 pfSense3 php_pfb[6603]: [pfBlockerNG] filterlog daemon stopped Sep 24 10:46:58 pfSense3 php[2869]: /etc/rc.packages: Configuration Change: (system): Intermediate config write during package removal for pfBlockerNG-devel. Sep 24 10:46:58 pfSense3 check_reload_status[439]: Syncing firewall Sep 24 10:46:58 pfSense3 php[2869]: /etc/rc.packages: Configuration Change: (system): pfBlockerNG: saving Unbound custom options Sep 24 10:46:58 pfSense3 check_reload_status[439]: Syncing firewall Sep 24 10:46:58 pfSense3 php[2869]: /etc/rc.packages: Configuration Change: (system): pfBlockerNG: saving DNSBL changes Sep 24 10:46:58 pfSense3 snort[40410]: Received notification of an IP address change on interface lo0. Sep 24 10:46:58 pfSense3 snort[40410]: Deleted address 10.10.10.1 from automatic interface IP Pass List for Legacy Blocking Mode. Sep 24 10:47:00 pfSense3 check_reload_status[439]: Reloading filter Sep 24 10:47:00 pfSense3 php[2869]: [pfBlockerNG] Restarting firewall filter daemon Sep 24 10:47:00 pfSense3 php[22471]: //etc/rc.packages: Beginning package installation for pfBlockerNG-devel . Sep 24 10:47:00 pfSense3 php[22471]: //etc/rc.packages: Configuration Change: (system): Intermediate config write during package install for pfBlockerNG-devel. Sep 24 10:47:00 pfSense3 check_reload_status[439]: Syncing firewallAnd there is NOTHING in the log after this relating to the package installation, in fact the next thing in the log is at 4am the following day from the auto config backup service.
By the way the process 22471 is still on 100% and I don't think it's going to complete...
-
kill also 22471
and as showed by @stephenw10 (see above) :
pkg remove pfSense-pkg-pfBlockerNGso pfSense doesn't try to re installed the already downloaded package and gets blocked gain.
-
@Gertjan It's
pkg remove pfSense-pkg-pfBlockerNG-develas it's the development package.Manually removing it and reinstalling it has fixed it.
I'm wondering if I should go back to the non-devel version - I've been on the devel version of pfBlockerNG forever because the standard package was a full major version behind for a long time, and was lacking a feature I needed, but they're both 3.2.xx now.
I'm guessing I would have to note down my configuration and manually reconfigure if I did this though as the system would treat this as a different package altogether with its own configuration ?
-
THX! This was working for me
-
@DBMandrake For what it's worth, I got out of this mess by removing the devel version from the command line and installing the non-devel version, and it appears as if all my config from the devel version carried over to non-devel.
-
@TheNarc said in "PHP Fatal error: Maximum execution time of 900 seconds exceeded in /etc/inc/util.inc on line 3728" after installing latest pfBlockerNG 3.2.0_12:
@DBMandrake For what it's worth, I got out of this mess by removing the devel version from the command line and installing the non-devel version, and it appears as if all my config from the devel version carried over to non-devel.
Did you install the non-devel version through the GUI package manager or from the command line ?
I would have to check but I think the only feature of pfBlockerNG that I'm actually using is the Google/Youtube/Bing safesearch DNS entries, as its a lot easier to tick a box in pfBlockerNG to achieve this than manually enter (and maintain) dozens of entries in Unbound!
If the config carries over I think I will do this so I'm not on the "bleeding edge". (Although this is the first time I've actually had an issue with the devel version to be fair)
-
Once you have removed the bad devel pkg by killing the script process you can reinstall either pfBlockerNG version and it should work. And include the existing settings.
-
@stephenw10 Is it unusual for two different packages in PFSense to share the same configuration files / backup ? Or are these two packages mutually exclusive so only one can be installed at the same time ? (Using the same configuration file directories)
-
@DBMandrake I did it via the cli because at the time the GUI was being problematic due to the original devel-version issue.
-
@DBMandrake said in "PHP Fatal error: Maximum execution time of 900 seconds exceeded in /etc/inc/util.inc on line 3728" after installing latest pfBlockerNG 3.2.0_12:
s it unusual for two different packages in PFSense to share the same configuration files / backup ?
Yes, I'm not sure any other package does (maybe HAProxy-dev). Yes they are mutually exclusive.
-
OH wow, i think i found my issue related to this pfblocker package. How can i just disable it via cli that way i can get into the gui and sort it out later?
-
@gacpac If the GUI is stuck and not responding log in with SSH and choose option 16 "Restart PHP-FPM" - that's what I did when the GUI was stuck after accessing the package manager.
-
G gacpac referenced this topic on
-
Mmm, I didn't try that but that should work because php gets stuck on the uninstall script.
It wouldn't complete the uninstall of the bad package but that may not matter since all the important stuff has already happened by that point.
-
@DBMandrake i tried that and nothing. i happen to be able to restore from backup right before installing pfblocker update, I assume it's because the config has nothing to do with the package itself.
Over cli how do i guess downgrade pfblocker or something between those lines?
-
-
G Gertjan referenced this topic on
-
@Gertjan is removing it the only way, I can't just disable it?
-
You could probably remove the pfBlocker widget from the dashboard.
But killing the script from the CLI when you remove it is not too hard. See: https://forum.netgate.com/post/1185688
