"PHP Fatal error: Maximum execution time of 900 seconds exceeded in /etc/inc/util.inc on line 3728" after installing latest pfBlockerNG 3.2.0_12

DBMandrake

Just to add a data point.

Two boxes running CE 2.7.2, one "live" system, one hot spare.

Both were previously on PFBLocker-NG 3.2.0_15, both updated yesterday although not at exactly the same time, the "live" system updated to 3.2.0_17 seemingly without issue, (phew) while the spare system got stuck with the update never completing in the GUI, although after I refreshed the GUI things seemed OK at first and I didn't look any further as it is not a live system.

However I got an alert a while later which ironically was a CPU temperature alert... and logging into the GUI I saw a crash report which included "PHP Fatal error: Maximum execution time of 900 seconds exceeded in /etc/inc/util.inc on line...." although I can't currently get back to the crash log as the Web UI has hung.

Two processes - pfblockerng.php cron and pfSense-pkg-pfBlockerNG-devel POST-INSTALL were stuck on 100% each so fully utilising 2 of the 4 cores on the system. In the htop output below I had tried to go to the package manager in the GUI, this caused the web interface processes to also get stuck in a high CPU loop as well making the Web UI unresponsive:

 4374 root       107   0  140M 51460 R  43.3  0.3 18:39.17 php-fpm: pool nginx
40181 root       106   0  140M 50880 R  42.1  0.3  4:11.17 php-fpm: pool nginx
22471 root       106   0 70596 48840 R  41.1  0.3 23h50:37 //usr/local/bin/php -f //etc/rc.packages pfSense-pkg-pfBlockerNG-devel POST-INSTALL
  399 root       107   0  140M 54144 R  41.0  0.3 19:04.01 php-fpm: pool nginx
81451 root       106   0  140M 50040 R  40.9  0.3  3:30.04 php-fpm: pool nginx
89567 root       107   0 70340 47936 R  40.3  0.3  7h48:53 /usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php cron
46010 root       106   0  140M 50444 R  39.5  0.3 18:28.28 php-fpm: pool nginx
68826 root       106   0  140M 50920 R  39.1  0.3 18:40.88 php-fpm: pool nginx
21846 root       106   0  140M 52448 R  38.8  0.3 18:55.89 php-fpm: pool nginx

@stephenw10 As this is not a live system I can afford to do further forensics on it if it's helpful to figure out what went wrong, before attempting a fix ? If no further info is required to help troubleshoot the cause I will attempt a manual fix.

Gertjan

@DBMandrake

As seen elsewhere :

kill 89567

and then check if process 22471 continues finishing its job.

DBMandrake

@Gertjan Killed. how long should I give it ?

The POST-INSTALL process is still on 100%.

However judging by my CPU graph, the cron process got stuck in a 100% loop AFTER the install process did... the install process was already sitting at 100% CPU for hours before that.

(Cron runs at 3am)

DBMandrake

@DBMandrake Checking system.log I can see I started the update at 10:47am: (UK time)

Sep 24 10:46:58 pfSense3 lighttpd_pfb[4067]: [pfBlockerNG] DNSBL Webserver stopped
Sep 24 10:46:58 pfSense3 tail_pfb[6074]: [pfBlockerNG] Firewall Filter Service stopped
Sep 24 10:46:58 pfSense3 php_pfb[6603]: [pfBlockerNG] filterlog daemon stopped
Sep 24 10:46:58 pfSense3 php[2869]: /etc/rc.packages: Configuration Change: (system): Intermediate config write during package removal for pfBlockerNG-devel.
Sep 24 10:46:58 pfSense3 check_reload_status[439]: Syncing firewall
Sep 24 10:46:58 pfSense3 php[2869]: /etc/rc.packages: Configuration Change: (system): pfBlockerNG: saving Unbound custom options
Sep 24 10:46:58 pfSense3 check_reload_status[439]: Syncing firewall
Sep 24 10:46:58 pfSense3 php[2869]: /etc/rc.packages: Configuration Change: (system): pfBlockerNG: saving DNSBL changes
Sep 24 10:46:58 pfSense3 snort[40410]: Received notification of an IP address change on interface lo0.
Sep 24 10:46:58 pfSense3 snort[40410]: Deleted address 10.10.10.1 from automatic interface IP Pass List for Legacy Blocking Mode.
Sep 24 10:47:00 pfSense3 check_reload_status[439]: Reloading filter
Sep 24 10:47:00 pfSense3 php[2869]: [pfBlockerNG] Restarting firewall filter daemon
Sep 24 10:47:00 pfSense3 php[22471]: //etc/rc.packages: Beginning package installation for pfBlockerNG-devel .
Sep 24 10:47:00 pfSense3 php[22471]: //etc/rc.packages: Configuration Change: (system): Intermediate config write during package install for pfBlockerNG-devel.
Sep 24 10:47:00 pfSense3 check_reload_status[439]: Syncing firewall

And there is NOTHING in the log after this relating to the package installation, in fact the next thing in the log is at 4am the following day from the auto config backup service.

By the way the process 22471 is still on 100% and I don't think it's going to complete...

Gertjan

@DBMandrake

kill also 22471

and as showed by @stephenw10 (see above) :

pkg remove pfSense-pkg-pfBlockerNG

so pfSense doesn't try to re installed the already downloaded package and gets blocked gain.

DBMandrake

@Gertjan It's pkg remove pfSense-pkg-pfBlockerNG-devel as it's the development package.

Manually removing it and reinstalling it has fixed it.

I'm wondering if I should go back to the non-devel version - I've been on the devel version of pfBlockerNG forever because the standard package was a full major version behind for a long time, and was lacking a feature I needed, but they're both 3.2.xx now.

I'm guessing I would have to note down my configuration and manually reconfigure if I did this though as the system would treat this as a different package altogether with its own configuration ?

OpIT GmbH

@stephenw10

THX! This was working for me

TheNarc

@DBMandrake For what it's worth, I got out of this mess by removing the devel version from the command line and installing the non-devel version, and it appears as if all my config from the devel version carried over to non-devel.

DBMandrake

@TheNarc said in "PHP Fatal error: Maximum execution time of 900 seconds exceeded in /etc/inc/util.inc on line 3728" after installing latest pfBlockerNG 3.2.0_12:

@DBMandrake For what it's worth, I got out of this mess by removing the devel version from the command line and installing the non-devel version, and it appears as if all my config from the devel version carried over to non-devel.

Did you install the non-devel version through the GUI package manager or from the command line ?

I would have to check but I think the only feature of pfBlockerNG that I'm actually using is the Google/Youtube/Bing safesearch DNS entries, as its a lot easier to tick a box in pfBlockerNG to achieve this than manually enter (and maintain) dozens of entries in Unbound!

If the config carries over I think I will do this so I'm not on the "bleeding edge". (Although this is the first time I've actually had an issue with the devel version to be fair)

stephenw10

Once you have removed the bad devel pkg by killing the script process you can reinstall either pfBlockerNG version and it should work. And include the existing settings.

DBMandrake

@stephenw10 Is it unusual for two different packages in PFSense to share the same configuration files / backup ? Or are these two packages mutually exclusive so only one can be installed at the same time ? (Using the same configuration file directories)

TheNarc

@DBMandrake I did it via the cli because at the time the GUI was being problematic due to the original devel-version issue.

stephenw10

@DBMandrake said in "PHP Fatal error: Maximum execution time of 900 seconds exceeded in /etc/inc/util.inc on line 3728" after installing latest pfBlockerNG 3.2.0_12:

s it unusual for two different packages in PFSense to share the same configuration files / backup ?

Yes, I'm not sure any other package does (maybe HAProxy-dev). Yes they are mutually exclusive.

gacpac

OH wow, i think i found my issue related to this pfblocker package. How can i just disable it via cli that way i can get into the gui and sort it out later?

DBMandrake

@gacpac If the GUI is stuck and not responding log in with SSH and choose option 16 "Restart PHP-FPM" - that's what I did when the GUI was stuck after accessing the package manager.

stephenw10

Mmm, I didn't try that but that should work because php gets stuck on the uninstall script.

It wouldn't complete the uninstall of the bad package but that may not matter since all the important stuff has already happened by that point.

gacpac

@DBMandrake i tried that and nothing. i happen to be able to restore from backup right before installing pfblocker update, I assume it's because the config has nothing to do with the package itself.

Over cli how do i guess downgrade pfblocker or something between those lines?

Gertjan

@gacpac

To make a long story ... not shorter, go here (the top of the forum thread).
Read ....
And you know what to do

gacpac

@Gertjan is removing it the only way, I can't just disable it?

stephenw10

You could probably remove the pfBlocker widget from the dashboard.

But killing the script from the CLI when you remove it is not too hard. See: https://forum.netgate.com/post/1185688