Crash, once a day in the morning

[Macom2007]

Hi,

every morning a got a message that a crash occured. It is every time a php script which times out, but not alway the same script. Sample message :

Crash report details:

PHP Errors:
[01-Oct-2012 03:37:28 UTC] PHP Fatal error:  Maximum execution time of 900 seconds exceeded in /etc/inc/notices.inc on line 351

Any ideas how to solve this ?

[jimp]

Those aren't really "crashes" per se, not like there was a panic/reboot.

I'd look in the logs and see what was happening 900 seconds before that timestamp to see what it was doing.

[Macom2007]

I dont got this solved yet. But is has to do with lcdproc_client.php. Once i stop the client, the error is gone. This error happens once a day, after the error lcdproc_client is stopped.

Any ideas ?

Crash report begins.  Anonymous machine information:

i386
8.3-RELEASE-p4
FreeBSD 8.3-RELEASE-p4 #1: Sat Oct 13 14:30:31 EDT 2012    root@snapshots-8_3-i386.builders.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8

Crash report details:

PHP Errors:
[18-Oct-2012 14:07:06 UTC] PHP Fatal error:  Maximum execution time of 900 seconds exceeded in /etc/inc/interfaces.inc on line 68

[phil.davis]

/etc/rc.php_ini_setup sets:

max_execution_time = 900

Normal things done on the GUI, even downloading and installing an update, should never use more than 900 CPU seconds on any platform.
Things like lcdproc are scripts that run forever in the background, so it will clock up 900 seconds of CPU eventually and get terminated by the PHP interpreter.
By adding set_time_limit(0) before loop_status is called in lcdproc_client.php it should be allowed to run forever:

/* Connect to the LCDd port and interface with the LCD */
$lcd = fsockopen(LCDPROC_HOST, LCDPROC_PORT, $errno, $errstr, 10);
if (!$lcd) {
  lcdproc_warn("Failed to connect to LCDd process $errstr ($errno)");
} else {
  set_time_limit(0);
  build_interface($lcd);
  loop_status($lcd);
  /* loop exited? Close fd and wait for the script to kick in again */
  fclose($lcd);
}

(The fragment above is from the version in the lcdproc package - lcdproc-dev is a little different, but same principle.)
Give this a try, then this should be fixed in the lcdproc / lcdproc-dev packages.
Any other packages that use a PHP script as a background process that runs forever will also eventually reach the 900 second limit - they will also need set_time_limit(0) added.

[stephenw10]

Hmm, this is interesting.
The lcdproc (original) package has a separate script, lcdclient.sh, that runs a loop continuously restarting lcdclient.php. In an effort to reduce the complexity this was removed from the dev package. At the time there seemed no reason for it to be there and it seemed like one less thing to worry about while trying to squash some bugs in the startup script. This could have been an error.  ::)
See:
http://forum.pfsense.org/index.php/topic,44034.msg239547/topicseen.html#msg239547

Steve

[phil.davis]

max_execution_time = 900 was only done a few months ago, now on 2.1 stuff needs to be aware of that.
It means that a script that has gone into a loop should eventually get killed by the PHP interpreter when it chews up 900 seconds of CPU time. That might be helpful interactively. After losing control of a remote site, you might be able to get a coffee or 2 (or 3) and then find that the thing that went into a loop has been zapped and the WebGUI responds again.
Anything that needs to run "forever" or for an extended (CPU) time should now specify that by calling set_time_limit - this just needs to be brought to the attention of whoever maintains lcdproc.
The new code in lcdproc-dev looks like it is handling the various error/retry conditions it needs, it just needs to also avoid having the PHP interpreter kill it.

[stephenw10]

Well that's good and bad. The client crashes out on my box under 2.0.1 and this would have been a nice explanation. Back to debugging.  ;)

Steve

[Macom2007]

Your solution works for me. Thanks!

@phil.davis:

/etc/rc.php_ini_setup sets:

max_execution_time = 900

Normal things done on the GUI, even downloading and installing an update, should never use more than 900 CPU seconds on any platform.
Things like lcdproc are scripts that run forever in the background, so it will clock up 900 seconds of CPU eventually and get terminated by the PHP interpreter.
By adding set_time_limit(0) before loop_status is called in lcdproc_client.php it should be allowed to run forever:

/* Connect to the LCDd port and interface with the LCD */
$lcd = fsockopen(LCDPROC_HOST, LCDPROC_PORT, $errno, $errstr, 10);
if (!$lcd) {
  lcdproc_warn("Failed to connect to LCDd process $errstr ($errno)");
} else {
  set_time_limit(0);
  build_interface($lcd);
  loop_status($lcd);
  /* loop exited? Close fd and wait for the script to kick in again */
  fclose($lcd);
}

(The fragment above is from the version in the lcdproc package - lcdproc-dev is a little different, but same principle.)
Give this a try, then this should be fixed in the lcdproc / lcdproc-dev packages.
Any other packages that use a PHP script as a background process that runs forever will also eventually reach the 900 second limit - they will also need set_time_limit(0) added.