2.3.3 reboot crash report

[dennypage]

The current build of 2.3.3 is drooping a crash report on each reboot:

PHP Errors:
[10-Sep-2016 07:00:18 America/Los_Angeles] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/suhosin.so' - /usr/local/lib/php/20131226/suhosin.so: Undefined symbol "ps_globals" in Unknown on line 0

[jimp]

Hmm, I'm not seeing that on mine, though I didn't run that snapshot I went from one mid-last-week to one from today. Is it gone if you update to a current snapshot now?

[dennypage]

I'm still seeing it with the current build:

Crash report begins.  Anonymous machine information:

amd64
10.3-RELEASE-p7
FreeBSD 10.3-RELEASE-p7 #256 9156c9a(RELENG_2_3): Mon Sep 12 02:29:37 CDT 2016    root@ce23-amd64-builder:/builder/pfsense/tmp/obj/builder/pfsense/tmp/FreeBSD-src/sys/pfSense

Crash report details:

PHP Errors:
[12-Sep-2016 08:30:35 America/Los_Angeles] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/suhosin.so' - /usr/local/lib/php/20131226/suhosin.so: Undefined symbol "ps_globals" in Unknown on line 0

[jimp]

What does your extensions.ini look like?

: cat /usr/local/etc/php/extensions.ini 
extension=bcmath.so
extension=bz2.so
extension=ctype.so
extension=curl.so
extension=dom.so
extension=filter.so
extension=gettext.so
extension=hash.so
extension=json.so
extension=ldap.so
extension=mbstring.so
extension=mcrypt.so
extension=openssl.so
extension=pcntl.so
extension=pdo.so
extension=pdo_sqlite.so
extension=pfSense.so
extension=posix.so
extension=radius.so
extension=readline.so
extension=rrd.so
extension=session.so
extension=shmop.so
extension=simplexml.so
extension=sockets.so
extension=sqlite3.so
extension=ssh2.so
extension=suhosin.so
extension=sysvmsg.so
extension=sysvsem.so
extension=sysvshm.so
extension=tokenizer.so
extension=xml.so
extension=xmlreader.so
extension=xmlwriter.so
extension=zlib.so
extension=zmq.so
zend_extension=/usr/local/lib/php/20131226/opcache.so
zend_extension=/usr/local/lib/php/20131226/xdebug.so

Is suhosin listed twice, perhaps?

[dennypage]

/root: cat /usr/local/etc/php/extensions.ini 
extension=bcmath.so
extension=bz2.so
extension=ctype.so
extension=curl.so
extension=dom.so
extension=filter.so
extension=gettext.so
extension=hash.so
extension=json.so
extension=ldap.so
extension=mbstring.so
extension=mcrypt.so
extension=openssl.so
extension=pcntl.so
extension=pdo.so
extension=pdo_sqlite.so
extension=posix.so
extension=readline.so
extension=session.so
extension=shmop.so
extension=simplexml.so
extension=sockets.so
extension=sqlite3.so
extension=sysvmsg.so
extension=sysvsem.so
extension=sysvshm.so
extension=tokenizer.so
extension=xml.so
extension=xmlreader.so
extension=xmlwriter.so
extension=zlib.so
zend_extension=/usr/local/lib/php/20131226/opcache.so
/root:

[dennypage]

I note that suhosin.so hasn't changed since Jan 16

/root: ls -l /usr/local/lib/php/20131226
total 3484
-r--r--r--  1 root  wheel    37912 Sep  8 13:24 bcmath.so
-r--r--r--  1 root  wheel    22096 Sep  8 13:24 bz2.so
-r--r--r--  1 root  wheel    13200 Sep  8 13:25 ctype.so
-r--r--r--  1 root  wheel    82744 Sep  8 13:24 curl.so
-r--r--r--  1 root  wheel   188336 Sep  8 13:25 dom.so
-r--r--r--  1 root  wheel    42720 Sep  8 13:24 filter.so
-r--r--r--  1 root  wheel    12904 Sep  8 13:25 gettext.so
-r--r--r--  1 root  wheel   169888 Sep  8 13:25 hash.so
-r--r--r--  1 root  wheel    41832 Sep  8 13:24 json.so
-r--r--r--  1 root  wheel    59824 Sep  8 13:25 ldap.so
-r--r--r--  1 root  wheel  1033720 Sep  8 13:25 mbstring.so
-r--r--r--  1 root  wheel    45688 Sep  8 13:25 mcrypt.so
-r--r--r--  1 root  wheel   143792 Sep  8 13:24 opcache.so
-r--r--r--  1 root  wheel   155552 Sep  8 13:25 openssl.so
-r--r--r--  1 root  wheel    30280 Sep  8 13:24 pcntl.so
-r--r--r--  1 root  wheel   102144 Sep  8 13:24 pdo.so
-r--r--r--  1 root  wheel    25208 Sep  8 13:25 pdo_sqlite.so
-r--r--r--  1 root  wheel    63032 Feb 11  2016 pfSense.so
-r--r--r--  1 root  wheel    29176 Sep  8 13:24 posix.so
-r--r--r--  1 root  wheel    46864 Jul  5 14:39 radius.so
-r--r--r--  1 root  wheel    30536 Sep  8 13:24 readline.so
-r--r--r--  1 root  wheel    29880 Jul  5 14:40 rrd.so
-r--r--r--  1 root  wheel    79032 Sep  8 13:24 session.so
-r--r--r--  1 root  wheel    12040 Sep  8 13:25 shmop.so
-r--r--r--  1 root  wheel    55040 Sep  8 13:25 simplexml.so
-r--r--r--  1 root  wheel    83784 Sep  8 13:24 sockets.so
-r--r--r--  1 root  wheel    46264 Sep  8 13:25 sqlite3.so
-r--r--r--  1 root  wheel    67432 Jan  8  2016 ssh2.so
-r--r--r--  1 root  wheel   146136 Jan  8  2016 suhosin.so
-r--r--r--  1 root  wheel    16600 Sep  8 13:24 sysvmsg.so
-r--r--r--  1 root  wheel     9240 Sep  8 13:25 sysvsem.so
-r--r--r--  1 root  wheel    12480 Sep  8 13:24 sysvshm.so
-r--r--r--  1 root  wheel    20648 Sep  8 13:25 tokenizer.so
-r--r--r--  1 root  wheel   232904 Jul  5 14:41 xdebug.so
-r--r--r--  1 root  wheel    53000 Sep  8 13:24 xml.so
-r--r--r--  1 root  wheel    34584 Sep  8 13:25 xmlreader.so
-r--r--r--  1 root  wheel    45152 Sep  8 13:24 xmlwriter.so
-r--r--r--  1 root  wheel    36144 Sep  8 13:24 zlib.so
-r--r--r--  1 root  wheel    73160 May  4 09:30 zmq.so
[2.3.3-DEVELOPMENT][root@test.cococafe.com]/root: 

[jimp]

Strange that it's not listed in there at all. Do you have a pkg installed for it?

: pkg info -x php
php-suhosin-0.9.38
php-xdebug-2.4.0
php56-5.6.25
php56-bcmath-5.6.25
php56-bz2-5.6.25
php56-ctype-5.6.25
php56-curl-5.6.25
php56-dom-5.6.25
php56-filter-5.6.25
php56-gettext-5.6.25
php56-hash-5.6.25
php56-json-5.6.25
php56-ldap-5.6.25
php56-mbstring-5.6.25
php56-mcrypt-5.6.25
php56-opcache-5.6.25_1
php56-openssl-5.6.25
php56-pcntl-5.6.25
php56-pdo-5.6.25
php56-pdo_sqlite-5.6.25
php56-pfSense-module-0.12
php56-posix-5.6.25
php56-readline-5.6.25
php56-session-5.6.25
php56-shmop-5.6.25
php56-simplexml-5.6.25
php56-sockets-5.6.25
php56-sqlite3-5.6.25
php56-sysvmsg-5.6.25
php56-sysvsem-5.6.25
php56-sysvshm-5.6.25
php56-tokenizer-5.6.25
php56-xml-5.6.25
php56-xmlreader-5.6.25
php56-xmlwriter-5.6.25
php56-zlib-5.6.25

: ls -l /usr/local/lib/php/20131226/
total 3484
-r--r--r--  1 root  wheel    37912 Sep  8 16:24 bcmath.so
-r--r--r--  1 root  wheel    22096 Sep  8 16:24 bz2.so
-r--r--r--  1 root  wheel    13200 Sep  8 16:25 ctype.so
-r--r--r--  1 root  wheel    82744 Sep  8 16:24 curl.so
-r--r--r--  1 root  wheel   188336 Sep  8 16:25 dom.so
-r--r--r--  1 root  wheel    42720 Sep  8 16:24 filter.so
-r--r--r--  1 root  wheel    12904 Sep  8 16:25 gettext.so
-r--r--r--  1 root  wheel   169888 Sep  8 16:25 hash.so
-r--r--r--  1 root  wheel    41832 Sep  8 16:24 json.so
-r--r--r--  1 root  wheel    59824 Sep  8 16:25 ldap.so
-r--r--r--  1 root  wheel  1033720 Sep  8 16:25 mbstring.so
-r--r--r--  1 root  wheel    45688 Sep  8 16:25 mcrypt.so
-r--r--r--  1 root  wheel   143792 Sep  8 16:24 opcache.so
-r--r--r--  1 root  wheel   155552 Sep  8 16:25 openssl.so
-r--r--r--  1 root  wheel    30280 Sep  8 16:24 pcntl.so
-r--r--r--  1 root  wheel   102144 Sep  8 16:24 pdo.so
-r--r--r--  1 root  wheel    25208 Sep  8 16:25 pdo_sqlite.so
-r--r--r--  1 root  wheel    63032 Aug 29 08:07 pfSense.so
-r--r--r--  1 root  wheel    29176 Sep  8 16:24 posix.so
-r--r--r--  1 root  wheel    46864 Aug 29 08:06 radius.so
-r--r--r--  1 root  wheel    30536 Sep  8 16:24 readline.so
-r--r--r--  1 root  wheel    29880 Aug 29 08:06 rrd.so
-r--r--r--  1 root  wheel    79032 Sep  8 16:24 session.so
-r--r--r--  1 root  wheel    12040 Sep  8 16:25 shmop.so
-r--r--r--  1 root  wheel    55040 Sep  8 16:25 simplexml.so
-r--r--r--  1 root  wheel    83784 Sep  8 16:24 sockets.so
-r--r--r--  1 root  wheel    46264 Sep  8 16:25 sqlite3.so
-r--r--r--  1 root  wheel    67432 Aug 29 08:07 ssh2.so
-r--r--r--  1 root  wheel   146384 Aug 29 08:07 suhosin.so
-r--r--r--  1 root  wheel    16600 Sep  8 16:24 sysvmsg.so
-r--r--r--  1 root  wheel     9240 Sep  8 16:25 sysvsem.so
-r--r--r--  1 root  wheel    12480 Sep  8 16:24 sysvshm.so
-r--r--r--  1 root  wheel    20648 Sep  8 16:25 tokenizer.so
-r--r--r--  1 root  wheel   232904 Aug 29 08:06 xdebug.so
-r--r--r--  1 root  wheel    53000 Sep  8 16:24 xml.so
-r--r--r--  1 root  wheel    34584 Sep  8 16:25 xmlreader.so
-r--r--r--  1 root  wheel    45152 Sep  8 16:24 xmlwriter.so
-r--r--r--  1 root  wheel    36144 Sep  8 16:24 zlib.so
-r--r--r--  1 root  wheel    73160 Aug 29 08:06 zmq.so

[dennypage]

@jimp:

Strange that it's not listed in there at all. Do you have a pkg installed for it?

Very good question. I have several packages installed:

Cron	         0.3.7	 
mailreport	 3.0_1	 
mtr-nox11	 0.85.6_1	 
Notes	 0.2.9_2	 
ntopng	 0.8.4_4	 
nut	         2.7.4_2	  
Service_Watchdog	 1.8.3	 
Status_Traffic_Totals	 1.2_1	 
System_Patches

I'll prune some packages and see what happens.

[jimp]

Those packages wouldn't help/hurt, I was more interested in the "pkg info -x php" command output I showed above.

[dennypage]

Sorry, here it is:

/root: pkg info -x php
php-suhosin-0.9.38
php-xdebug-2.4.0
php56-5.6.25
php56-bcmath-5.6.25
php56-bz2-5.6.25
php56-ctype-5.6.25
php56-curl-5.6.25
php56-dom-5.6.25
php56-filter-5.6.25
php56-gettext-5.6.25
php56-hash-5.6.25
php56-json-5.6.25
php56-ldap-5.6.25
php56-mbstring-5.6.25
php56-mcrypt-5.6.25
php56-opcache-5.6.25_1
php56-openssl-5.6.25
php56-pcntl-5.6.25
php56-pdo-5.6.25
php56-pdo_sqlite-5.6.25
php56-pfSense-module-0.12
php56-posix-5.6.25
php56-readline-5.6.25
php56-session-5.6.25
php56-shmop-5.6.25
php56-simplexml-5.6.25
php56-sockets-5.6.25
php56-sqlite3-5.6.25
php56-sysvmsg-5.6.25
php56-sysvsem-5.6.25
php56-sysvshm-5.6.25
php56-tokenizer-5.6.25
php56-xml-5.6.25
php56-xmlreader-5.6.25
php56-xmlwriter-5.6.25
php56-zlib-5.6.25
/root: 

[jimp]

Strange that your suhosin.so is a different file size from mine, though the version and architecture match.

Try reinstalling it:

pkg upgrade -f php-suhosin-0.9.38

[dennypage]

Reinstalling suhosin fixes the crash. I wonder how it got out of sync?

Looking at the directory, it seems there are others (xdebug, rrd, radius, zmq, pfSense, ssh2) that are out of sync as well. Is there anything that you would like me to check on before I force a reinstall?

[jimp]

Nothing comes to mind. Was this box freshly installed with 2.3.3 or was it upgrade from 2.3.2?

[dennypage]

Upgraded from 2.3.2. It's a test system that follows prerelease snapshots of the 2.3.X series.

[dennypage]

And thanks for all the quick responses Jim. Much appreciated.

[dennypage]

This issue has appeared again with the 2.3.2_1 release:

https://forum.pfsense.org/index.php?topic=119344.0
https://forum.pfsense.org/index.php?topic=119298.0

In my case, this was a production SG-4860, never used for dev/test. Just continually updated with release versions since install June 2015. I think there is something here.

It would be great if there were a way to perform a system wide package validation or force a re-install of all packages.

[brianc69]

I believe there is a reinstall all packages in one of the menus.

[dennypage]

Are you are thinking of the higher level pfSense packages, such as pfBlockerNG, nut, ntopng, etc., which are what you would find in the menu system? I'm referring to the 150+ lower level FreeBSD packages that pfSense is built upon.