2.3.2_1 crash report
-
The reinstall likely didn't do exactly what you think. Odds are the module file itself was fine or unchanged, but the reinstall action corrected the entries in extensions.ini so that all the modules were loaded.
That should be much better on 2.3.3 and 2.4 where the FreeBSD ports tree now uses a much better method of listing extensions that does not rely on us manually maintaining that file.
-
The reinstall likely didn't do exactly what you think. Odds are the module file itself was fine or unchanged, but the reinstall action corrected the entries in extensions.ini so that all the modules were loaded.
The module files were actually incorrect. I checksummed them before and after. Also, using suhosin.so as an example, you can see that the file size was incorrect from the listing above: 146144 vs 146640. pfSense.so as another example: 63032 vs 63048.
-
I have another unit with incorrect content in /usr/local/lib/php/20131226. This one is an SG-2220 which also has only had production code since install. This unit is not turning a crash report.
This output is from a "bad" system:
[2.3.2-RELEASE][root@badsys]/root: pkg which /usr/local/lib/php/20131226/pfSense.so /usr/local/lib/php/20131226/pfSense.so was installed by package php56-pfSense-module-0.12 [2.3.2-RELEASE][root@badsys]/root: sum /usr/local/lib/php/20131226/pfSense.so 9569 62 /usr/local/lib/php/20131226/pfSense.so [2.3.2-RELEASE][root@badsys]/root: pkg which /usr/local/lib/php/20131226/suhosin.so /usr/local/lib/php/20131226/suhosin.so was installed by package php-suhosin-0.9.38 [2.3.2-RELEASE][root@badsys]/root: sum /usr/local/lib/php/20131226/suhosin.so 56994 143 /usr/local/lib/php/20131226/suhosin.so [2.3.2-RELEASE][root@badsys]/root:
This output if from a "good" system:
[2.3.2-RELEASE][root@goodsys]/root: pkg which /usr/local/lib/php/20131226/pfSense.so /usr/local/lib/php/20131226/pfSense.so was installed by package php56-pfSense-module-0.12 [2.3.2-RELEASE][root@goodsys]/root: sum /usr/local/lib/php/20131226/pfSense.so 28667 62 /usr/local/lib/php/20131226/pfSense.so [2.3.2-RELEASE][root@goodsys]/root: pkg which /usr/local/lib/php/20131226/suhosin.so /usr/local/lib/php/20131226/suhosin.so was installed by package php-suhosin-0.9.38 [2.3.2-RELEASE][root@goodsys]/root: sum /usr/local/lib/php/20131226/suhosin.so 53117 144 /usr/local/lib/php/20131226/suhosin.so [2.3.2-RELEASE][root@goodsys]/root:
As you can see, both systems believe they have version 0.12 of the pfSense php shared object, but the actual files are different. In this directory, ssh2.so, suhosin.so, pfSense.so, zmq.so, rrd.so, radius.so, xdebug.so are all incorrect.
I think it's pretty clear that there is a serious problem here. I have left the bad system in the broken state to preserve information. Please let me what I can to do help track this down.
-
Does pkg find a problem with either of those?
pkg check -s | & egrep -v '(Checking all packages|missing file)'
Also, use sha256 (or md5) rather than just "sum" for a more accurate hash.
-
Does pkg find a problem with either of those?
pkg check -s | & egrep -v '(Checking all packages|missing file)'
pkg does not report problems with any packages other than expected missing files.
-
And "goodsys" and "badsys" are both amd64?
-
And "goodsys" and "badsys" are both amd64?
Goodsys is an SG-4860, running factory image. Upgraded with release versions only since install in June 2015. This is the system that was turning the crash report that started this thread, and was fixed by running pkg installs by hand following the upgrade to 2.3.2_1.
Badsys is an SG-2220, running factory image. Upgraded with release versions only since install in September 2015.
-
OK, mostly I wanted to be sure they were running versions that should actually be the same and not different in some way.
-
This should be easy to reproduce: I just pulled a brand new SG-4860 from the box. It came shipped with 2.3.2. I did a config restore with the configuration file download from the old SG-4860. After reboot, I did an upgrade to 2.3.2_1.
This output is from the old SG-4860 (aka goodsys above):
[2.3.2-RELEASE][root@goodsys]/root: pkg which /usr/local/lib/php/20131226/pfSense.so /usr/local/lib/php/20131226/pfSense.so was installed by package php56-pfSense-module-0.12 [2.3.2-RELEASE][root@goodsys]/root: sha1 /usr/local/lib/php/20131226/pfSense.so SHA1 (/usr/local/lib/php/20131226/pfSense.so) = f29b07c11d823ca0b5a75113122abcd1925073fe [2.3.2-RELEASE][root@goodsys]/root: pkg which /usr/local/lib/php/20131226/suhosin.so /usr/local/lib/php/20131226/suhosin.so was installed by package php-suhosin-0.9.38 [2.3.2-RELEASE][root@goodsys]/root: sha1 /usr/local/lib/php/20131226/suhosin.so SHA1 (/usr/local/lib/php/20131226/suhosin.so) = 531f0ffb1b33b18e3ca41a264589355f1f29e46e [2.3.2-RELEASE][root@goodsys]/root:
This output is from the new SG-4860 after the upgrade to 2.3.2_1:
[2.3.2-RELEASE][root@newsys]/root: pkg which /usr/local/lib/php/20131226/pfSense.so /usr/local/lib/php/20131226/pfSense.so was installed by package php56-pfSense-module-0.12 [2.3.2-RELEASE][root@newsys]/root: sha1 /usr/local/lib/php/20131226/pfSense.so SHA1 (/usr/local/lib/php/20131226/pfSense.so) = 90eecff835a534b878fecfb2086141632e640448 [2.3.2-RELEASE][root@newsys]/root: pkg which /usr/local/lib/php/20131226/suhosin.so /usr/local/lib/php/20131226/suhosin.so was installed by package php-suhosin-0.9.38 [2.3.2-RELEASE][root@newsys]/root: sha1 /usr/local/lib/php/20131226/suhosin.so SHA1 (/usr/local/lib/php/20131226/suhosin.so) = db27be42a763a4f9386ef4dfc9a72c4076f7a672 [2.3.2-RELEASE][root@newsys]/root:
In the /usr/local/lib/php/20131226 directory, the following files were not updated by the 2.3.2_1 upgrade: pfSense.so, xdebug.so, zmq.so, radius.so, suhosin.so, rrd.so, ssh2.so. You can easily see it with ls -lt.
-
Jim? Any update?
-
Packages having minor changes between releases but their revision not explicitly bumped?
-
Packages having minor changes between releases but their revision not explicitly bumped?
The opposite. The package repository shows that the versions have been bumped, but the package files have not been updated.
-
Out of town for a few days, unless someone else can test it in the meantime I'll give it a shot when I'm back on Wednesday
-
Out of town for a few days, unless someone else can test it in the meantime I'll give it a shot when I'm back on Wednesday
Okay, thanks Jim.
-
https://forum.pfsense.org/index.php?topic=112543.0 is this related?
-
The system was updated several times from 2.3 to 2.3.2-p1
[2.3.2-RELEASE][root@pf.net]/root: pkg which /usr/local/lib/php/20131226/pfSense.so /usr/local/lib/php/20131226/pfSense.so was installed by package php56-pfSense-module-0.12 [2.3.2-RELEASE][root@pf.net]/root: sha1 /usr/local/lib/php/20131226/pfSense.so SHA1 (/usr/local/lib/php/20131226/pfSense.so) = 90eecff835a534b878fecfb2086141632e640448 [2.3.2-RELEASE][root@pf.net]/root: pkg which /usr/local/lib/php/20131226/suhosin.so /usr/local/lib/php/20131226/suhosin.so was installed by package php-suhosin-0.9.38 [2.3.2-RELEASE][root@pf.net]/root: sha1 /usr/local/lib/php/20131226/suhosin.so SHA1 (/usr/local/lib/php/20131226/suhosin.so) = 3d2cdcbb696ee37d76885918ce805497f329eb17
EDIT: 2.3.2-RELEASE-p1 (amd64)
built on Tue Sep 27 12:13:07 CDT 2016
FreeBSD 10.3-RELEASE-p9 -
Using the 4860 formally known as goodsys, I did the following:
1. Perform a fresh install from factory image for 2.3.2
2. Collect checksums of all files on the system
3. Perform an on-line upgrade to 2.3.2_1
4. Collect checksums of all files on the system
5. Perform a fresh install from factory image for 2.3.2_1
6. Collect checksums of all files on the systemBase on the results of steps 4 & 6, I find the following 85 files to be incorrect:
usr/local/bin/delv usr/local/bin/dig usr/local/bin/host usr/local/bin/nslookup usr/local/bin/nsupdate usr/local/bin/rrdtool usr/local/lib/ipsec/libcharon.a usr/local/lib/ipsec/libradius.a usr/local/lib/ipsec/libsimaka.a usr/local/lib/ipsec/libstrongswan.a usr/local/lib/ipsec/libtls.a usr/local/lib/ipsec/libvici.a usr/local/lib/ipsec/plugins/libstrongswan-addrblock.a usr/local/lib/ipsec/plugins/libstrongswan-aes.a usr/local/lib/ipsec/plugins/libstrongswan-attr.a usr/local/lib/ipsec/plugins/libstrongswan-blowfish.a usr/local/lib/ipsec/plugins/libstrongswan-cmac.a usr/local/lib/ipsec/plugins/libstrongswan-constraints.a usr/local/lib/ipsec/plugins/libstrongswan-curl.a usr/local/lib/ipsec/plugins/libstrongswan-curl.so usr/local/lib/ipsec/plugins/libstrongswan-des.a usr/local/lib/ipsec/plugins/libstrongswan-dnskey.a usr/local/lib/ipsec/plugins/libstrongswan-eap-dynamic.a usr/local/lib/ipsec/plugins/libstrongswan-eap-identity.a usr/local/lib/ipsec/plugins/libstrongswan-eap-md5.a usr/local/lib/ipsec/plugins/libstrongswan-eap-mschapv2.a usr/local/lib/ipsec/plugins/libstrongswan-eap-peap.a usr/local/lib/ipsec/plugins/libstrongswan-eap-radius.a usr/local/lib/ipsec/plugins/libstrongswan-eap-sim-file.a usr/local/lib/ipsec/plugins/libstrongswan-eap-sim.a usr/local/lib/ipsec/plugins/libstrongswan-eap-tls.a usr/local/lib/ipsec/plugins/libstrongswan-eap-ttls.a usr/local/lib/ipsec/plugins/libstrongswan-fips-prf.a usr/local/lib/ipsec/plugins/libstrongswan-hmac.a usr/local/lib/ipsec/plugins/libstrongswan-ipseckey.a usr/local/lib/ipsec/plugins/libstrongswan-kernel-pfkey.a usr/local/lib/ipsec/plugins/libstrongswan-kernel-pfroute.a usr/local/lib/ipsec/plugins/libstrongswan-md4.a usr/local/lib/ipsec/plugins/libstrongswan-md5.a usr/local/lib/ipsec/plugins/libstrongswan-nonce.a usr/local/lib/ipsec/plugins/libstrongswan-openssl.a usr/local/lib/ipsec/plugins/libstrongswan-pem.a usr/local/lib/ipsec/plugins/libstrongswan-pgp.a usr/local/lib/ipsec/plugins/libstrongswan-pkcs1.a usr/local/lib/ipsec/plugins/libstrongswan-pkcs12.a usr/local/lib/ipsec/plugins/libstrongswan-pkcs7.a usr/local/lib/ipsec/plugins/libstrongswan-pkcs8.a usr/local/lib/ipsec/plugins/libstrongswan-pubkey.a usr/local/lib/ipsec/plugins/libstrongswan-random.a usr/local/lib/ipsec/plugins/libstrongswan-rc2.a usr/local/lib/ipsec/plugins/libstrongswan-resolve.a usr/local/lib/ipsec/plugins/libstrongswan-revocation.a usr/local/lib/ipsec/plugins/libstrongswan-sha1.a usr/local/lib/ipsec/plugins/libstrongswan-sha2.a usr/local/lib/ipsec/plugins/libstrongswan-socket-default.a usr/local/lib/ipsec/plugins/libstrongswan-sshkey.a usr/local/lib/ipsec/plugins/libstrongswan-stroke.a usr/local/lib/ipsec/plugins/libstrongswan-unbound.a usr/local/lib/ipsec/plugins/libstrongswan-unity.a usr/local/lib/ipsec/plugins/libstrongswan-updown.a usr/local/lib/ipsec/plugins/libstrongswan-vici.a usr/local/lib/ipsec/plugins/libstrongswan-whitelist.a usr/local/lib/ipsec/plugins/libstrongswan-x509.a usr/local/lib/ipsec/plugins/libstrongswan-xauth-eap.a usr/local/lib/ipsec/plugins/libstrongswan-xauth-generic.a usr/local/lib/ipsec/plugins/libstrongswan-xcbc.a usr/local/lib/libgio-2.0.a usr/local/lib/libglib-2.0.a usr/local/lib/libgmodule-2.0.a usr/local/lib/libgobject-2.0.a usr/local/lib/libgthread-2.0.a usr/local/lib/librrd.a usr/local/lib/php/20131226/pfSense.so usr/local/lib/php/20131226/rrd.so usr/local/lib/php/20131226/suhosin.so usr/local/lib/php/20131226/xdebug.so usr/local/lib/php/20131226/zmq.so usr/local/sbin/dnssec-dsfromkey usr/local/sbin/dnssec-importkey usr/local/sbin/dnssec-keyfromlabel usr/local/sbin/dnssec-keygen usr/local/sbin/dnssec-revoke usr/local/sbin/dnssec-settime usr/local/sbin/dnssec-signzone usr/local/sbin/dnssec-verify
Until this is addressed with a new release, I don't think one should trust anything short of a full install.
-
Bug report filed: https://redmine.pfsense.org/issues/6858
-
Thanks for the bug report.ย Assigned, bumped priority.
Will get this looked at early tomorrow.
-
Thanks Jim. Much appreciated.