2.3.2_1 crash report



  • Following the upgrade from 2.3.2 to to 2.3.2_1, I'm receiving a crash report on each reboot:

    					Crash report begins.  Anonymous machine information:
    
    amd64
    10.3-RELEASE-p9
    FreeBSD 10.3-RELEASE-p9 #1 5fc1b19(RELENG_2_3_2): Tue Sep 27 12:25:49 CDT 2016     root@factory23-amd64-builder:/builder/factory-232/tmp/obj/builder/factory-232/tmp/FreeBSD-src/sys/pfSense
    
    Crash report details:
    
    PHP Errors:
    [07-Oct-2016 21:30:33 PST8PDT] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/suhosin.so' - /usr/local/lib/php/20131226/suhosin.so: Undefined symbol "ps_globals" in Unknown on line 0
    
    

    This seems to be very similar to a problem I had with the 2.3.3 snapshot series, but in this case this is a SG-4860 unit that has only had release versions on it. No alpha/beta snapshots at all.



  • Since you are going to ask:

    [2.3.2-RELEASE]/root: pkg info -x php
    php-suhosin-0.9.38
    php-xdebug-2.4.0
    php56-5.6.26
    php56-bcmath-5.6.26
    php56-bz2-5.6.26
    php56-ctype-5.6.26
    php56-curl-5.6.26
    php56-dom-5.6.26
    php56-filter-5.6.26
    php56-gettext-5.6.26
    php56-hash-5.6.26
    php56-json-5.6.26
    php56-ldap-5.6.26
    php56-mbstring-5.6.26
    php56-mcrypt-5.6.26
    php56-opcache-5.6.26_1
    php56-openssl-5.6.26
    php56-pcntl-5.6.26
    php56-pdo-5.6.26
    php56-pdo_sqlite-5.6.26
    php56-pfSense-module-0.12
    php56-posix-5.6.26
    php56-readline-5.6.26
    php56-session-5.6.26
    php56-shmop-5.6.26
    php56-simplexml-5.6.26
    php56-sockets-5.6.26
    php56-sqlite3-5.6.26
    php56-sysvmsg-5.6.26
    php56-sysvsem-5.6.26
    php56-sysvshm-5.6.26
    php56-tokenizer-5.6.26
    php56-xml-5.6.26
    php56-xmlreader-5.6.26
    php56-xmlwriter-5.6.26
    php56-zlib-5.6.26
    [2.3.2-RELEASE]/root: 
    
    
    [2.3.2-RELEASE]/root: ls -l /usr/local/lib/php/20131226
    total 3484
    -r--r--r--  1 root  wheel    37912 Sep 27 09:15 bcmath.so
    -r--r--r--  1 root  wheel    22096 Sep 27 09:14 bz2.so
    -r--r--r--  1 root  wheel    13200 Sep 27 09:14 ctype.so
    -r--r--r--  1 root  wheel    82760 Sep 27 09:15 curl.so
    -r--r--r--  1 root  wheel   188336 Sep 27 09:14 dom.so
    -r--r--r--  1 root  wheel    42720 Sep 27 09:15 filter.so
    -r--r--r--  1 root  wheel    12904 Sep 27 09:15 gettext.so
    -r--r--r--  1 root  wheel   169888 Sep 27 09:14 hash.so
    -r--r--r--  1 root  wheel    41896 Sep 27 09:14 json.so
    -r--r--r--  1 root  wheel    59928 Sep 27 09:14 ldap.so
    -r--r--r--  1 root  wheel  1033752 Sep 27 09:15 mbstring.so
    -r--r--r--  1 root  wheel    45688 Sep 27 09:14 mcrypt.so
    -r--r--r--  1 root  wheel   143792 Sep 27 09:15 opcache.so
    -r--r--r--  1 root  wheel   155552 Sep 27 09:15 openssl.so
    -r--r--r--  1 root  wheel    30280 Sep 27 09:14 pcntl.so
    -r--r--r--  1 root  wheel   102144 Sep 27 09:14 pdo.so
    -r--r--r--  1 root  wheel    25208 Sep 27 09:14 pdo_sqlite.so
    -r--r--r--  1 root  wheel    63032 Apr  6  2016 pfSense.so
    -r--r--r--  1 root  wheel    29176 Sep 27 09:14 posix.so
    -r--r--r--  1 root  wheel    46864 Jul 19 09:32 radius.so
    -r--r--r--  1 root  wheel    30536 Sep 27 09:14 readline.so
    -r--r--r--  1 root  wheel    29880 Jul 19 09:30 rrd.so
    -r--r--r--  1 root  wheel    79032 Sep 27 09:14 session.so
    -r--r--r--  1 root  wheel    12040 Sep 27 09:14 shmop.so
    -r--r--r--  1 root  wheel    55040 Sep 27 09:14 simplexml.so
    -r--r--r--  1 root  wheel    83784 Sep 27 09:14 sockets.so
    -r--r--r--  1 root  wheel    46264 Sep 27 09:15 sqlite3.so
    -r--r--r--  1 root  wheel    67432 Apr  6  2016 ssh2.so
    -r--r--r--  1 root  wheel   146144 Apr  6  2016 suhosin.so
    -r--r--r--  1 root  wheel    16600 Sep 27 09:14 sysvmsg.so
    -r--r--r--  1 root  wheel     9240 Sep 27 09:15 sysvsem.so
    -r--r--r--  1 root  wheel    12480 Sep 27 09:14 sysvshm.so
    -r--r--r--  1 root  wheel    20648 Sep 27 09:14 tokenizer.so
    -r--r--r--  1 root  wheel   232904 Jul 19 09:33 xdebug.so
    -r--r--r--  1 root  wheel    53176 Sep 27 09:14 xml.so
    -r--r--r--  1 root  wheel    34584 Sep 27 09:15 xmlreader.so
    -r--r--r--  1 root  wheel    45152 Sep 27 09:14 xmlwriter.so
    -r--r--r--  1 root  wheel    36368 Sep 27 09:14 zlib.so
    -r--r--r--  1 root  wheel    73160 May 16 12:55 zmq.so
    [2.3.2-RELEASE]/root: 
    
    


  • I had to update each of these by hand to get everything up to date in the /usr/local/lib/php/20131226 lib directory:

    pkg upgrade -f php-suhosin-0.9.38
    pkg upgrade -f pecl-ssh2-0.12
    pkg upgrade -f php56-pfSense-module-0.12
    pkg upgrade -f pecl-zmq-1.1.3_1
    pkg upgrade -f pecl-rrd-1.1.3_3
    pkg upgrade -f pecl-radius-1.3.0
    pkg upgrade -f php-xdebug-2.4.0
    
    

    I'm a bit concerned about other potential stale file issues. Is there any way to conduct a system wide validation?

    I'd rather not execute a complete re-install unless absolutely necessary.


  • Rebel Alliance Developer Netgate

    The reinstall likely didn't do exactly what you think. Odds are the module file itself was fine or unchanged, but the reinstall action corrected the entries in extensions.ini so that all the modules were loaded.

    That should be much better on 2.3.3 and 2.4 where the FreeBSD ports tree now uses a much better method of listing extensions that does not rely on us manually maintaining that file.



  • @jimp:

    The reinstall likely didn't do exactly what you think. Odds are the module file itself was fine or unchanged, but the reinstall action corrected the entries in extensions.ini so that all the modules were loaded.

    The module files were actually incorrect. I checksummed them before and after. Also, using suhosin.so as an example, you can see that the file size was incorrect from the listing above: 146144 vs 146640. pfSense.so as another example: 63032 vs 63048.



  • I have another unit with incorrect content in /usr/local/lib/php/20131226. This one is an SG-2220 which also has only had production code since install. This unit is not turning a crash report.

    This output is from a "bad" system:

    
    [2.3.2-RELEASE][root@badsys]/root: pkg which /usr/local/lib/php/20131226/pfSense.so 
    /usr/local/lib/php/20131226/pfSense.so was installed by package php56-pfSense-module-0.12
    [2.3.2-RELEASE][root@badsys]/root: sum /usr/local/lib/php/20131226/pfSense.so
    9569 62 /usr/local/lib/php/20131226/pfSense.so
    [2.3.2-RELEASE][root@badsys]/root: pkg which /usr/local/lib/php/20131226/suhosin.so
    /usr/local/lib/php/20131226/suhosin.so was installed by package php-suhosin-0.9.38
    [2.3.2-RELEASE][root@badsys]/root: sum /usr/local/lib/php/20131226/suhosin.so
    56994 143 /usr/local/lib/php/20131226/suhosin.so
    [2.3.2-RELEASE][root@badsys]/root: 
    
    

    This output if from a "good" system:

    
    [2.3.2-RELEASE][root@goodsys]/root: pkg which /usr/local/lib/php/20131226/pfSense.so
    /usr/local/lib/php/20131226/pfSense.so was installed by package php56-pfSense-module-0.12
    [2.3.2-RELEASE][root@goodsys]/root: sum /usr/local/lib/php/20131226/pfSense.so
    28667 62 /usr/local/lib/php/20131226/pfSense.so
    [2.3.2-RELEASE][root@goodsys]/root: pkg which /usr/local/lib/php/20131226/suhosin.so 
    /usr/local/lib/php/20131226/suhosin.so was installed by package php-suhosin-0.9.38
    [2.3.2-RELEASE][root@goodsys]/root: sum /usr/local/lib/php/20131226/suhosin.so
    53117 144 /usr/local/lib/php/20131226/suhosin.so
    [2.3.2-RELEASE][root@goodsys]/root: 
    
    

    As you can see, both systems believe they have version 0.12 of the pfSense php shared object, but the actual files are different. In this directory, ssh2.so, suhosin.so, pfSense.so, zmq.so, rrd.so, radius.so, xdebug.so are all incorrect.

    I think it's pretty clear that there is a serious problem here. I have left the bad system in the broken state to preserve information. Please let me what I can to do help track this down.


  • Rebel Alliance Developer Netgate

    Does pkg find a problem with either of those?

    pkg check -s | & egrep -v '(Checking all packages|missing file)'
    

    Also, use sha256 (or md5) rather than just "sum" for a more accurate hash.



  • @jimp:

    Does pkg find a problem with either of those?

    pkg check -s | & egrep -v '(Checking all packages|missing file)'
    

    pkg does not report problems with any packages other than expected missing files.


  • Rebel Alliance Developer Netgate

    And "goodsys" and "badsys" are both amd64?



  • @jimp:

    And "goodsys" and "badsys" are both amd64?

    Goodsys is an SG-4860, running factory image. Upgraded with release versions only since install in June 2015. This is the system that was turning the crash report that started this thread, and was fixed by running pkg installs by hand following the upgrade to 2.3.2_1.

    Badsys is an SG-2220, running factory image. Upgraded with release versions only since install in September 2015.


  • Rebel Alliance Developer Netgate

    OK, mostly I wanted to be sure they were running versions that should actually be the same and not different in some way.



  • This should be easy to reproduce: I just pulled a brand new SG-4860 from the box. It came shipped with 2.3.2. I did a config restore with the configuration file download from the old SG-4860. After reboot, I did an upgrade to 2.3.2_1.

    This output is from the old SG-4860 (aka goodsys above):

    [2.3.2-RELEASE][root@goodsys]/root: pkg which /usr/local/lib/php/20131226/pfSense.so 
    /usr/local/lib/php/20131226/pfSense.so was installed by package php56-pfSense-module-0.12
    [2.3.2-RELEASE][root@goodsys]/root: sha1 /usr/local/lib/php/20131226/pfSense.so
    SHA1 (/usr/local/lib/php/20131226/pfSense.so) = f29b07c11d823ca0b5a75113122abcd1925073fe
    [2.3.2-RELEASE][root@goodsys]/root: pkg which /usr/local/lib/php/20131226/suhosin.so 
    /usr/local/lib/php/20131226/suhosin.so was installed by package php-suhosin-0.9.38
    [2.3.2-RELEASE][root@goodsys]/root: sha1 /usr/local/lib/php/20131226/suhosin.so
    SHA1 (/usr/local/lib/php/20131226/suhosin.so) = 531f0ffb1b33b18e3ca41a264589355f1f29e46e
    [2.3.2-RELEASE][root@goodsys]/root: 
    
    

    This output is from the new SG-4860 after the upgrade to 2.3.2_1:

    [2.3.2-RELEASE][root@newsys]/root: pkg which /usr/local/lib/php/20131226/pfSense.so 
    /usr/local/lib/php/20131226/pfSense.so was installed by package php56-pfSense-module-0.12
    [2.3.2-RELEASE][root@newsys]/root: sha1 /usr/local/lib/php/20131226/pfSense.so
    SHA1 (/usr/local/lib/php/20131226/pfSense.so) = 90eecff835a534b878fecfb2086141632e640448
    [2.3.2-RELEASE][root@newsys]/root: pkg which /usr/local/lib/php/20131226/suhosin.so 
    /usr/local/lib/php/20131226/suhosin.so was installed by package php-suhosin-0.9.38
    [2.3.2-RELEASE][root@newsys]/root: sha1 /usr/local/lib/php/20131226/suhosin.so
    SHA1 (/usr/local/lib/php/20131226/suhosin.so) = db27be42a763a4f9386ef4dfc9a72c4076f7a672
    [2.3.2-RELEASE][root@newsys]/root: 
    
    

    In the /usr/local/lib/php/20131226 directory, the following files were not updated by the 2.3.2_1 upgrade: pfSense.so, xdebug.so, zmq.so, radius.so, suhosin.so, rrd.so, ssh2.so. You can easily see it with ls -lt.



  • Jim? Any update?



  • Packages having minor changes between releases but their revision not explicitly bumped?



  • @JorgeOliveira:

    Packages having minor changes between releases but their revision not explicitly bumped?

    The opposite. The package repository shows that the versions have been bumped, but the package files have not been updated.


  • Rebel Alliance Developer Netgate

    Out of town for a few days, unless someone else can test it in the meantime I'll give it a shot when I'm back on Wednesday



  • @jimp:

    Out of town for a few days, unless someone else can test it in the meantime I'll give it a shot when I'm back on Wednesday

    Okay, thanks Jim.





  • The system was updated several times from 2.3 to 2.3.2-p1

    
    [2.3.2-RELEASE][root@pf.net]/root: pkg which /usr/local/lib/php/20131226/pfSense.so
    /usr/local/lib/php/20131226/pfSense.so was installed by package php56-pfSense-module-0.12
    [2.3.2-RELEASE][root@pf.net]/root: sha1 /usr/local/lib/php/20131226/pfSense.so
    SHA1 (/usr/local/lib/php/20131226/pfSense.so) = 90eecff835a534b878fecfb2086141632e640448
    [2.3.2-RELEASE][root@pf.net]/root: pkg which /usr/local/lib/php/20131226/suhosin.so
    /usr/local/lib/php/20131226/suhosin.so was installed by package php-suhosin-0.9.38
    [2.3.2-RELEASE][root@pf.net]/root: sha1 /usr/local/lib/php/20131226/suhosin.so
    SHA1 (/usr/local/lib/php/20131226/suhosin.so) = 3d2cdcbb696ee37d76885918ce805497f329eb17
    
    

    EDIT: 2.3.2-RELEASE-p1 (amd64)
    built on Tue Sep 27 12:13:07 CDT 2016
    FreeBSD 10.3-RELEASE-p9



  • Using the 4860 formally known as goodsys, I did the following:

    1. Perform a fresh install from factory image for 2.3.2
    2. Collect checksums of all files on the system
    3. Perform an on-line upgrade to 2.3.2_1
    4. Collect checksums of all files on the system
    5. Perform a fresh install from factory image for 2.3.2_1
    6. Collect checksums of all files on the system

    Base on the results of steps 4 & 6, I find the following 85 files to be incorrect:

    usr/local/bin/delv
    usr/local/bin/dig
    usr/local/bin/host
    usr/local/bin/nslookup
    usr/local/bin/nsupdate
    usr/local/bin/rrdtool
    usr/local/lib/ipsec/libcharon.a
    usr/local/lib/ipsec/libradius.a
    usr/local/lib/ipsec/libsimaka.a
    usr/local/lib/ipsec/libstrongswan.a
    usr/local/lib/ipsec/libtls.a
    usr/local/lib/ipsec/libvici.a
    usr/local/lib/ipsec/plugins/libstrongswan-addrblock.a
    usr/local/lib/ipsec/plugins/libstrongswan-aes.a
    usr/local/lib/ipsec/plugins/libstrongswan-attr.a
    usr/local/lib/ipsec/plugins/libstrongswan-blowfish.a
    usr/local/lib/ipsec/plugins/libstrongswan-cmac.a
    usr/local/lib/ipsec/plugins/libstrongswan-constraints.a
    usr/local/lib/ipsec/plugins/libstrongswan-curl.a
    usr/local/lib/ipsec/plugins/libstrongswan-curl.so
    usr/local/lib/ipsec/plugins/libstrongswan-des.a
    usr/local/lib/ipsec/plugins/libstrongswan-dnskey.a
    usr/local/lib/ipsec/plugins/libstrongswan-eap-dynamic.a
    usr/local/lib/ipsec/plugins/libstrongswan-eap-identity.a
    usr/local/lib/ipsec/plugins/libstrongswan-eap-md5.a
    usr/local/lib/ipsec/plugins/libstrongswan-eap-mschapv2.a
    usr/local/lib/ipsec/plugins/libstrongswan-eap-peap.a
    usr/local/lib/ipsec/plugins/libstrongswan-eap-radius.a
    usr/local/lib/ipsec/plugins/libstrongswan-eap-sim-file.a
    usr/local/lib/ipsec/plugins/libstrongswan-eap-sim.a
    usr/local/lib/ipsec/plugins/libstrongswan-eap-tls.a
    usr/local/lib/ipsec/plugins/libstrongswan-eap-ttls.a
    usr/local/lib/ipsec/plugins/libstrongswan-fips-prf.a
    usr/local/lib/ipsec/plugins/libstrongswan-hmac.a
    usr/local/lib/ipsec/plugins/libstrongswan-ipseckey.a
    usr/local/lib/ipsec/plugins/libstrongswan-kernel-pfkey.a
    usr/local/lib/ipsec/plugins/libstrongswan-kernel-pfroute.a
    usr/local/lib/ipsec/plugins/libstrongswan-md4.a
    usr/local/lib/ipsec/plugins/libstrongswan-md5.a
    usr/local/lib/ipsec/plugins/libstrongswan-nonce.a
    usr/local/lib/ipsec/plugins/libstrongswan-openssl.a
    usr/local/lib/ipsec/plugins/libstrongswan-pem.a
    usr/local/lib/ipsec/plugins/libstrongswan-pgp.a
    usr/local/lib/ipsec/plugins/libstrongswan-pkcs1.a
    usr/local/lib/ipsec/plugins/libstrongswan-pkcs12.a
    usr/local/lib/ipsec/plugins/libstrongswan-pkcs7.a
    usr/local/lib/ipsec/plugins/libstrongswan-pkcs8.a
    usr/local/lib/ipsec/plugins/libstrongswan-pubkey.a
    usr/local/lib/ipsec/plugins/libstrongswan-random.a
    usr/local/lib/ipsec/plugins/libstrongswan-rc2.a
    usr/local/lib/ipsec/plugins/libstrongswan-resolve.a
    usr/local/lib/ipsec/plugins/libstrongswan-revocation.a
    usr/local/lib/ipsec/plugins/libstrongswan-sha1.a
    usr/local/lib/ipsec/plugins/libstrongswan-sha2.a
    usr/local/lib/ipsec/plugins/libstrongswan-socket-default.a
    usr/local/lib/ipsec/plugins/libstrongswan-sshkey.a
    usr/local/lib/ipsec/plugins/libstrongswan-stroke.a
    usr/local/lib/ipsec/plugins/libstrongswan-unbound.a
    usr/local/lib/ipsec/plugins/libstrongswan-unity.a
    usr/local/lib/ipsec/plugins/libstrongswan-updown.a
    usr/local/lib/ipsec/plugins/libstrongswan-vici.a
    usr/local/lib/ipsec/plugins/libstrongswan-whitelist.a
    usr/local/lib/ipsec/plugins/libstrongswan-x509.a
    usr/local/lib/ipsec/plugins/libstrongswan-xauth-eap.a
    usr/local/lib/ipsec/plugins/libstrongswan-xauth-generic.a
    usr/local/lib/ipsec/plugins/libstrongswan-xcbc.a
    usr/local/lib/libgio-2.0.a
    usr/local/lib/libglib-2.0.a
    usr/local/lib/libgmodule-2.0.a
    usr/local/lib/libgobject-2.0.a
    usr/local/lib/libgthread-2.0.a
    usr/local/lib/librrd.a
    usr/local/lib/php/20131226/pfSense.so
    usr/local/lib/php/20131226/rrd.so
    usr/local/lib/php/20131226/suhosin.so
    usr/local/lib/php/20131226/xdebug.so
    usr/local/lib/php/20131226/zmq.so
    usr/local/sbin/dnssec-dsfromkey
    usr/local/sbin/dnssec-importkey
    usr/local/sbin/dnssec-keyfromlabel
    usr/local/sbin/dnssec-keygen
    usr/local/sbin/dnssec-revoke
    usr/local/sbin/dnssec-settime
    usr/local/sbin/dnssec-signzone
    usr/local/sbin/dnssec-verify
    
    

    Until this is addressed with a new release, I don't think one should trust anything short of a full install.




  • Netgate

    Thanks for the bug report.  Assigned, bumped priority.

    Will get this looked at early tomorrow.



  • Thanks Jim. Much appreciated.


  • Administrator

    Actually it's not a bug, it's expected and it's how pkg is designed to work.

    When we moved to 2.3.2_1 we cherry-picked some package upgrades from FreeBSD ports tree since these upgrades fixes some vulnerabilities listed by 'pkg audit'. Following ports were updated:

    php56
    perl5
    libxml2
    libidn
    curl

    Due to these updates, when poudriere starts to build our ports set, it deletes all packages that depends of above listed packages and rebuild them. At this time, strongswan was rebuilt as many other ports, and a new package with same version was created.

    This new package was included in 2.3.2-p1, so when you install it directly you will see the new package, and checksum differs.

    During upgrade, since any shared library version has bumped, pkg understands packages like strongswan don't need to be reinstalled, because libraries it depends didn't have any ABI changes. Then you end up with the version built for 2.3.2.

    If you compare the built date for strongswan package on both systems you will see this:

    • 2.3.2
    
    [2.3.2-RELEASE][admin@pf232.home]/root: pkg info strongswan
    strongswan-5.5.0
    Name           : strongswan
    Version        : 5.5.0
    Installed on   : Wed Jul 20 15:39:17 2016 UTC
    Origin         : security/strongswan
    Architecture   : freebsd:10:x86:64
    Prefix         : /usr/local
    Categories     : security
    Licenses       : GPLv2
    Maintainer     : strongswan@nanoteq.com
    WWW            : http://www.strongswan.org
    Comment        : Open Source IKEv2 IPsec-based VPN solution
    
    
    • 2.3.2-p1
    
    [2.3.2-RELEASE][admin@pfs232-1.home]/root: pkg info strongswan
    strongswan-5.5.0
    Name           : strongswan
    Version        : 5.5.0
    Installed on   : Mon Oct 17 23:14:33 2016 UTC
    Origin         : security/strongswan
    Architecture   : freebsd:10:x86:64
    Prefix         : /usr/local
    Categories     : security
    Licenses       : GPLv2
    Maintainer     : strongswan@nanoteq.com
    WWW            : http://www.strongswan.org
    Comment        : Open Source IKEv2 IPsec-based VPN solution
    
    


  • Renato, thank you for the write up.

    Does this cover file /usr/local/lib/php/20131226/suhosin.so? This shared object is the one that triggered the original crash report. It is different in 2.3.2_1 install vs upgrade. Was package php-suhosin-0.9.38 also part of the cherry picking?

    Is there another explanation for the crash report?


  • Administrator

    @dennypage:

    Renato, thank you for the write up.

    Does this cover file /usr/local/lib/php/20131226/suhosin.so? This shared object is the one that triggered the original crash report. It is different in 2.3.2_1 install vs upgrade. Was package php-suhosin-0.9.38 also part of the cherry picking?

    Is there another explanation for the crash report?

    php-suhosin was rebuilt because lang/php56 was updated. But the original issue is a problem we faced sometimes on pfSense 2.3. In some hard-to-reproduce circumstances extensions.ini ended up out of order and suhosin was loaded before session, that is a dependency.

    The real fix for this was done in FreeBSD ports and is already on 2.4 and 2.3.3 snapshots. It was reworked the way PHP extensions create their config files to make sure they are loaded in correct order.



  • Thank you Renato, I really appreciate you checking this out and providing such a thorough explanation.

    I am looking forward to the new version of FreeBSD ports.


  • Rebel Alliance Developer Netgate

    Back today and getting caught up. Looks like this was what I suspected to start with, but with the added twist that the same version of the package could have different files. A red herring, but an interesting red herring.

    I, more than most, am happy to see FreeBSD finally have a way to deal with the extension ordering in PHP. I've been ranting about it being broken (and advocating to get a fix in) for over 10 years now (as of yesterday).



  • @jimp:

    I, more than most, am happy to see FreeBSD finally have a way to deal with the extension ordering in PHP. I've been ranting about it being broken (and advocating to get a fix in) for over 10 years now (as of yesterday).

    Happy Anniversary

    :)