25.11: broken PHP on Netgate 7100

sgw

Tried an upgrade from 25.07.1 to 25.11 on a netgate 7100.

Started in the CLI, it pulled and installed packages, reported to reboot, didn't reboot.

Rebooted, started again.

It seems it's on FreeBSD 16 already, but the packages are somehow inconsistent. Tried to upgrade them via ssh, right nowI can't ssh in anymore:

ld-elf.so.1: /usr/local/lib/libpcre2-8.so.0: version PCRE2_10.47 required by /usr/local/bin/php-cgi not defined
*** *** *** *** *** ***

Password MUST be changed.
If the password has already been changed in the GUI, press ctrl-c to cancel.
*** *** *** *** *** ***
ld-elf.so.1: /usr/local/lib/libpcre2-8.so.0: version PCRE2_10.47 required by /usr/local/bin/php-cgi not defined
ld-elf.so.1: /usr/local/lib/libpcre2-8.so.0: version PCRE2_10.47 required by /usr/local/bin/php-cgi not defined
Connection to x.yz closed.

I have GUI access. How can I fix this via "Command" in the GUI maybe?

sgw

I assume I could proceed by reinstalling pcre2 or so.

I restarted unbound etc, the functionality of network, VPN and DHCP works so far.

As mentioned, no ssh. And I am remote right now.
So I think of fixing pcre2 (is that the name?) via WebGUI and "Command". Unfortunately that seems to have DNS issues (just a guess):

Updating FreeBSD-ports repository catalogue...
FreeBSD-ports repository is up to date.
Updating FreeBSD-ports-kmods repository catalogue...
FreeBSD-ports-kmods repository is up to date.
Updating pfSense-core repository catalogue...
pkg: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-core/meta.conf: No error
pkg: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-core/meta.txz: No error
repository pfSense-core has no meta file, using default settings
pkg: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-core/data.pkg: No error
pkg: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-core/data.tzst: No error
pkg: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-core/packagesite.pkg: No error
pkg: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-core/packagesite.tzst: No error
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
pkg: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-pfSense_plus_v25_11/meta.conf: No error
pkg: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-pfSense_plus_v25_11/meta.txz: No error
repository pfSense has no meta file, using default settings
pkg: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-pfSense_plus_v25_11/data.pkg: No error
pkg: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-pfSense_plus_v25_11/data.tzst: No error
pkg: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-pfSense_plus_v25_11/packagesite.pkg: No error
pkg: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-pfSense_plus_v25_11/packagesite.tzst: No error
Unable to update repository pfSense
Error updating repositories!

It's late here, networking works, so I leave it for now and wait for ideas or suggestions. I might drive there tomorrow for serial console access.

sgw

I assume I could somehow log in via ssh and avoid the pfSense admin-menu? straight ssh works for my non-admin user, but unfortunately "sudo" does not.

sgw

I try my luck by using

ssh admin@pfsense pkg-static ... etc

Tried to push libpcre2-8.so.0 into the system manually, so far I have no luck pulling it from somewhere (target system is a Intel Atom).

sgw

See the following Crash report from the GUI.
off to bed now

Crash report begins.  Anonymous machine information:

amd64
16.0-CURRENT
FreeBSD 16.0-CURRENT #41 plus-RELENG_25_11-n256500-a43915e7ef55: Mon Dec  1 19:02:15 UTC 2025     root@pfsense-build-release-amd64-1.eng.atx.netgate.com:/var/jenkins/workspace/pfSense-Plus-snapshots-25_11-main/obj/amd64/mjYGPXLl/var/jenkins/workspace/pfSe

Crash report details:

PHP Errors:
[16-Dec-2025 19:08:00 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20230831/intl.so (Shared object "libicuio.so.74" not found, required by "intl.so"), /usr/local/lib/php/20230831/intl.so.so (Cannot open "/usr/local/lib/php/20230831/intl.so.so")) in Unknown on line 0
[16-Dec-2025 19:08:12 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20230831/intl.so (Shared object "libicuio.so.74" not found, required by "intl.so"), /usr/local/lib/php/20230831/intl.so.so (Cannot open "/usr/local/lib/php/20230831/intl.so.so")) in Unknown on line 0
[16-Dec-2025 19:08:29 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20230831/intl.so (Shared object "libicuio.so.74" not found, required by "intl.so"), /usr/local/lib/php/20230831/intl.so.so (Cannot open "/usr/local/lib/php/20230831/intl.so.so")) in Unknown on line 0
[16-Dec-2025 19:08:48 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20230831/intl.so (Shared object "libicuio.so.74" not found, required by "intl.so"), /usr/local/lib/php/20230831/intl.so.so (Cannot open "/usr/local/lib/php/20230831/intl.so.so")) in Unknown on line 0
[16-Dec-2025 19:08:49 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20230831/intl.so (Shared object "libicuio.so.74" not found, required by "intl.so"), /usr/local/lib/php/20230831/intl.so.so (Cannot open "/usr/local/lib/php/20230831/intl.so.so")) in Unknown on line 0
[16-Dec-2025 19:08:55 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20230831/intl.so (Shared object "libicuio.so.74" not found, required by "intl.so"), /usr/local/lib/php/20230831/intl.so.so (Cannot open "/usr/local/lib/php/20230831/intl.so.so")) in Unknown on line 0
[16-Dec-2025 19:08:56 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20230831/intl.so (Shared object "libicuio.so.74" not found, required by "intl.so"), /usr/local/lib/php/20230831/intl.so.so (Cannot open "/usr/local/lib/php/20230831/intl.so.so")) in Unknown on line 0
[16-Dec-2025 19:08:58 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20230831/intl.so (Shared object "libicuio.so.74" not found, required by "intl.so"), /usr/local/lib/php/20230831/intl.so.so (Cannot open "/usr/local/lib/php/20230831/intl.so.so")) in Unknown on line 0
[16-Dec-2025 19:08:58 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20230831/intl.so (Shared object "libicuio.so.74" not found, required by "intl.so"), /usr/local/lib/php/20230831/intl.so.so (Cannot open "/usr/local/lib/php/20230831/intl.so.so")) in Unknown on line 0
[16-Dec-2025 19:09:00 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20230831/intl.so (Shared object "libicuio.so.74" not found, required by "intl.so"), /usr/local/lib/php/20230831/intl.so.so (Cannot open "/usr/local/lib/php/20230831/intl.so.so")) in Unknown on line 0
[16-Dec-2025 19:09:03 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20230831/intl.so (Shared object "libicuio.so.74" not found, required by "intl.so"), /usr/local/lib/php/20230831/intl.so.so (Cannot open "/usr/local/lib/php/20230831/intl.so.so")) in Unknown on line 0
[16-Dec-2025 19:09:17 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20230831/intl.so (Shared object "libicuio.so.74" not found, required by "intl.so"), /usr/local/lib/php/20230831/intl.so.so (Cannot open "/usr/local/lib/php/20230831/intl.so.so")) in Unknown on line 0
[16-Dec-2025 19:09:46 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20230831/intl.so (Shared object "libicuio.so.74" not found, required by "intl.so"), /usr/local/lib/php/20230831/intl.so.so (Cannot open "/usr/local/lib/php/20230831/intl.so.so")) in Unknown on line 0
[16-Dec-2025 19:09:50 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20230831/intl.so (Shared object "libicuio.so.74" not found, required by "intl.so"), /usr/local/lib/php/20230831/intl.so.so (Cannot open "/usr/local/lib/php/20230831/intl.so.so")) in Unknown on line 0
[16-Dec-2025 19:09:51 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20230831/intl.so (Shared object "libicuio.so.74" not found, required by "intl.so"), /usr/local/lib/php/20230831/intl.so.so (Cannot open "/usr/local/lib/php/20230831/intl.so.so")) in Unknown on line 0
[16-Dec-2025 19:10:00 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20230831/intl.so (Shared object "libicuio.so.74" not found, required by "intl.so"), /usr/local/lib/php/20230831/intl.so.so (Cannot open "/usr/local/lib/php/20230831/intl.so.so")) in Unknown on line 0
[16-Dec-2025 19:10:00 UTC] PHP Warning:  PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20230831/intl.so (Shared object "libicuio.so.74" not found, required by "intl.so"), /usr/local/lib/php/20230831/intl.so.so (Cannot open "/usr/local/lib/php/20230831/intl.so.so")) in Unknown on line 0



No FreeBSD crash data found.

stephenw10

Can you roll back a ZFS BE snapshot and try the upgrade again?

Are you able to access the console locally?

You could try a full forced pkg reinstall from the gui command prompt but it's a risk if you don;t have other access.

sgw

@stephenw10

Unfortunately no BEs: this box was pre-installed on UFS back then.

Console: as mentioned I haven't yet been there on site since this happened.

I wait for the users and their feedback: if things work so far we don't have much urgency and I might plan a visit within the next weeks to either reinstall (with ZFS) or try things via USB console.

I went through https://docs.netgate.com/pfsense/en/latest/troubleshooting/pkg-broken-database.html already yesterday. There seems to be something broken on a basic level, these things don't work anymore either.

But I can retry things if you have suggestions, sure.

sgw

It seems to me as if the (whole?) routing, firewalling etc works fine. That is good as it takes some urgency out of this.

I assume I could proceed by "sideloading" /usr/local/lib/libpcre2-8.so.0 via ssh, this might give me shell access to ssh (the admin-menu seems to crash right now) and from there I might fix more things maybe.

(if one could provide me that file for an Intel Atom platform, that would be great, so far I wasn't successful googling that. And I don't have another similar box that I can steal that file from)

I am unsure why pkg-static update fails. DNS should work, I can successfully do things like:

ssh admin@pfsense host forum.netgate.com
forum.netgate.com has address 208.123.73.77
forum.netgate.com has IPv6 address 2610:160:11:11::6

but:

 ssh admin@pfsense pkg-static -d4 update 
Updating FreeBSD-ports repository catalogue...
DBG(1)[2921]> PkgRepo: verifying update for FreeBSD-ports
DBG(1)[2921]> Pkgrepo, begin update of '/var/db/pkg/repos/FreeBSD-ports/db'
DBG(1)[2921]> (fetch) Request to fetch pkg+https://pkg.FreeBSD.org/FreeBSD:16:amd64/latest/meta.conf
DBG(1)[2921]> (fetch) Fetch: fetcher used: pkg+https
DBG(1)[2921]> (fetch) Request to fetch pkg+https://pkg.FreeBSD.org/FreeBSD:16:amd64/latest/data.pkg
DBG(1)[2921]> (fetch) Fetch: fetcher used: pkg+https
FreeBSD-ports repository is up to date.
Updating FreeBSD-ports-kmods repository catalogue...
DBG(1)[2921]> PkgRepo: verifying update for FreeBSD-ports-kmods
DBG(1)[2921]> Pkgrepo, begin update of '/var/db/pkg/repos/FreeBSD-ports-kmods/db'
DBG(1)[2921]> (fetch) Request to fetch pkg+https://pkg.FreeBSD.org/FreeBSD:16:amd64/kmods_latest/meta.conf
DBG(1)[2921]> (fetch) Fetch: fetcher used: pkg+https
DBG(1)[2921]> (fetch) Request to fetch pkg+https://pkg.FreeBSD.org/FreeBSD:16:amd64/kmods_latest/data.pkg
DBG(1)[2921]> (fetch) Fetch: fetcher used: pkg+https
FreeBSD-ports-kmods repository is up to date.
Updating pfSense-core repository catalogue...
DBG(1)[2921]> PkgRepo: verifying update for pfSense-core
DBG(1)[2921]> Pkgrepo, begin update of '/var/db/pkg/repos/pfSense-core/db'
DBG(1)[2921]> (fetch) Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-core/meta.conf
DBG(1)[2921]> (fetch) Fetch: fetcher used: pkg+https
pkg-static: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-core/meta.conf: No error

[...]

and

 ssh admin@pfsense pkg-static install -fy  pfSense-repo
Updating FreeBSD-ports repository catalogue...
FreeBSD-ports repository is up to date.
Updating FreeBSD-ports-kmods repository catalogue...
FreeBSD-ports-kmods repository is up to date.
Updating pfSense-core repository catalogue...
pkg-static: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-core/meta.conf: No error
repository pfSense-core has no meta file, using default settings
pkg-static: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-core/meta.txz: No error
pkg-static: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-core/data.pkg: No error
pkg-static: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-core/data.tzst: No error
pkg-static: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-core/packagesite.pkg: No error
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
pkg-static: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-core/packagesite.tzst: No error
pkg-static: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-pfSense_plus_v25_11/meta.conf: No error
repository pfSense has no meta file, using default settings
pkg-static: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-pfSense_plus_v25_11/meta.txz: No error
pkg-static: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-pfSense_plus_v25_11/data.pkg: No error
pkg-static: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-pfSense_plus_v25_11/data.tzst: No error
pkg-static: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-pfSense_plus_v25_11/packagesite.pkg: No error
Unable to update repository pfSense
Error updating repositories!
pkg-static: Failed to fetch https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_11_amd64-pfSense_plus_v25_11/packagesite.tzst: No error

and (essential maybe?):

pfSense-repoc: cannot read pfSense pkg prefix

stephenw10

You likely need to run pfSense-repoc -N to get a current cert for pkg.

You might try SSHing in as root which doesn't have to have the menu unlike admin.

sgw

@stephenw10

 ssh admin@pfsense  pfSense-repoc -N
failed to collect the system information.
pfSense-repoc: cannot read pfSense pkg prefix

sgw

@stephenw10 said in 25.11: broken PHP on Netgate 7100:

You might try SSHing in as root which doesn't have to have the menu unlike admin.

Ah, sure ;-) yes, that works also

stephenw10

Hmm, so no pkg or pkg-static commands work?

Try pkg-static info pkg and pkg-static info -x pfsense

Seems like you might have a bad pkg version on there somehow.

sgw

@stephenw10

thanks for your help, happy to test this:

# pkg-static info pkg 
pkg-2.4.2_1
Name           : pkg
Version        : 2.4.2_1
Installed on   : Tue Dec 16 19:51:30 2025 CET
Origin         : ports-mgmt/pkg
Architecture   : FreeBSD:16:amd64
Prefix         : /usr/local
Categories     : ports-mgmt
Licenses       : BSD2CLAUSE
Maintainer     : pkg@FreeBSD.org
WWW            : https://github.com/freebsd/pkg
Comment        : Package manager
Options        :
	DOCS           : on
Shared Libs required:
	libarchive.so.7
	libc.so.7
	libcrypto.so.35
	libelf.so.2
	libjail.so.1
	libm.so.5
	libssl.so.35
	libthr.so.3
	libutil.so.10
	libz.so.6
Shared Libs provided:
	libpkg.so.4
Annotations    :
	FreeBSD_version: 1600005
	build_timestamp: 2025-12-11T01:08:56+0000
	built_by       : poudriere-git-3.4.4-8-ge4f2486b
	port_checkout_unclean: no
	port_git_hash  : 236ce023bd14008e44bb91ea3222f8950942d78f
	ports_top_checkout_unclean: no
	ports_top_git_hash: 9749e408286c7f2632e85faba3927521c5bad069
	repo_type      : binary
	repository     : FreeBSD-ports
Flat size      : 29.3MiB
Description    :
Package management tool

# pkg-static info -x pfsense
pkg-static: No package(s) matching pfsense

stephenw10

OK yes that's a newer version. It was likely pulled in during Beta/RC?

Check /var/cache/pkg for pkg-2.3.1. If it's there you can reinstall it from there directly.

sgw

@stephenw10

I see

/var/cache/pkg/pkg-2.4.2_1.pkg

stephenw10

What branches do you see available in System > Update?

sgw

@stephenw10

Current Stable Version (25.11)
Previous Stable Version (25.07.1)

I tried toggling that already without much effect (imo).

sgw

@stephenw10 said in 25.11: broken PHP on Netgate 7100:

OK yes that's a newer version. It was likely pulled in during Beta/RC?

I never ran Beta on that appliance.

stephenw10

Hmm, can you see in the system logs when pkg was updated?

sgw

@stephenw10

something like this:

Dec 16 19:28:28 sgw pkg-static[66328]: pfSense-upgrade upgraded: 1.3.11 -> 1.3.15 
Dec 16 19:31:32 sgw pkg-static[60813]: pfSense-boot upgraded: 25.07.1 -> 25.11 
Dec 16 19:31:43 sgw pkg-static[64490]: pfSense-kernel-pfSense upgraded: 25.07.1 -> 25.11 
Dec 16 19:33:42 sgw pkg-static[55644]: pkg upgraded: 1.21.3_5 -> 2.3.1 
Dec 16 19:39:16 sgw pkg-static[39057]: pkg reinstalled: 2.3.1 -> 2.3.1 
Dec 16 19:39:38 sgw pkg-static[492]: pfSense-base upgraded: 25.07.1 -> 25.11 
Dec 16 19:44:30 sgw pkg-static[96926]: pkg upgraded: 2.3.1 -> 2.4.2_1