PHP errors after upgrade to 23.09: "libicuio.so.72" not found

DominikHoffmann · Nov 6, 2023, 8:05 PM

After installing 23.09 from the WebConfigurator I have these problems:

Diagnostics → Crash Reporter is full of these kinds of messages: [06-Nov-2023 19:43:00 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20220829/intl.so (Shared object "libicuio.so.72" not found, required by "intl.so"), /usr/local/lib/php/20220829/intl.so.so (Cannot open "/usr/local/lib/php/20220829/intl.so.so")) in Unknown on line 0
The dashboard shows that I am still on 23.05.1, while System → Update → System Update shows “Current Base System 23.09.”
I can’t ssh in with either password or public key to try remove and reinstall the icu package (see a previous thread). Executing those commands at Diagnostics → Command Prompt won’t work, because they require user input for confirmation.

DominikHoffmann · Nov 6, 2023, 8:23 PM

I went to Diagnostics → Command Prompt and did this:

pkg unlock icu
pkg delete -yf icu
pkg install -yf icu

The “y” option in the pkg delete command made it non-interactive and could therefore be executed.

I am still getting the same errors Diagnostics → Crash Reporter, about two per minute. I am afraid that this might lead to a DOS, when the file system is full (on a Netgate 1100). This is a client’s system, which is why I am especially concerned.

stephenw10 · Nov 6, 2023, 10:49 PM

Sounds like it partially upgraded. Try running: pkg-static info -x pfSense to see what pkg versions you have.

If it did stop part way through installing there is usually a reason. Did it run out of drive space perhaps?

DominikHoffmann · Nov 6, 2023, 11:14 PM

@stephenw10: The output of that is

pfSense-23.05.1
pfSense-Status_Monitoring-php82-1.8_3
pfSense-base-23.09
pfSense-boot-23.09
pfSense-default-config-serial-23.05.1
pfSense-kernel-pfSense-23.09
pfSense-pkg-Avahi-2.2_4
pfSense-pkg-Cron-0.3.8_3
pfSense-pkg-Notes-0.2.9_5
pfSense-pkg-openvpn-client-export-1.9_1
pfSense-repo-23.05.1
pfSense-repoc-20230912
pfSense-u-boot-1100-20220428
pfSense-u-boot-2100-20210930_1
pfSense-u-boot-env-20230123
pfSense-upgrade-1.2_6
php82-pfSense-module-0.95

As for disk space:

stephenw10 · Nov 7, 2023, 3:48 AM

Interesting it has pulled in somethings.

Ok try running: pkg-static upgrade and see what packages it offers to you. Do not agree to the pkg upgrades yet.

DominikHoffmann · Nov 7, 2023, 3:48 AM

@stephenw10

Updating pfSense-core repository catalogue...
Fetching meta.conf:
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
Fetching meta.conf:
pfSense repository is up to date.
All repositories are up to date.
Checking for upgrades (55 candidates): .......... done
Processing candidates (55 candidates): .......... done
Checking integrity... done (0 conflicting)
The following 65 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	boost-libs: 1.82.0_1 [pfSense]
	dhcpcd: 10.0.2 [pfSense]
	dhcpleases6: 0.1_3 [pfSense]
	gmp: 6.3.0 [pfSense]
	kea: 2.4.0 [pfSense]
	log4cplus: 2.1.0 [pfSense]
	pfSense-composer-deps: 0.1 [pfSense]
	php82-gmp: 8.2.11 [pfSense]
	py311-libzfs: 1.1.2023020700 [pfSense]
	py311-setuptools: 63.1.0_1 [pfSense]

Installed packages to be UPGRADED:
	bind-tools: 9.18.13 -> 9.18.16 [pfSense]
	filterlog: 0.1_9 -> 0.1_10 [pfSense]
	igmpproxy: 0.3_1,1 -> 0.4,1 [pfSense]
	isc-dhcp44-relay: 4.4.3P1 -> 4.4.3P1_3 [pfSense]
	isc-dhcp44-server: 4.4.3P1 -> 4.4.3P1_3 [pfSense]
	links: 2.28,1 -> 2.29_1,1 [pfSense]
	lua-resty-core: 0.1.24 -> 0.1.27 [pfSense]
	miniupnpd: 2.2.1_1,1 -> 2.3.3_1,1 [pfSense]
	mobile-broadband-provider-info: 20221107 -> 20230416 [pfSense]
	mpd5: 5.9_14 -> 5.9_16 [pfSense]
	nginx: 1.22.1_5,3 -> 1.24.0_11,3 [pfSense]
	ntp: 4.2.8p15_5 -> 4.2.8p17_1 [pfSense]
	openldap26-client: 2.6.4 -> 2.6.6 [pfSense]
	openvpn: 2.6.2 -> 2.6.5 [pfSense]
	pcsc-lite: 1.9.9,2 -> 2.0.0,2 [pfSense]
	pfSense: 23.05.1 -> 23.09 [pfSense]
	pfSense-default-config-serial: 23.05.1 -> 23.09 [pfSense]
	pfSense-repo: 23.05.1 -> 23.09 [pfSense]
	pftop: 0.8_2 -> 0.8_4 [pfSense]
	php82-bcmath: 8.2.4 -> 8.2.11 [pfSense]
	php82-bz2: 8.2.4 -> 8.2.11 [pfSense]
	php82-ctype: 8.2.4 -> 8.2.11 [pfSense]
	php82-curl: 8.2.4 -> 8.2.11 [pfSense]
	php82-dom: 8.2.4 -> 8.2.11 [pfSense]
	php82-filter: 8.2.4 -> 8.2.11 [pfSense]
	php82-gettext: 8.2.4 -> 8.2.11 [pfSense]
	php82-intl: 8.2.4_1 -> 8.2.11 [pfSense]
	php82-ldap: 8.2.4 -> 8.2.11 [pfSense]
	php82-mbstring: 8.2.4 -> 8.2.11 [pfSense]
	php82-opcache: 8.2.4 -> 8.2.11 [pfSense]
	php82-openssl_x509_crl: 1.3_2 -> 1.3_3 [pfSense]
	php82-pcntl: 8.2.4 -> 8.2.11 [pfSense]
	php82-pdo: 8.2.4 -> 8.2.11 [pfSense]
	php82-pdo_sqlite: 8.2.4 -> 8.2.11 [pfSense]
	php82-pear-Mail: 1.4.1,1 -> 1.5.0,1 [pfSense]
	php82-posix: 8.2.4 -> 8.2.11 [pfSense]
	php82-readline: 8.2.4 -> 8.2.11 [pfSense]
	php82-session: 8.2.4 -> 8.2.11 [pfSense]
	php82-shmop: 8.2.4 -> 8.2.11 [pfSense]
	php82-simplexml: 8.2.4 -> 8.2.11 [pfSense]
	php82-sockets: 8.2.4 -> 8.2.11 [pfSense]
	php82-sqlite3: 8.2.4 -> 8.2.11 [pfSense]
	php82-sysvmsg: 8.2.4 -> 8.2.11 [pfSense]
	php82-sysvsem: 8.2.4 -> 8.2.11 [pfSense]
	php82-sysvshm: 8.2.4 -> 8.2.11 [pfSense]
	php82-tokenizer: 8.2.4 -> 8.2.11 [pfSense]
	php82-xmlreader: 8.2.4 -> 8.2.11 [pfSense]
	php82-xmlwriter: 8.2.4 -> 8.2.11 [pfSense]
	py39-setuptools: 63.1.0 -> 63.1.0_1 [pfSense]
	smartmontools: 7.3 -> 7.4 [pfSense]
	sqlite3: 3.41.0_1,1 -> 3.42.0,1 [pfSense]
	strongswan: 5.9.10_2 -> 5.9.11_1 [pfSense]
	uclcmd: 0.1_3 -> 0.2.20211204 [pfSense]
	voucher: 0.1_2 -> 0.1_3 [pfSense]
	zstd: 1.5.4_1 -> 1.5.5 [pfSense]

Number of packages to be installed: 10
Number of packages to be upgraded: 55

The process will require 380 MiB more space.

Does this suggest a course of action? By the way, I updated a second Netgate 1100 (my own) today, as well, and had no problems. If anything, mine uses more resources (installed packages, configurations, etc.).

DominikHoffmann · Nov 7, 2023, 6:03 AM

A second Netgate 1100 is exhibiting the same symptoms as the one that prompted me to start this thread:

and

Finally, output from pkg-static upgrade:

Updating pfSense-core repository catalogue...
Fetching meta.conf:
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
Fetching meta.conf:
pfSense repository is up to date.
All repositories are up to date.
Checking for upgrades (73 candidates): .......... done
Processing candidates (73 candidates): .......... done
Checking integrity... done (0 conflicting)
The following 83 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	boost-libs: 1.82.0_1 [pfSense]
	dhcpcd: 10.0.2 [pfSense]
	dhcpleases6: 0.1_3 [pfSense]
	gmp: 6.3.0 [pfSense]
	kea: 2.4.0 [pfSense]
	log4cplus: 2.1.0 [pfSense]
	pfSense-composer-deps: 0.1 [pfSense]
	php82-gmp: 8.2.11 [pfSense]
	py311-libzfs: 1.1.2023020700 [pfSense]
	py311-setuptools: 63.1.0_1 [pfSense]

Installed packages to be UPGRADED:
	bind-tools: 9.18.13 -> 9.18.16 [pfSense]
	ca_root_nss: 3.89 -> 3.92 [pfSense]
	curl: 8.0.1 -> 8.4.0 [pfSense]
	cyrus-sasl: 2.1.28 -> 2.1.28_1 [pfSense]
	filterlog: 0.1_9 -> 0.1_10 [pfSense]
	glib: 2.76.1,2 -> 2.76.4,2 [pfSense]
	icu: 72.1,1 -> 73.2,1 [pfSense]
	igmpproxy: 0.3_1,1 -> 0.4,1 [pfSense]
	isc-dhcp44-relay: 4.4.3P1 -> 4.4.3P1_3 [pfSense]
	isc-dhcp44-server: 4.4.3P1 -> 4.4.3P1_3 [pfSense]
	libgcrypt: 1.9.4_1 -> 1.10.2 [pfSense]
	libgpg-error: 1.46 -> 1.47 [pfSense]
	libnghttp2: 1.52.0 -> 1.55.1_1 [pfSense]
	libpsl: 0.21.2_2 -> 0.21.2_3 [pfSense]
	libssh2: 1.10.0_1,3 -> 1.11.0_1,3 [pfSense]
	libuv: 1.44.2 -> 1.46.0 [pfSense]
	links: 2.28,1 -> 2.29_1,1 [pfSense]
	lua-resty-core: 0.1.24 -> 0.1.27 [pfSense]
	luajit-openresty: 2.1.20230119 -> 2.1.20230410 [pfSense]
	miniupnpd: 2.2.1_1,1 -> 2.3.3_1,1 [pfSense]
	mobile-broadband-provider-info: 20221107 -> 20230416 [pfSense]
	mpd5: 5.9_14 -> 5.9_16 [pfSense]
	nginx: 1.22.1_5,3 -> 1.24.0_11,3 [pfSense]
	ntp: 4.2.8p15_5 -> 4.2.8p17_1 [pfSense]
	openldap26-client: 2.6.4 -> 2.6.6 [pfSense]
	openvpn: 2.6.2 -> 2.6.5 [pfSense]
	pcsc-lite: 1.9.9,2 -> 2.0.0,2 [pfSense]
	perl5: 5.32.1_3 -> 5.34.1_3 [pfSense]
	pfSense: 23.05.1 -> 23.09 [pfSense]
	pfSense-default-config-serial: 23.05.1 -> 23.09 [pfSense]
	pfSense-repo: 23.05.1 -> 23.09 [pfSense]
	pftop: 0.8_2 -> 0.8_4 [pfSense]
	php82-bcmath: 8.2.4 -> 8.2.11 [pfSense]
	php82-bz2: 8.2.4 -> 8.2.11 [pfSense]
	php82-ctype: 8.2.4 -> 8.2.11 [pfSense]
	php82-curl: 8.2.4 -> 8.2.11 [pfSense]
	php82-dom: 8.2.4 -> 8.2.11 [pfSense]
	php82-filter: 8.2.4 -> 8.2.11 [pfSense]
	php82-gettext: 8.2.4 -> 8.2.11 [pfSense]
	php82-intl: 8.2.4_1 -> 8.2.11 [pfSense]
	php82-ldap: 8.2.4 -> 8.2.11 [pfSense]
	php82-mbstring: 8.2.4 -> 8.2.11 [pfSense]
	php82-opcache: 8.2.4 -> 8.2.11 [pfSense]
	php82-openssl_x509_crl: 1.3_2 -> 1.3_3 [pfSense]
	php82-pcntl: 8.2.4 -> 8.2.11 [pfSense]
	php82-pdo: 8.2.4 -> 8.2.11 [pfSense]
	php82-pdo_sqlite: 8.2.4 -> 8.2.11 [pfSense]
	php82-pear-Mail: 1.4.1,1 -> 1.5.0,1 [pfSense]
	php82-posix: 8.2.4 -> 8.2.11 [pfSense]
	php82-readline: 8.2.4 -> 8.2.11 [pfSense]
	php82-session: 8.2.4 -> 8.2.11 [pfSense]
	php82-shmop: 8.2.4 -> 8.2.11 [pfSense]
	php82-simplexml: 8.2.4 -> 8.2.11 [pfSense]
	php82-sockets: 8.2.4 -> 8.2.11 [pfSense]
	php82-sqlite3: 8.2.4 -> 8.2.11 [pfSense]
	php82-sysvmsg: 8.2.4 -> 8.2.11 [pfSense]
	php82-sysvsem: 8.2.4 -> 8.2.11 [pfSense]
	php82-sysvshm: 8.2.4 -> 8.2.11 [pfSense]
	php82-tokenizer: 8.2.4 -> 8.2.11 [pfSense]
	php82-xml: 8.2.4 -> 8.2.11 [pfSense]
	php82-xmlreader: 8.2.4 -> 8.2.11 [pfSense]
	php82-xmlwriter: 8.2.4 -> 8.2.11 [pfSense]
	py39-setuptools: 63.1.0 -> 63.1.0_1 [pfSense]
	python311: 3.11.2_2 -> 3.11.4 [pfSense]
	python39: 3.9.16_2 -> 3.9.17 [pfSense]
	smartmontools: 7.3 -> 7.4 [pfSense]
	sqlite3: 3.41.0_1,1 -> 3.42.0,1 [pfSense]
	strongswan: 5.9.10_2 -> 5.9.11_1 [pfSense]
	uclcmd: 0.1_3 -> 0.2.20211204 [pfSense]
	unbound: 1.17.1_3 -> 1.18.0 [pfSense]
	voucher: 0.1_2 -> 0.1_3 [pfSense]
	zstd: 1.5.4_1 -> 1.5.5 [pfSense]

Installed packages to be REINSTALLED:
	rrdtool-1.8.0_2 [pfSense] (direct dependency changed: perl5)

Number of packages to be installed: 10
Number of packages to be upgraded: 72
Number of packages to be reinstalled: 1

The process will require 386 MiB more space.

stephenw10 · Nov 7, 2023, 1:42 PM

Yes, I'm not sure why they would both have failed to upgrade those pkgs.

You could check the upgrade logs in /conf to see if there's a common error.

The base 23.09 install has run already and completed, I assume it rebooted there?

But yes installing those packages is what I would try next.

Steve

DominikHoffmann · Nov 7, 2023, 2:35 PM

@stephenw10

Following your suggestion, /conf/upgrade_log.txt only contains

>>> Updating repositories metadata... done.
23.09 version of pfSense is available

The other files there are

acb/
backup/
config.xml
copyright
dyndns_wandyndns'********'0.cache
enableserial_force
lastpfSbackup.txt
pkg_log_pfSense-pkg-Avahi.txt
pkg_log_pfSense-pkg-Cron.txt
pkg_log_pfSense-pkg-Notes.txt
pkg_log_pfSense-pkg-WireGuard.txt
pkg_log_pfSense-pkg-aws-wizard.txt
pkg_log_pfSense-pkg-ipsec-profile-wizard.txt
pkg_log_pfSense-pkg-openvpn-client-export.txt
rules.debug.old
syslog_default_uncompressed
upgrade_log.txt

None of the others look like they would contain relevant information.

As for updating packages individually, I am not that familiar with FreeBSD. What are the commands for packages that need to be installed from scratch and those for updating packages? Can those commands all be issued from Diagnostics → Command Prompt → Execute Shell Command?

stephenw10 · Nov 7, 2023, 2:52 PM

Hmm, I guess it never completed the upgrade and wrote out the log then. I'd expect to see upgrade_log.latest.txt there.

I would want to run any pkg upgrade command from a real command line, via SSH, not from the GUI. Otherwise you cannot interrupt the command or see what's happening whilst it's running.

When you run pkg-static upgrade it should upgrade any installed packages for which newer versions are available. And it will pull in any new dependencies.

You can also force reinstall all packages as a last resort:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/upgrades.html#forced-pkg-reinstall

Steve

DominikHoffmann · Nov 7, 2023, 3:26 PM

@stephenw10

I don’t have physical access nor ssh access to the real command line, the latter presumably, because sshd is in limbo, somehow.

I will try pkg-static upgrade from Diagnostics → Command Prompt → Execute Shell Command.

stephenw10 · Nov 7, 2023, 3:30 PM

You will need to use pkg-static upgrade -y at the gui command prompt because it's non-interactive.

Do you know if it rebooted after the initial failed upgrade?

DominikHoffmann · Nov 7, 2023, 3:33 PM

@stephenw10

The output of pkg-static upgrade contains the line The process will require 386 MiB more space. Does that point to the source of the problem?

DominikHoffmann · Nov 7, 2023, 3:55 PM

@stephenw10

I am not entirely sure that it rebooted. I have notifications via email enabled on the system, and received the notification that the system was going to go down for the update. However, I never did receive the notification that the reboot was complete.

I just tried rebooting it from Diagnostics → Reboot. After that, when I got the WebConfigurator login prompt again, I looked at the uptime: 21:24 h. It did not reboot.

I think, I need to get physical access and reboot the unit from the console.

stephenw10 · Nov 7, 2023, 3:56 PM

@DominikHoffmann said in PHP errors after upgrade to 23.09: "libicuio.so.72" not found:

The output of pkg-static upgrade contains the line The process will require 386 MiB more space. Does that point to the source of the problem?

No, that's just telling you how much additional space it will require.

Check the uptime on the dashboard. That will tell you when it last rebooted.

It's possible it simply never rebooted after the first part of the upgrade. There's probably a reason for that though if it's the case.

Is it running ZFS? If so you can always roll back but that requires some local access.

DominikHoffmann · Nov 7, 2023, 3:58 PM

@stephenw10 Yes, it is running ZFS.

stephenw10 · Nov 7, 2023, 4:58 PM

What about the uptime? Does it look like it rebooted?

DominikHoffmann · Nov 7, 2023, 5:00 PM

Doing pkg-static upgrade -y at Diagnostics → Command Prompt → Execute Shell Command updated the packages correctly. I then forced a reboot at Diagnostics → Reboot. Uptime is now in the minutes range. This seems to have worked. Making an ssh connection to the router works, too. It looks like all is good.

stephenw10 · Nov 7, 2023, 5:05 PM

Ah, excellent!

DominikHoffmann · Nov 7, 2023, 5:15 PM

@stephenw10 Thanks so much, Stephen, for your excellent help!