PHP errors after upgrade to 23.09: "libicuio.so.72" not found

DominikHoffmann

A second Netgate 1100 is exhibiting the same symptoms as the one that prompted me to start this thread:

Screenshot 2023-11-07 at 12.59.23 AM.png

and

Screenshot 2023-11-07 at 12.59.50 AM.png

Finally, output from pkg-static upgrade:

Updating pfSense-core repository catalogue...
Fetching meta.conf:
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
Fetching meta.conf:
pfSense repository is up to date.
All repositories are up to date.
Checking for upgrades (73 candidates): .......... done
Processing candidates (73 candidates): .......... done
Checking integrity... done (0 conflicting)
The following 83 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	boost-libs: 1.82.0_1 [pfSense]
	dhcpcd: 10.0.2 [pfSense]
	dhcpleases6: 0.1_3 [pfSense]
	gmp: 6.3.0 [pfSense]
	kea: 2.4.0 [pfSense]
	log4cplus: 2.1.0 [pfSense]
	pfSense-composer-deps: 0.1 [pfSense]
	php82-gmp: 8.2.11 [pfSense]
	py311-libzfs: 1.1.2023020700 [pfSense]
	py311-setuptools: 63.1.0_1 [pfSense]

Installed packages to be UPGRADED:
	bind-tools: 9.18.13 -> 9.18.16 [pfSense]
	ca_root_nss: 3.89 -> 3.92 [pfSense]
	curl: 8.0.1 -> 8.4.0 [pfSense]
	cyrus-sasl: 2.1.28 -> 2.1.28_1 [pfSense]
	filterlog: 0.1_9 -> 0.1_10 [pfSense]
	glib: 2.76.1,2 -> 2.76.4,2 [pfSense]
	icu: 72.1,1 -> 73.2,1 [pfSense]
	igmpproxy: 0.3_1,1 -> 0.4,1 [pfSense]
	isc-dhcp44-relay: 4.4.3P1 -> 4.4.3P1_3 [pfSense]
	isc-dhcp44-server: 4.4.3P1 -> 4.4.3P1_3 [pfSense]
	libgcrypt: 1.9.4_1 -> 1.10.2 [pfSense]
	libgpg-error: 1.46 -> 1.47 [pfSense]
	libnghttp2: 1.52.0 -> 1.55.1_1 [pfSense]
	libpsl: 0.21.2_2 -> 0.21.2_3 [pfSense]
	libssh2: 1.10.0_1,3 -> 1.11.0_1,3 [pfSense]
	libuv: 1.44.2 -> 1.46.0 [pfSense]
	links: 2.28,1 -> 2.29_1,1 [pfSense]
	lua-resty-core: 0.1.24 -> 0.1.27 [pfSense]
	luajit-openresty: 2.1.20230119 -> 2.1.20230410 [pfSense]
	miniupnpd: 2.2.1_1,1 -> 2.3.3_1,1 [pfSense]
	mobile-broadband-provider-info: 20221107 -> 20230416 [pfSense]
	mpd5: 5.9_14 -> 5.9_16 [pfSense]
	nginx: 1.22.1_5,3 -> 1.24.0_11,3 [pfSense]
	ntp: 4.2.8p15_5 -> 4.2.8p17_1 [pfSense]
	openldap26-client: 2.6.4 -> 2.6.6 [pfSense]
	openvpn: 2.6.2 -> 2.6.5 [pfSense]
	pcsc-lite: 1.9.9,2 -> 2.0.0,2 [pfSense]
	perl5: 5.32.1_3 -> 5.34.1_3 [pfSense]
	pfSense: 23.05.1 -> 23.09 [pfSense]
	pfSense-default-config-serial: 23.05.1 -> 23.09 [pfSense]
	pfSense-repo: 23.05.1 -> 23.09 [pfSense]
	pftop: 0.8_2 -> 0.8_4 [pfSense]
	php82-bcmath: 8.2.4 -> 8.2.11 [pfSense]
	php82-bz2: 8.2.4 -> 8.2.11 [pfSense]
	php82-ctype: 8.2.4 -> 8.2.11 [pfSense]
	php82-curl: 8.2.4 -> 8.2.11 [pfSense]
	php82-dom: 8.2.4 -> 8.2.11 [pfSense]
	php82-filter: 8.2.4 -> 8.2.11 [pfSense]
	php82-gettext: 8.2.4 -> 8.2.11 [pfSense]
	php82-intl: 8.2.4_1 -> 8.2.11 [pfSense]
	php82-ldap: 8.2.4 -> 8.2.11 [pfSense]
	php82-mbstring: 8.2.4 -> 8.2.11 [pfSense]
	php82-opcache: 8.2.4 -> 8.2.11 [pfSense]
	php82-openssl_x509_crl: 1.3_2 -> 1.3_3 [pfSense]
	php82-pcntl: 8.2.4 -> 8.2.11 [pfSense]
	php82-pdo: 8.2.4 -> 8.2.11 [pfSense]
	php82-pdo_sqlite: 8.2.4 -> 8.2.11 [pfSense]
	php82-pear-Mail: 1.4.1,1 -> 1.5.0,1 [pfSense]
	php82-posix: 8.2.4 -> 8.2.11 [pfSense]
	php82-readline: 8.2.4 -> 8.2.11 [pfSense]
	php82-session: 8.2.4 -> 8.2.11 [pfSense]
	php82-shmop: 8.2.4 -> 8.2.11 [pfSense]
	php82-simplexml: 8.2.4 -> 8.2.11 [pfSense]
	php82-sockets: 8.2.4 -> 8.2.11 [pfSense]
	php82-sqlite3: 8.2.4 -> 8.2.11 [pfSense]
	php82-sysvmsg: 8.2.4 -> 8.2.11 [pfSense]
	php82-sysvsem: 8.2.4 -> 8.2.11 [pfSense]
	php82-sysvshm: 8.2.4 -> 8.2.11 [pfSense]
	php82-tokenizer: 8.2.4 -> 8.2.11 [pfSense]
	php82-xml: 8.2.4 -> 8.2.11 [pfSense]
	php82-xmlreader: 8.2.4 -> 8.2.11 [pfSense]
	php82-xmlwriter: 8.2.4 -> 8.2.11 [pfSense]
	py39-setuptools: 63.1.0 -> 63.1.0_1 [pfSense]
	python311: 3.11.2_2 -> 3.11.4 [pfSense]
	python39: 3.9.16_2 -> 3.9.17 [pfSense]
	smartmontools: 7.3 -> 7.4 [pfSense]
	sqlite3: 3.41.0_1,1 -> 3.42.0,1 [pfSense]
	strongswan: 5.9.10_2 -> 5.9.11_1 [pfSense]
	uclcmd: 0.1_3 -> 0.2.20211204 [pfSense]
	unbound: 1.17.1_3 -> 1.18.0 [pfSense]
	voucher: 0.1_2 -> 0.1_3 [pfSense]
	zstd: 1.5.4_1 -> 1.5.5 [pfSense]

Installed packages to be REINSTALLED:
	rrdtool-1.8.0_2 [pfSense] (direct dependency changed: perl5)

Number of packages to be installed: 10
Number of packages to be upgraded: 72
Number of packages to be reinstalled: 1

The process will require 386 MiB more space.

stephenw10

Yes, I'm not sure why they would both have failed to upgrade those pkgs.

You could check the upgrade logs in /conf to see if there's a common error.

The base 23.09 install has run already and completed, I assume it rebooted there?

But yes installing those packages is what I would try next.

Steve

DominikHoffmann

@stephenw10

Following your suggestion, /conf/upgrade_log.txt only contains

>>> Updating repositories metadata... done.
23.09 version of pfSense is available

The other files there are

acb/
backup/
config.xml
copyright
dyndns_wandyndns'********'0.cache
enableserial_force
lastpfSbackup.txt
pkg_log_pfSense-pkg-Avahi.txt
pkg_log_pfSense-pkg-Cron.txt
pkg_log_pfSense-pkg-Notes.txt
pkg_log_pfSense-pkg-WireGuard.txt
pkg_log_pfSense-pkg-aws-wizard.txt
pkg_log_pfSense-pkg-ipsec-profile-wizard.txt
pkg_log_pfSense-pkg-openvpn-client-export.txt
rules.debug.old
syslog_default_uncompressed
upgrade_log.txt

None of the others look like they would contain relevant information.

As for updating packages individually, I am not that familiar with FreeBSD. What are the commands for packages that need to be installed from scratch and those for updating packages? Can those commands all be issued from Diagnostics → Command Prompt → Execute Shell Command?

stephenw10

Hmm, I guess it never completed the upgrade and wrote out the log then. I'd expect to see upgrade_log.latest.txt there.

I would want to run any pkg upgrade command from a real command line, via SSH, not from the GUI. Otherwise you cannot interrupt the command or see what's happening whilst it's running.

When you run pkg-static upgrade it should upgrade any installed packages for which newer versions are available. And it will pull in any new dependencies.

You can also force reinstall all packages as a last resort:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/upgrades.html#forced-pkg-reinstall

Steve

DominikHoffmann

@stephenw10

I don’t have physical access nor ssh access to the real command line, the latter presumably, because sshd is in limbo, somehow.

I will try pkg-static upgrade from Diagnostics → Command Prompt → Execute Shell Command.

stephenw10

You will need to use pkg-static upgrade -y at the gui command prompt because it's non-interactive.

Do you know if it rebooted after the initial failed upgrade?

DominikHoffmann

@stephenw10

The output of pkg-static upgrade contains the line The process will require 386 MiB more space. Does that point to the source of the problem?

DominikHoffmann

@stephenw10

I am not entirely sure that it rebooted. I have notifications via email enabled on the system, and received the notification that the system was going to go down for the update. However, I never did receive the notification that the reboot was complete.

I just tried rebooting it from Diagnostics → Reboot. After that, when I got the WebConfigurator login prompt again, I looked at the uptime: 21:24 h. It did not reboot.

I think, I need to get physical access and reboot the unit from the console.

stephenw10

@DominikHoffmann said in PHP errors after upgrade to 23.09: "libicuio.so.72" not found:

The output of pkg-static upgrade contains the line The process will require 386 MiB more space. Does that point to the source of the problem?

No, that's just telling you how much additional space it will require.

Check the uptime on the dashboard. That will tell you when it last rebooted.

It's possible it simply never rebooted after the first part of the upgrade. There's probably a reason for that though if it's the case.

Is it running ZFS? If so you can always roll back but that requires some local access.

DominikHoffmann

@stephenw10 Yes, it is running ZFS.

stephenw10

What about the uptime? Does it look like it rebooted?

DominikHoffmann

Doing pkg-static upgrade -y at Diagnostics → Command Prompt → Execute Shell Command updated the packages correctly. I then forced a reboot at Diagnostics → Reboot. Uptime is now in the minutes range. This seems to have worked. Making an ssh connection to the router works, too. It looks like all is good.

stephenw10

Ah, excellent!

DominikHoffmann

@stephenw10 Thanks so much, Stephen, for your excellent help!