pfSense CE 2.8.1 -> 2.9.0 upgrade broken PHP / libcrypto.so.35 not found (Proxmox VM)

cabgrall

pfSense CE 2.8.1 -> 2.9.0 upgrade broken PHP / libcrypto.so.35 not found (Proxmox VM)

Environment:

Platform: Proxmox VE 9 VM
pfSense: upgrading from 2.8.1-RELEASE to 2.9.0-RELEASE
VM config: 4 vCPU, 6 GB RAM, 4x network interfaces (VirtIO)
BIOS: SeaBIOS
In-place upgrade performed via the pfSense webGUI (System > Update)

Summary:

Upgraded via System > Update in the GUI (branch set to 2.9.0). The upgrade process appeared to run but did not complete cleanly. Afterwards the system is in a broken state — PHP fails to run because required shared libraries are missing.

Symptoms:

The webGUI is (please confirm: unreachable / partially working / other).

From the shell:

# ls -l /lib/libcrypto.so*
-r--r--r--  1 root wheel 3083872 Jun 28  2023 /lib/libcrypto.so.111
-r--r--r--  1 root wheel 4672904 Aug 28  2025 /lib/libcrypto.so.30

# ls -l /usr/libcrypto.so*
ls: /usr/libcrypto.so*: No such file or directory

# ls -l /usr/lib/libcrypto.so*
lrwxr-xr-x  1 root wheel 25 Aug 28  2025 /usr/lib/libcrypto.so -> libcrypto.so.30

# find / -name 'libcrypto.so*' 2>/dev/null
/lib/libcrypto.so.30
/lib/libcrypto.so.111
/usr/lib/libcrypto.so

# ldd /usr/local/bin/php 2>/dev/null
/usr/local/bin/php:
        libcrypto.so.35 => not found (0)
        libssl.so.35 => not found (0)
        libelf.so.2 => /lib/libelf.so.2 (0x227f5100d000)
        libutil.so.10 => not found (0)
        libm.so.5 => /lib/libm.so.5 (0x227f52a55000)
        libthr.so.3 => /lib/libthr.so.3 (0x227f53513000)
        libxml2.so.16 => /usr/local/lib/libxml2.so.16 (0x227f5239a...)
        libpcre2-8.so.0 => /usr/local/lib/libpcre2-8.so.0 (0x227f51...)
        libz.so.6 => /lib/libz.so.6 (0x227f53ba0000)
        libc.so.7 => /lib/libc.so.7 (0x227f5542a000)
        libsys.so.7 => /lib/libsys.so.7 (0x227f55cf3000)

So php (and presumably pkg) now require libcrypto.so.35 / libssl.so.35 / libutil.so.10, but the base system on disk only has libcrypto.so.30 and libcrypto.so.111. The base OS libraries were apparently never updated to match the new package set, so the upgrade seems to have stalled partway through (base system not fully updated to FreeBSD 16-CURRENT / new OpenSSL before/while php and other packages were already upgraded).

Because PHP can't run, I cannot use the GUI, pfSense-repoc, or most recovery tools that depend on PHP.

What I've tried so far:

I ran the diagnostic commands above (ls, find, ldd) to confirm the missing libraries, but since PHP was broken I did not attempt any repair commands. Since this is a firewall in active use, I rolled back to my pre-upgrade Proxmox snapshot (2.8.1-RELEASE, fully functional) rather than risk further troubleshooting on a production VM.

Question:

Is this a known issue with the 2.9.0 upgrade path (base system / php library version mismatch), similar to past libcrypto.so.30 not found, required by php issues seen in earlier releases?
Is there a documented forced-reinstall procedure for base libs (similar to the forced pkg reinstall doc) that would fix this without a full reinstall, in case this happens again?
Is this specific to Proxmox VMs, or has it also been seen on physical hardware / other hypervisors?
Before I retry the upgrade, is there anything specific I should check or do differently (e.g. verify free disk space, package state, or network stability during the upgrade) to avoid hitting this again?

Happy to provide any additional debug output if it would help track this down — I still have the broken disk snapshot available if useful before I clean it up.

Thanks in advance.

abidsaleem92

We are also facing same issue on same upgrade from 2.8.1 to 2.9.0 on Proxmox 9 VM.
Whereas on our vSphere 7 vSAN cluster 2 out of 4 VMs are upgraded successfully.

We are running PFSense Master/Backup CARP clusters in our different security zones. Where some master are on Proxmox Ceph cluster and their backups nodes are on vSphere vSAN cluster and vice versa.

Can netgate admin or anyone else guide us how to solve this issue ?

stephenw10

Not a known issue. I would retry the upgrade from the cli using: pfSense-upgrade -d. That will give you far more debug output. If you do it from the serial console you will see the full upgrade process and hopefully where it's failing.

abidsaleem92

Dear PFSense community

We have solve this issue by tricking the system.
We copied the following 3 files from the upgraded 2.9.0 pfsense VMs to the 2.8.1 VM first and then start upgrade using GUI and the system booted successfully after upgrade.

libcrypto.so.35
libutil.so.10
libssl.so.35

You can get it from my google drive:
https://drive.google.com/file/d/1Iaj8FQzcj-9K-7YHMkqRVL_b7gyTzFcZ/view?usp=sharing

stephenw10

Do you have some unusual config or packages installed? I never hit that in countless 2.8.1 > 2.9.0 upgrades.

cabgrall

Thanks for the feedback.

I rolled back to my pre-upgrade Proxmox snapshot, so the system is currently back on a fully working pfSense CE 2.8.1-RELEASE.

I checked the package list, disk space, ZFS status, and network connectivity before attempting the upgrade again.

Installed additional packages

The main non-default packages I have installed are:

pfBlockerNG
Zabbix Agent 7
OpenVPN Client Export
Service Watchdog
Status Monitoring

I also have a number of additional packages/tools installed, including nginx, kea, strongswan, qemu-guest-agent, etc. The full pkg info output is available if useful.

The relevant pfSense packages currently installed are:

pfSense-2.8.1
pfSense-base-2.8.1
pfSense-boot-2.8.1
pfSense-kernel-pfSense-2.8.1
pfSense-repo-2.8.1
pfSense-upgrade-1.3.10

PHP is currently:

php83-8.3.19

Disk space

There is plenty of free disk space:

Filesystem                    Size    Used   Avail Capacity
pfSense/ROOT/default           33G    1.5G     32G     4%

The ZFS pool is healthy:

pool: pfSense
state: ONLINE

READ WRITE CKSUM
0    0     0

errors: No known data errors

The pool has approximately 32 GB available.

ZFS datasets

The ZFS layout appears normal, and there is plenty of available space across the datasets.

Network connectivity

Internet connectivity is working correctly:

ping -c 3 1.1.1.1

3 packets transmitted, 3 packets received, 0.0% packet loss

DNS resolution is also working. drill pkg.pfsense.org returns successfully with NOERROR.

So I don't currently see an obvious disk space, ZFS, or network connectivity issue that would explain why the previous upgrade stopped with the new PHP packages requiring:

libcrypto.so.35
libssl.so.35
libutil.so.10

while the base system still only contained the older libraries.

Before starting the new upgrade, is there anything else you would specifically recommend checking, especially regarding the installed packages listed above?

If the upgrade fails again, I will not reboot immediately and will capture the complete debug output and the exact point where the upgrade fails.

Thanks.

stephenw10

How is the service watchdog configured? It always raises some concerns seeing that.

cabgrall

Service Watchdog is configured to monitor only the following services:

pfb_filter — pfBlockerNG firewall filter service
pfb_dnsbl — pfBlockerNG DNSBL service
bsnmpd — SNMP service
dpinger — gateway monitoring daemon

There are no other services configured in Service Watchdog.

The Service Watchdog configuration is otherwise using the default behavior: automatically restart a monitored service if it crashes or stops.

Do you see any of these monitored services, particularly pfb_filter or pfb_dnsbl, as potentially problematic during the 2.8.1 → 2.9.0 upgrade?

If you recommend it, I can completely disable/remove the Service Watchdog configuration before retrying the upgrade.

stephenw10

I would certainly disable the Service Watchdog before upgrade. There's a good chance it's restarting something during the upgrade that should have been stopped and that's causing some problem.
In general I would only ever use the Service Watchdog for troubleshooting or some specific issue.

hassanktk

This post is deleted!

hassanktk

@abidsaleem92
Thanks a lot for sharing this! I tried the same workaround, and it worked perfectly for me. My pfSense 2.8.1 VM successfully upgraded to 2.9.0 and booted without any issues. Really appreciate your help!

cabgrall

### Update: 2.8.1 → 2.9.0 upgrade succeeded

Good news: I retried the upgrade from the CLI using:

```sh
pfSense-upgrade -d

This time the upgrade completed successfully, including the reboot, and the VM is now running pfSense CE 2.9.0-RELEASE.

The warning shown just before the reboot was:

pkg-static: Warning: Major OS version upgrade detected.
Running "pkg bootstrap -f" recommended
The following 255 package(s) will be affected (of 0 checked):

I did not manually run pkg bootstrap -f or copy any libraries from another system.

After the reboot, pfSense came back normally and the migration is complete.

For reference, the environment is unchanged:

Proxmox VE 9 VM
4 vCPU
6 GB RAM
4 VirtIO network interfaces
SeaBIOS
ZFS
Upgrade path: 2.8.1-RELEASE → 2.9.0-RELEASE

So, at this point, I cannot reproduce the original failure on a second attempt.

The initial failed upgrade resulted in missing:

libcrypto.so.35
libssl.so.35
libutil.so.10

with a partially upgraded system. The successful CLI upgrade did not require any manual intervention.

Thanks for the suggestions — especially the recommendation to retry using pfSense-upgrade -d. It appears to have made the difference, although I cannot say exactly what caused the first attempt to fail.