[SOLVED] Update 2.3.4 to 2.4 failed ("Unable to check for updates", …)

felix.wolfsteller

Hey there

we converted a NanoBSD into a full 64b installation using the procedure from the wiki (boot slices, script).
We ran the update, which failed due to timeouts during package download 2 or 3 times.
After reboot, the GUI still shows version "2.3.4-RELEASE-p1 (amd64)".

Afterwards we

disabled the RAM Disk feature (shouldnt this be proposed in the nanobsd->full-wiki), as logging behaviour and other stuff seemed weird.
deleted a /var/db/pkg symlink mess as found in another thread
switched System->Update->Update Settings branch back and forth (no difference)
Tried to update via pfSense-upgrade and pkg

So far the update attempts fail.
I will post some command output which seems helpful to debug the situation (unfortunately I have no deeper BSD knowledge)


$ pkg info
pkg: Warning: Major OS version upgrade detected.  Running "pkg-static install -f pkg" recommended


$ pkg-static install -f pkg
pkg-static install -f pkg
pkg-static: Warning: Major OS version upgrade detected.  Running "pkg-static install -f pkg" recommended
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	pkg: 1.10.1_1 [pfSense]
# .... we hit 'y' and 'Enter', nothing happens, process seem to hang


$  pfSense-upgrade -d
pkg-static: Warning: Major OS version upgrade detected.  Running "pkg-static install -f pkg" recommended
pkg-static: No package(s) matching pfSense-kernel-*
pkg-static: Warning: Major OS version upgrade detected.  Running "pkg-static install -f pkg" recommended
ERROR: It was not possible to identify which pfSense kernel is installed


$ ls -lha /var/db/pkg
total 1559
drwxr-xr-x   2 root  wheel   512B Oct 21 16:32 .
drwxr-xr-x  13 root  wheel   1.5K Oct 21 15:46 ..
-rw-r--r--   1 root  wheel   352K Oct 21 16:32 local.sqlite
-rw-r--r--   1 root  wheel   246B Oct 10 14:19 pfSense-core.meta
-rw-r--r--   1 root  wheel   246B Oct 21 12:57 pfSense.meta
lrwxr-xr-x   1 root  wheel    21B Oct 21 02:27 pkg -> ../../root/var/db/pkg
-rw-r--r--   1 root  wheel   208K Oct 10 14:19 repo-pfSense-core.sqlite
-rw-r--r--   1 root  wheel   984K Oct 21 12:57 repo-pfSense.sqlite

Any idea how to debug this further?

Steps to resolve

1. Cleaned up link-mess in /var/cache/pkg

$ ls -lha /var/cache/pkg 
lrwxr-xr-x  1 root  wheel    24B Oct 21 02:27 /var/cache/pkg -> ../../root/var/cache/pkg

2. Delete an ".empty" file in /usr/local/share/pfSense/keys/pkg/revoked/ (check below for other key-issues)

3. Installed packages via pkg-static and upgrade

pkg-static update -f
pkg-static install -f pkg
pkg-static install -y pfSense pfSense-base pfSense-repo pfSense-kernel-pfSense pfSense-rc pfSense-repo pfSense-upgrade

pkg-static unlock pfSense-kernel-pfSense
pkg-static upgrade -f
pkg-static lock pfSense-kernel-pfSense

pfSense-upgrade -d

4. Installed and Removed acme package via GUI to get rid of crash reports

Thanks everybody who helped.
Btw update to 2.4.1 went smooth afterwards, too.

discostur

Hi,

having exact the same problem!

PiBa

Internet connectivity from pfSense box itself works fine?

Check if you have any 0 byte packages in the /var/pkg/cache , and perhaps a file called 'empty' in the

If you see any zero byte files in the directories below, such as a file named ".empty", remove them.
/usr/local/share/pfSense/keys/pkg/trusted/
/usr/local/share/pfSense/keys/pkg/revoked/
https://forum.pfsense.org/index.php?topic=136950.msg749972#msg749972

If above doesn't help, perhaps run "truss -f -o /root/pkg_truss.log pkg-static install -f pkg" and attach the file that gets created. It might tell what pkg is trying to do..

JohnnyBeGood

Hi all,

Had same issue while trying to upgrade using web interface. Rebooted pfSense using web interface and tried again and it worked!
Here's the successful upgrade log https://pastebin.com/b0j8JEzy

felix.wolfsteller

@PiBa:

Internet connectivity from pfSense box itself works fine?

Yes, it did.

@PiBa:

Check if you have any 0 byte packages in the /var/pkg/cache , and perhaps a file called 'empty' in the

You mean /var/cache/pkg:


$ ls -lha /var/cache/pkg 
lrwxr-xr-x  1 root  wheel    24B Oct 21 02:27 /var/cache/pkg -> ../../root/var/cache/pkg

Probably again such a symlink thing (target does not exist). I deleted (and mkdir'd) /var/cache/pkg; afterwards pkg-static install -f pkg succeeded.

Now /var/cache/pkg looks like this:

$ ls -lha /var/cache/pkg/
total 2917
drwxr-xr-x 2 root wheel 512B Oct 22 12:37 .
drwxr-xr-x 3 root wheel 512B Oct 22 12:37 ..
-rw-r–r-- 1 root wheel 2.8M Oct 8 23:44 pkg-1.10.1_1-5a32648e84.txz
lrwxr-xr-x 1 root wheel 27B Oct 22 12:37 pkg-1.10.1_1.txz -> pkg-1.10.1_1-5a32648e84.txz

@PiBa:

If you see any zero byte files in the directories below, such as a file named ".empty", remove them.
/usr/local/share/pfSense/keys/pkg/trusted/
/usr/local/share/pfSense/keys/pkg/revoked/
https://forum.pfsense.org/index.php?topic=136950.msg749972#msg749972

In "trusted" there are two 95B files, in "revoked" one .empty, which i deleted.

@PiBa:

If above doesn't help, perhaps run "truss -f -o /root/pkg_truss.log pkg-static install -f pkg" and attach the file that gets created. It might tell what pkg is trying to do..

I did not do this because "pkg-static install -f pkg" succeeded after the cache-symlink-cleanup (see above)

Thanks a lot PiBa! At least this allowed to go one step further. The GUI still shows "unable to check for updates", but at least the pkg thing seems to be clean now. I guess I could start the update via shell (pfSense-upgrade?) now - will try this when a slightly more relaxed time slot falls into my life. soon.

felix.wolfsteller

I though I'd at least check if pfSense-upgrade would show me available package updates. Following happens:


$ pfSense-upgrade 
pkg-static: No package(s) matching pfSense-kernel-*
>>> Updating repositories metadata... 
Updating pfSense-core repository catalogue...
pkg-static: Repository pfSense-core has a wrong packagesite, need to re-create database
Fetching meta.txz: . done
Fetching packagesite.txz: . done
Processing entries: . done
pfSense-core repository update completed. 7 packages processed.
Updating pfSense repository catalogue...
pkg-static: Repository pfSense has a wrong packagesite, need to re-create database
Fetching meta.txz: . done
Fetching packagesite.txz: .......... done
Processing entries: .......... done
pfSense repository update completed. 485 packages processed.
All repositories are up to date.
ERROR: Unable to compare version of pfSense-repo

That seems not so dramatic, but following scares me quite a lot:


$ pkg info
Shared object "libssl.so.8" not found, required by "pkg"

[code]
$ find / -name "libssl.so*"
/usr/lib/libssl.so.7
/usr/lib/libssl.so
[/code]

[code]
$ pkg update
Shared object "libssl.so.8" not found, required by "pkg"
[/code]

I guess I should not reboot that box now ... ? :)
Any help appreciated. The cheapest try would be to symlink `ln -s /usr/lib/libssl.so /usr/lib/libssl.so.8` - but this really gives me goosebumps.

Grimson

Just do a fresh 2.4 install and recover/restore the config backup, saves a lot of time and you are sure the system is in a clean state.

PiBa

A reinstall would probably be best indeed.

If you want to keep trying to salvage the current running installation then some commands to try below. But be prepared to reinstall anyhow as it might break functionality further..


pkg-static update -f
pkg-static install -f pkg
pkg-static install -y pfSense pfSense-base pfSense-repo pfSense-kernel-pfSense pfSense-rc pfSense-repo pfSense-upgrade

pkg-static unlock pfSense-kernel-pfSense
pkg-static upgrade -f
pkg-static lock pfSense-kernel-pfSense

pfSense-upgrade -d

felix.wolfsteller

@PiBa:

A reinstall would probably be best indeed.

If you want to keep trying to salvage the current running installation then some commands to try below. But be prepared to reinstall anyhow as it might break functionality further..
pkg-static update -f
pkg-static install -f pkg
pkg-static install -y pfSense pfSense-base pfSense-repo pfSense-kernel-pfSense pfSense-rc pfSense-repo pfSense-upgrade

pkg-static unlock pfSense-kernel-pfSense
pkg-static upgrade -f
pkg-static lock pfSense-kernel-pfSense

pfSense-upgrade -d

Okay. Our users are allergic to downtimes, so this needs a bit of preparation. Will post the outcome and edit original question if solution is found within PiBas Wisdom (thanks a lot!).

felix.wolfsteller

Awesome PiBa, that worked (I executed all of the commands). Got quite some further warnings about libssl, libcrypto etc pp but the box rebooted and the logs look fine to me! The GU shows "version 2.4.0" and "latest version". Is there anything to do to clean thing up?

Once I gained confidence that the box behaves stable, I will update the original Post.

felix.wolfsteller

This is the relevant crash info (appears on every reboot). Creating a symlink to libssl.so.{8,7} leads to similar issues (I think then libcrypto.so is missing).
I found various other threads about this issue (https://forum.pfsense.org/index.php?topic=137941 / ACME, https://forum.pfsense.org/index.php?topic=137939) but did not dare to dive into it - for now the system seems okay. I cross fingers for a 2.4.1 update moving everything back into place.


			Crash report begins.  Anonymous machine information:

amd64
11.1-RELEASE-p1
FreeBSD 11.1-RELEASE-p1 #83 r313908+d77c47fe50c(RELENG_2_4_0): Tue Oct 10 06:48:42 CDT 2017     root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense

Crash report details:

PHP Errors:
[23-Oct-2017 07:25:06 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:06 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:08 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:34 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:34 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:34 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:36 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:36 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:37 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:37 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:43 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0

No FreeBSD crash data found.

PiBa

Installing acme package from system\packagemanager and then removing it 'should' be a safe action to take..

echu2016

I got crazy to with 4 PFSense VMs…..
After a day, I realized the way to go...
The problem begins when following the suggestion "Major OS blahblah..." and installing pkg again.... Because, it installs FreeBSD 11 pkg package.
So, to make the upgrade work again, and not to deal with a manual upgrade what I did (worked) was:

cat /usr/local/sbin/pkg-static > /usr/sbin/pkg

This is, switching back (manually) to pkg-static version.
After issuing this command, upgrade should be done automatically as usual.
Try to issue a pkg -v before and after the command I wrote here, notice de difference.
Regards,

we converted a NanoBSD into a full 64b installation using the procedure from the wiki (boot slices, script).

Netgates Blog - pfSense version 2.4.0 is here

Important Information
_A few noteworthy items about running or upgrading to pfSense 2.4:

32-bit x86 and NanoBSD have been deprecated and are not supported on pfSense 2.4.
Hardware capable of running 64-bit images should be reinstalled with a 64-bit version.
32-bit x86 hardware can continue to run pfSense software version 2.3.x, which will receive
security updates for at least a year after pfSense 2.4.0-RELEASE.
NanoBSD installs on 64-bit hardware must be changed to run a full installation. This can be
accomplished by reinstalling or by following the manual upgrade procedure to convert NanoBSD to a full installation.
To use ZFS, a reinstall of the operating system is required. It is not possible to upgrade in-place from UFS to ZFS at this time.
Wireless interfaces must be created on the Wireless tab under Interfaces > Assignments before they are
available for assignment
Some hardware devices may not boot pfSense 2.4.0 installation images, for example, due to UEFI compatibility changes.
These are primarily BIOS issues and not issues with the installer images. Upgrading in place from pfSense 2.3.x typically
allows affected hardware to run version pfSense 2.4.x. In some cases, manually adjusting the installer images can allow
the hardware to boot._

A clean, fresh and full install was causing only ~30 minutes with a configuration play back!
I was choosing to install and use ZFS and so it was a "must be" with the fresh install but
without any problems, the upgrade from 2.4.0 to 2.4.1 was also not a nice trip here,
and I mean this really, so once more again I was installing fresh, and full and now I
am staying here on 2.4.1 so long as I can do it!

We ran the update, which failed due to timeouts during package download 2 or 3 times.

Deinstalling all packages and reinstalling all packages will be needing the same time to do a clean, fresh and full
install and play back the config.

After reboot, the GUI still shows version "2.3.4-RELEASE-p1 (amd64)".

Try out a fresh and full install against all other things, it not really popular but faster then all other methods
brings you fully ZFS support and you get not into trouble with IPSec VPN too likes much other users.

felix.wolfsteller

Thanks everybody. Updated the issue and marked as solved. I was able to have a (for now) stable box without a fresh install.
I think it remains very important that updates are possible (think of remote hardware) and the devs and users should know where updates are tricky - to solve issues and improve behaviour in the future.
Of course fresh installations might be easier and faster and thanks god export and import of the configuration works.

talon4x4

@felix.wolfsteller:

Steps to resolve

1. Cleaned up link-mess in /var/cache/pkg
$ ls -lha /var/cache/pkg 
lrwxr-xr-x  1 root  wheel    24B Oct 21 02:27 /var/cache/pkg -> ../../root/var/cache/pkg
2. Delete an ".empty" file in /usr/local/share/pfSense/keys/pkg/revoked/ (check below for other key-issues)

3. Installed packages via pkg-static and upgrade
pkg-static update -f
pkg-static install -f pkg
pkg-static install -y pfSense pfSense-base pfSense-repo pfSense-kernel-pfSense pfSense-rc pfSense-repo pfSense-upgrade

pkg-static unlock pfSense-kernel-pfSense
pkg-static upgrade -f
pkg-static lock pfSense-kernel-pfSense

pfSense-upgrade -d
4. Installed and Removed acme package via GUI to get rid of crash reports

Just wanted to thank everyone for sharing this fix. It worked perfectly and I'm now upgraded to v2.4.2

AllGamer

Thank you OP, this was a nice learning experience.

Well worse case scenario, I only had to re-install from media, then restore the config xml file. as I've done in previous failed major version upgrade.

but is good to learn how to re-cover from a bad upgrade.

Almost every major version upgrade like this 2.3.4 to 2.4, or 2.x to 2.3, and all previous major number changes, seems to have problem upgrading properly.