Wrong kernel after fight with automatic update

fruitypulp

System (wrap) had 2.3 installed, I used the gui option to automatically update to whatever it offered (like a typical user…).
Update broke midway, and I experienced many of the problems we can see in the forums for the past 2 months.

After some time spent looking at forum and manually fixing packages, the following problems remain:

From the gui:

/pkg_mgr_install.php?id=firmware reports Unable to retrieve system versions.
/pkg_mgr_installed.php reports Unable to retrieve package information.

I now have the following packages installed (note kernel version):


pfSense-2.3.2
pfSense-Status_Monitoring-1.4.4_2
pfSense-default-config-serial-2.3.2
pfSense-kernel-pfSense_wrap-2.3
pfSense-rc-2.3.2
pfSense-repo-2.3.2
pkg-1.8.7_1

Using ssh option 13:


>>> Updating repositories metadata... 
Updating pfSense-core repository catalogue...
pfSense-core repository is up-to-date.
Updating pfSense repository catalogue...
pfSense repository is up-to-date.
All repositories are up-to-date.
**** WARNING ****
Duplicate slice required!!

Before starting the upgrade process, the currently mounted nanobsd partition
needs to be cloned to the secondary partition, where the update will happen

After installation a reboot will be required to switch partition.

Proceed with upgrade? (y/N) y
>>> Cleaning secondary partition... done.
>>> Duplicating current slice... done.
>>> Restoring slice label... done.
>>> Testing duplicated partition integrity... done.
>>> Mounting second partition to run upgrade... done.
>>> Unlocking package pfSense-kernel-pfSense_wrap... done.
>>> Downloading upgrade packages... 
Updating pfSense-core repository catalogue...
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
Unable to update repository pfSense
All repositories are up-to-date.
pkg: Repository pfSense-core cannot be opened. 'pkg update' required
pkg: Repository pfSense cannot be opened. 'pkg update' required
Checking for upgrades (0 candidates): . done
Processing candidates (0 candidates): . done
Checking integrity... done (0 conflicting)
Your packages are up to date.

So, since ipv6 has been suspected as being involved, I tried to avoid this potential problem. By using ssh menu option 8 and pfSense-upgrade with -4 option, we can discard ipv6 as being the suspect, right? Look at confusing error messages regarding repositories and failure to update kernel:


[2.3-RELEASE][root@pfap]/var/log: pfSense-upgrade -d4y
>>> Updating repositories metadata... 
Updating pfSense-core repository catalogue...
pfSense-core repository is up-to-date.
Updating pfSense repository catalogue...
pfSense repository is up-to-date.
All repositories are up-to-date.
**** WARNING ****
Duplicate slice required!!

Before starting the upgrade process, the currently mounted nanobsd partition
needs to be cloned to the secondary partition, where the update will happen

After installation a reboot will be required to switch partition.

>>> Cleaning secondary partition... 
1+0 records in
1+0 records out
1048576 bytes transferred in 0.134504 secs (7795868 bytes/sec)
>>> Duplicating current slice... 
14772+1 records in
14772+1 records out
968155648 bytes transferred in 83.459596 secs (11600292 bytes/sec)
>>> Restoring slice label... 
>>> Testing duplicated partition integrity... 
** /dev/ufs/pfsense0
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
17705 files, 981697 used, 877661 free (5085 frags, 109072 blocks, 0.3% fragmentation)

***** FILE SYSTEM IS CLEAN *****
>>> Mounting second partition to run upgrade... 
>>> Unlocking package pfSense-kernel-pfSense_wrap... 
Unlocking pfSense-kernel-pfSense_wrap-2.3
>>> Downloading upgrade packages... 
Updating pfSense-core repository catalogue...
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
Unable to update repository pfSense
All repositories are up-to-date.
pkg: Repository pfSense-core cannot be opened. 'pkg update' required
pkg: Repository pfSense cannot be opened. 'pkg update' required
Checking for upgrades (0 candidates): . done
Processing candidates (0 candidates): . done
Checking integrity... done (0 conflicting)
Your packages are up to date.
pkg: Repository pfSense-core cannot be opened. 'pkg update' required
pkg: Repository pfSense cannot be opened. 'pkg update' required
pkg: No packages available to upgrade matching 'pfSense-kernel-pfSense_wrap' have been found in the repositories
pkg: Repository pfSense-core cannot be opened. 'pkg update' required
pkg: Repository pfSense cannot be opened. 'pkg update' required
pkg: Repository pfSense-core cannot be opened. 'pkg update' required
pkg: Repository pfSense cannot be opened. 'pkg update' required
>>> Setting secondary partition as active... 
active set on ada0s1
Upgrade is complete.  Rebooting in 10 seconds.
Upgrade is complete.  Rebooting in 10 seconds.                                 
>>> Locking package pfSense-kernel-pfSense_wrap... 
Locking pfSense-kernel-pfSense_wrap-2.3

Any hints on how to proceed to fix the remaining problems? (wrong kernel, broken firmware/packages gui)

Note that this pfsense system is only a dumb bridging AP, no rules, no extra packages, so there should be no difficulties.
Please do not advise to "just" reset/reinstall/reflash as I have many other systems to upgrade.

fruitypulp

Since my pfsense box is only a dumb AP, I can easily use tcpdump on upstream router.

What this shows is that no network traffic occurs while pfSense-upgrade script reports:


Updating pfSense-core repository catalogue...
Unable to update repository pfSense-core

Puzzling.

So I start pfSense-upgrade script with debug:


sh -x /usr/local/sbin/pfSense-upgrade -d4y

Then I see that this is hanging when trying to update stuff IN THE CHROOT.

So I start a shell in the chroot because I'm tired of reading magical suggestions in the forum.
Then do


pkg -ddd update -F

And I see that it can't retrieve the packages… with no useful error message.
Upstream router shows no traffic... even my nameservers shows nothing.

Of course, when /etc/resolv.conf is a symlink to /var/etc/resolv.conf ... and /var/etc doesn't even exist, it doesn't work.


mkdir /var/etc
echo "nameserver 8.8.8.8" > /var/etc/resolv.conf

Suddenly, upgrade works…

fruitypulp

Still replying to myself…

Package pfSense-base was missing. I was sure I had this correct... but no (was it installed to some other slice?)

Adding this missing package (pkg install pfSense-base) fixed http://pfap.snet/pkg_mgr_install.php?id=firmware which was previously reporting "Unable to retrieve..."

(reference: /etc/inc/pkg-utils.inc get_base_pkg_name()).