Wrong kernel after fight with automatic update
-
System (wrap) had 2.3 installed, I used the gui option to automatically update to whatever it offered (like a typical user…).
Update broke midway, and I experienced many of the problems we can see in the forums for the past 2 months.After some time spent looking at forum and manually fixing packages, the following problems remain:
From the gui:
- /pkg_mgr_install.php?id=firmware reports Unable to retrieve system versions.
- /pkg_mgr_installed.php reports Unable to retrieve package information.
I now have the following packages installed (note kernel version):
pfSense-2.3.2 pfSense-Status_Monitoring-1.4.4_2 pfSense-default-config-serial-2.3.2 pfSense-kernel-pfSense_wrap-2.3 pfSense-rc-2.3.2 pfSense-repo-2.3.2 pkg-1.8.7_1
Using ssh option 13:
>>> Updating repositories metadata... Updating pfSense-core repository catalogue... pfSense-core repository is up-to-date. Updating pfSense repository catalogue... pfSense repository is up-to-date. All repositories are up-to-date. **** WARNING **** Duplicate slice required!! Before starting the upgrade process, the currently mounted nanobsd partition needs to be cloned to the secondary partition, where the update will happen After installation a reboot will be required to switch partition. Proceed with upgrade? (y/N) y >>> Cleaning secondary partition... done. >>> Duplicating current slice... done. >>> Restoring slice label... done. >>> Testing duplicated partition integrity... done. >>> Mounting second partition to run upgrade... done. >>> Unlocking package pfSense-kernel-pfSense_wrap... done. >>> Downloading upgrade packages... Updating pfSense-core repository catalogue... Unable to update repository pfSense-core Updating pfSense repository catalogue... Unable to update repository pfSense All repositories are up-to-date. pkg: Repository pfSense-core cannot be opened. 'pkg update' required pkg: Repository pfSense cannot be opened. 'pkg update' required Checking for upgrades (0 candidates): . done Processing candidates (0 candidates): . done Checking integrity... done (0 conflicting) Your packages are up to date.
So, since ipv6 has been suspected as being involved, I tried to avoid this potential problem. By using ssh menu option 8 and pfSense-upgrade with -4 option, we can discard ipv6 as being the suspect, right? Look at confusing error messages regarding repositories and failure to update kernel:
[2.3-RELEASE][root@pfap]/var/log: pfSense-upgrade -d4y >>> Updating repositories metadata... Updating pfSense-core repository catalogue... pfSense-core repository is up-to-date. Updating pfSense repository catalogue... pfSense repository is up-to-date. All repositories are up-to-date. **** WARNING **** Duplicate slice required!! Before starting the upgrade process, the currently mounted nanobsd partition needs to be cloned to the secondary partition, where the update will happen After installation a reboot will be required to switch partition. >>> Cleaning secondary partition... 1+0 records in 1+0 records out 1048576 bytes transferred in 0.134504 secs (7795868 bytes/sec) >>> Duplicating current slice... 14772+1 records in 14772+1 records out 968155648 bytes transferred in 83.459596 secs (11600292 bytes/sec) >>> Restoring slice label... >>> Testing duplicated partition integrity... ** /dev/ufs/pfsense0 ** Last Mounted on / ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 17705 files, 981697 used, 877661 free (5085 frags, 109072 blocks, 0.3% fragmentation) ***** FILE SYSTEM IS CLEAN ***** >>> Mounting second partition to run upgrade... >>> Unlocking package pfSense-kernel-pfSense_wrap... Unlocking pfSense-kernel-pfSense_wrap-2.3 >>> Downloading upgrade packages... Updating pfSense-core repository catalogue... Unable to update repository pfSense-core Updating pfSense repository catalogue... Unable to update repository pfSense All repositories are up-to-date. pkg: Repository pfSense-core cannot be opened. 'pkg update' required pkg: Repository pfSense cannot be opened. 'pkg update' required Checking for upgrades (0 candidates): . done Processing candidates (0 candidates): . done Checking integrity... done (0 conflicting) Your packages are up to date. pkg: Repository pfSense-core cannot be opened. 'pkg update' required pkg: Repository pfSense cannot be opened. 'pkg update' required pkg: No packages available to upgrade matching 'pfSense-kernel-pfSense_wrap' have been found in the repositories pkg: Repository pfSense-core cannot be opened. 'pkg update' required pkg: Repository pfSense cannot be opened. 'pkg update' required pkg: Repository pfSense-core cannot be opened. 'pkg update' required pkg: Repository pfSense cannot be opened. 'pkg update' required >>> Setting secondary partition as active... active set on ada0s1 Upgrade is complete. Rebooting in 10 seconds. Upgrade is complete. Rebooting in 10 seconds. >>> Locking package pfSense-kernel-pfSense_wrap... Locking pfSense-kernel-pfSense_wrap-2.3
Any hints on how to proceed to fix the remaining problems? (wrong kernel, broken firmware/packages gui)
Note that this pfsense system is only a dumb bridging AP, no rules, no extra packages, so there should be no difficulties.
Please do not advise to "just" reset/reinstall/reflash as I have many other systems to upgrade. -
Since my pfsense box is only a dumb AP, I can easily use tcpdump on upstream router.
What this shows is that no network traffic occurs while pfSense-upgrade script reports:
Updating pfSense-core repository catalogue... Unable to update repository pfSense-core
Puzzling.
So I start pfSense-upgrade script with debug:
sh -x /usr/local/sbin/pfSense-upgrade -d4y
Then I see that this is hanging when trying to update stuff IN THE CHROOT.
So I start a shell in the chroot because I'm tired of reading magical suggestions in the forum.
Then dopkg -ddd update -F
And I see that it can't retrieve the packages… with no useful error message.
Upstream router shows no traffic... even my nameservers shows nothing.Of course, when /etc/resolv.conf is a symlink to /var/etc/resolv.conf ... and /var/etc doesn't even exist, it doesn't work.
mkdir /var/etc echo "nameserver 8.8.8.8" > /var/etc/resolv.conf
Suddenly, upgrade works…
-
Still replying to myself…
Package pfSense-base was missing. I was sure I had this correct... but no (was it installed to some other slice?)
Adding this missing package (pkg install pfSense-base) fixed http://pfap.snet/pkg_mgr_install.php?id=firmware which was previously reporting "Unable to retrieve..."
(reference: /etc/inc/pkg-utils.inc get_base_pkg_name()).