Secondary can't see update info 2.6 (pfSense-core contains no repodata table ???)

MrPete

I have a pair of essentially identical pfSense's running CARP/pfSync. All is working nicely in 2.6... EXCEPT:

Primary, ALL is fine
Secondary (even when enabled as Master) is having strange strange issues in the update/upgrade arena.

GUI level symptoms:

can't see update info for pfSense (no matter if set to 2.6 or 2.7 target)
packages know an update is available, but refuse because there's a new php version and they want that before updating (yet the primary is happy to update?!)
both pfsense and packages complain about meta version

From shell, running a debug update check gives

ERROR: It was not possible to identify which pfSense meta package is installed
>>> Updating repositories metadata...
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
 version of pfSense is available

Forcing a pkg update doesn't help. (is the spurious "pkg not installed" just due to the forced reinstall?)

I'm getting the sense I have a catch 22 somewhere here. There are various complaints about OS versions... but it won't update either.

Here's an example:

[2.6.0-RELEASE][root@jasmine.ds.org]/cf/conf: pkg-static update -f
Updating pfSense-core repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%    2 KiB   1.8kB/s    00:01
Processing entries: 100%
pfSense-core repository update completed. 7 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%  156 KiB 159.4kB/s    00:01
Processing entries:   0%
pkg-static: Newer FreeBSD version for package unzoo:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1400085
- running kernel: 1203500

pkg-static: repository pfSense contains packages for wrong OS version: FreeBSD:14:amd64
Processing entries: 100%
Unable to update repository pfSense
Error updating repositories!

(BTW, while working on this, something completely trashed my config.xml -- I'm being careful to keep spare copies around! Fortunately I'm doing this in proxmox so restoration only took a few seconds...)

MrPete

Continuing:

I switched target to 2.6 - that gets rid of part of the issue with pkg-static update -f :

[2.6.0-RELEASE][root@jasmine.ds.org]/cf/conf: pkg-static update -f
pkg-static: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
Updating pfSense-core repository catalogue...
pkg-static: sqlite error while executing SELECT count(name) FROM sqlite_master WHERE type='table' AND name='repodata'; in file pkgdb.c:2358: database is locked
pkg-static: Repository pfSense-core contains no repodata table, need to re-create database
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%    2 KiB   1.7kB/s    00:01
Processing entries: 100%
pfSense-core repository update completed. 7 packages processed.
Updating pfSense repository catalogue...
pkg-static: Repository pfSense contains no repodata table, need to re-create database
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%  150 KiB 153.5kB/s    00:01
Processing entries: 100%
pfSense repository update completed. 520 packages processed.
All repositories are up to date.

BUT that sqlite error has me concerned. I'm at the end of my understanding and/or ability to confidently try stuff ;)

Gertjan

@MrPete said in Secondary can't see update info 2.6:

Forcing a pkg update doesn't help. (is the spurious "pkg not installed" just due to the forced reinstall?)

Don't.
Current packages are meant to be used for the current version of pfSense, that is, today, 2.7.2, not the very ancient 2.6.0.
As you already suspected : kernel, system libraries etc evolved over time as amelioration, bugs, security, etc does tend to change things over time. If one of these packages pulls in a library that is far more recent as the one you use on your 2.6.0, it will be a firework show.

When you decide to stop upgrading pfSense - and pfSense has been upgraded, then you have to stop installing / upgrading packages.

If you want to resume installing / updating, first, bring your pfSense to 'current'.

But I get it : Your '2.6.0' can't 'see' the upgrade path anymore.
See it as : I have Windows 7 and want to install 11 straight. Microsoft will show you a 0x0800xxxxxx.

Not a real issue, as there is a way out, or actually : in :

Export your config.
Download the greatest and latest.
Locate a USB stick and do the Etcher thing.
Insert the stick there where it's needed, and boot from it.
Install pfSense.
Assign a minimal interfaces LAN and WAN.
Connect to the GUI.
Import the backed up config - from that device.
Have it rebooted ones more.
Done.

Btw : you have a master slave setup. Never saw such a setup myself, but, AFAIK, first the samlve and then then master.
An do fact check on me please on this one, see the dedicated "how to upgrade a HA master slave setup" documentation first.

MrPete

@Gertjan that all makes sense, except for one thing: the OTHER (primary) box doesn't have any issues at all. It's on 2.6 as well...

It knows 2.7 exists and is willing to upgrade when I am ready
The package manager is happy to do upgrades as well

So if that's true, I suspect the real issue has more to do with the error message saying pfSense-core is missing a repodata table... ie a sqlite database problem.

(BTW, I didn't "decide to stop upgrading pfSense" ;) ... it wouldn't auto-upgrade to a major new release, and I've had a few distractions like the death of my 23 year coworker and close family friend to work through!)

Gertjan

@MrPete said in Secondary can't see update info 2.6 (pfSense-core contains no repodata table ???):

more to do with the error message saying pfSense-core is missing a repodata table... ie a sqlite database problem.

That's the real issue.
IMHO : don't even try to repair the database issue. It isn't worth the learning curve ^^
That's why I mentioned the more manual 'USB' upgrade.
You can upgrade your master the method you want, although (to be checked) : a new install proposes you the ZFS file system, which is more resilient as the older one that 2.6.0 used.

@MrPete said in Secondary can't see update info 2.6 (pfSense-core contains no repodata table ???):

BTW, I didn't "decide

Don't worry, every decision we make is made based on current personal conditions.
Let this one be an easy one for you

stephenw10

Hmm, that was running: pfSense-upgrade -dc ?

MrPete

@stephenw10 said in Secondary can't see update info 2.6 (pfSense-core contains no repodata table ???):

Hmm, that was running: pfSense-upgrade -dc ?

Yep. Just ran it again. NOTE: the first line below (ERROR...) is only in upgrade_log.txt -- not returned by pfSense-upgrade -dc.

ERROR: It was not possible to identify which pfSense meta package is installed
>>> Updating repositories metadata...
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
 version of pfSense is available

FWIW, as a data guy, it was not hard for me to go find the *.sqlite files and export to *.sql -- I don't know exactly what to expect of course, but they look reasonably similar on both systems.

I get the "cleanest" outcome by setting the target to 2.6 and running pkg-static -d update ... fewer errors but still the result is the same: the system won't actually do an update ("unable to check for updates" in the GUI). And ``pkg-static -d update` produces the same output. Metadata error, and blank version number in the log.

FWIW as a reminder, on 100% ok primary configured the same, this is the result of pkg-static -d update:

>>> Updating repositories metadata...
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
2.7.0 version of pfSense is available

@Gertjan is of course correct that I could start from scratch... but I have a hunch this upgrade issue will bite me again in the future. As a SW dev myself, I appreciate having 100% repeatable symptoms

stephenw10

Yeah, I'd guess it's pulled in some pkg the other system doesn't have.

Try running: pkg-static info -x pfsense on both nodes and comparing.

You may need to set it back to 2.6 and force reload something. Either the upgrade or repo pkg I'd guess.

Steve

MrPete

@stephenw10 wow is THAT a revelation! Guess what's missing from secondary... how can it even be running LOL:

pfSense-2.6.0
pfSense-base-2.6.0
pfSense-default-config-2.6.0
pfSense-kernel-pfSense-2.6.0
pfSense-rc-2.6.0
php74-pfSense-module-0.76

using pkg-static install -fy pfSense-2.6.0 pfSense-base-2.6.0 pfSense-default-config-2.6.0 pfSense-kernel-pfSense-2.6.0 pfSense-rc-2.6.0
then will add the new php74...

stephenw10

It actually doesn't have either of those pkgs?

Make sure the repo is set to 2.6 and install them if so.

MrPete

@stephenw10 YEP!

And voila, ALL is well again. THANK YOU @stephenw10

Easy peasy.

stephenw10

Nice!

MrPete

@stephenw10
Spoke too quickly. System upgrade to 2.7 failed w/ a pile of sqlite3 errors. TIme to install 2.7 from scratch. Something's seriously wrong beyond even those obvious things.

stephenw10

Hmm, disappointing. It's common to see a lot of errors at upgrade if you're looking at the console. It should still succeed of course!