[WORKAROUND] Unable to update and package install (ERROR: It was not possible to identify which pfSense kernel is installed)

Sergei_Shablovsky

Recently making fresh install on bare metal server 2.7.0-DEVELOPMENT and after restore config from backup:

Unable to check update from webGUI menu;
Ping and a Traceroute to acb.netgate.com working ok
Unable to compare version of pfSense-repo

>>> Upgrading pkg... done.
>>> Updating repositories metadata... 
Updating pfSense-core repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: . done
Processing entries: . done
pfSense-core repository update completed. 7 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
pfSense repository update completed. 513 packages processed.
All repositories are up to date.
ERROR: Unable to compare version of pfSense-repo

Unable to update pfSense from VGA console (dumb menu 13)
ERROR: It was not possible to identify which pfSense kernel is installed

Notice that two main alerts persist:

SSH KeyGen

 pfSense has started creating missing SSH keys. SSH Startup will be delayed. Please note that reloading the filter rules and changes will be delayed until this operation is completed.	@ 2022-05-01 12:55:10

General

 An error occurred while uploading the encrypted configuration to https://acb.netgate.com/save (Operation timed out after 30000 milliseconds with 0 bytes received)	@ 2022-05-01 12:59:30
 An error occurred while uploading the encrypted configuration to https://acb.netgate.com/save (Operation timed out after 30000 milliseconds with 0 bytes received)	@ 2022-05-01 13:00:01
 An error occurred while uploading the encrypted configuration to https://acb.netgate.com/save (Operation timed out after 30045 milliseconds with 0 bytes received)	@ 2022-05-01 13:00:38

So the main question is: what the source of problem and how to resolve it?
Thank You all for attention and suggestions.

P.S. The old config file are not so old, approximately 1year (from 2.6.....)

Sergei_Shablovsky

Which sequence You recommend for step-by-step restoring from my old config?

May be in this step-by-step procedure possible to find source of a problem. But I not sure at all...

I try to grouping Restore from backup items by impact on each other or the same services (if there are some ERRORS please correct me):

Step 1 - SYSTEM INSTALL
Fresh 2.7.0-DEV CE install from memstick
Assign WAN and LAN interface for future control of device

Step 2 - PACKAGES INSTALL
Install all needed packages (not make settings for each, just install and hardware reboot after each package)

Step 3 - STEP-BY-STEP RESTORE FROM BACKUP (restore one config section at a time, then hardware reboot, then goes to the next)
(I collect backup config items in a groups, expect that items are impact on each other within group OR together all in group impact on whole system).
——————————-
Syslog
System Tunables
Interfaces
Wake-On-Lan
——————————-
Package Manager
Aliases
VLANS
——————————-
DNS Forwarder
DNS Resolver
DHCP Server
DHCPv6 Server
Dynamic DNS
——————————-
Captive Portal
Captive Portal Vouchers
NAT
Static routes
Firewall rules
Limiters
Traffic shaper
———————————
System
———————————
IPSEC
OpenVPN
———————————
RRD Data
SNMP Server
Scheduled tasks
——————————-

So, In my case looks like troubles happened right after System applied and hardware reboot. I try to check double twice, need more time....

ADDED
Find tread related to same problem at 2.4.X snapshot here, but hard to conclude how it may be related to my problem: I cannot use RAM disk and not see the issues with simlink that describes in a thread...

ADDED
Rarely (not depend on package, but depend on how fas button RESORE pressed) the RED ERROR NOTE appear:

The following input errors were detected:
Warning, could not read file /tmp/phpzRGrnm

After selecting restore section from drop-down list, choosing the backup config file, and pressing RESTORE again, normal GREEN NOTE appear.

The configuration area has been restored. The firewall may need to be rebooted.

ADDED
There are also some mismatching on SSH keys level, because after restoring System section from backup .xml and hardware rebooting not possible to logging in webGUI: browser pop-up with “This browser must to support cookies** appear.
And only after resetting webGUI password to default from local VGA console, logging in webGUI possible.

ADDED
Restoring the *System section from backup .xml lead to ALL PREVIOUSLY PACKAGES DISAPPEAR BUT ONLY IN PACKAGE MANAGER PAGE: in other top menus the previously installed packages name still in place and appropriate package's page able to load with previously restored from backup .xml file settings.

ADDED
Interfaces Groups not restored after restoring from backup: Interfaces restored but Interfaces groups - not.
But MORE IMPORTANT behavior:
After manually creating Interfaces Group, in Firewall / Rules - RULES FOR THIS GROUP APPEAR!

Sergei_Shablovsky

This post is deleted!

stephenw10

What does the output of pkg info actually show?

Sergei_Shablovsky

This post is deleted!

NollipfSense

@sergei_shablovsky Should post in Development 2.7 section so developers are aware of issues.

Sergei_Shablovsky

@stephenw10 said in Unable to update and package install (ERROR: It was not possible to identify which pfSense kernel is installed):

What does the output of pkg info actually show?

A lot of installed packages.

As You suggest:

Ok, you are missing the pfSense-repo and pfSense-upgrade packages.
So I would run:
pkg-static clean -ay; pkg-static install -fy pkg pfSense-repo pfSense-upgrade
As shown here: https://docs.netgate.com/pfsense/en/latest/troubleshooting/upgrades.html#forced-pkg-reinstall

As a result:

6 .pkg files from /root/var/cache/pkg/ was deleted
3 packages reinstalled
after reboot I still cannot see installed before packages (achieved...)

But now for me MUCH IMPORTANT that after restoring on a fresh 2.7.0 install from backup, after all packages installed manually, SOME SETTINGS RESTORE, BUT SOME - NO (I not mean whole section not restored, I mean SOME SETTINGS VITHIN CATEGORY, as in my case, some Firewall rules for Interfaces Groups not restored, and this Interfaces Groups not restored...and may be something else, who knows?)

So now the main question are: how to manually ensure that settings from backup successfully restored ?, this mean step by step check each settings in backup config...

Sergei_Shablovsky

@nollipfsense said in Unable to update and package install (ERROR: It was not possible to identify which pfSense kernel is installed):

@sergei_shablovsky Should post in Development 2.7 section so developers are aware of issues.

After CE's developers make a public version with missing “;” at the end of string (see discussion, but conclusion are “there are no sufficient code control at all”), I am not sure that make big sense... :)

stephenw10

You should always restore the complete config if you restoring into a new install.

Restoring sections will only restore those parts and, more importantly, they do not get put through the config updater so you can only restore into the same config version.

Steve

Sergei_Shablovsky

@stephenw10 said in Unable to update and package install (ERROR: It was not possible to identify which pfSense kernel is installed):

You should always restore the complete config if you restoring into a new install.

Restoring sections will only restore those parts and, more importantly, they do not get put through the config updater so you can only restore into the same config version.

Thank You for explanation, mr. Steve!

Please confirm: the ability to restoring certain section of backup .xml valid and make sense only if I need restore on the SAME SUBVERSION, 2.6.4 -> 2.6.4, 2.6.0 - > 2.6.0, 2.6.0 -> 2.6.4, BUT NOT THE DIFFERENT VERSION (major or minor) 2.6.4 -> 2.7.0, 2.6.0 -> 2.7.0, 2.6.4 -> 2.7.2.

I understand You correctly?

If Your answer would be “yes”, so let’s to note that after I making already as You suggest (restoring all from .xml backup file and then restart):

previously installed packages disappear in Package manager pages, and from all main menus;
I find that some settings disappear (Interfaces Groups for example, but may be more, how to find that?);

So because of this, next try I decide making step-by-step: restoring from .xml backup file 1(one) category at a time , then hardware rebooting, to understanding on which step there are problem come in, to eliminate “damage” if possible to say that... ;)

stephenw10

The part that matters is the config version. So if you look in the config file you will see:

<?xml version="1.0"?>
<pfsense>
	<version>22.6</version>

When you import an older config into a newer pfSense version it runs it through a series of upgrade scripts for each version change so that the final imported config is compatible with the system. However the config version only appears once in the file so if you import only a section of it that won't contain the version and it will not be upgraded. You may end up with a config that cannot be loaded.
I personally never import sections of config, it's far safer to import only the complete config file.

See: https://docs.netgate.com/pfsense/en/latest/backup/restore.html#restore-options

Steve

Sergei_Shablovsky

@stephenw10 said in Unable to update and package install (ERROR: It was not possible to identify which pfSense kernel is installed):

The part that matters is the config version. So if you look in the config file you will see:
<?xml version="1.0"?>
<pfsense>
	<version>22.6</version>
When you import an older config into a newer pfSense version it runs it through a series of upgrade scripts for each version change so that the final imported config is compatible with the system. However the config version only appears once in the file so if you import only a section of it that won't contain the version and it will not be upgraded. You may end up with a config that cannot be loaded.
I personally never import sections of config, it's far safer to import only the complete config file.

See: https://docs.netgate.com/pfsense/en/latest/backup/restore.html#restore-options

Thank You for so detailed explanation. Of course, I re-read ALL Docs again carefully with a cup of tee.

Thank You again for patience and attention to my problem.

Sergei_Shablovsky

@stephenw10
So now the main question now are: how to manually ensure that settings from backup successfully restored ?, this mean step by step check each settings in backup config...

Because a lot of settings in each package, I do not remember exactly settings, of course. (May be needed to make PDF copy of each page of settings next time ;)

stephenw10

All the package settings are stored in the main config file. If you restore the complete config file all the package settings will come with it.
The settings shown in the GUI are only what is loaded from the config at boot.

Steve

Sergei_Shablovsky

@stephenw10 said in Unable to update and package install (ERROR: It was not possible to identify which pfSense kernel is installed):

The part that matters is the config version. So if you look in the config file you will see:
<?xml version="1.0"?>
<pfsense>
	<version>22.6</version>
When you import an older config into a newer pfSense version it runs it through a series of upgrade scripts for each version change so that the final imported config is compatible with the system. However the config version only appears once in the file so if you import only a section of it that won't contain the version and it will not be upgraded. You may end up with a config that cannot be loaded.
I personally never import sections of config, it's far safer to import only the complete config file.

See: https://docs.netgate.com/pfsense/en/latest/backup/restore.html#restore-options

Thank You for suggestions, Steve!

So, I check double twice: install pfSense on a bare metal, reboot, then install all needed packages (reboot after each package), then backup from .xml backup file, wait 2h, then reboot, the result are the same: packages disappear from installed, not possible to update/upgrade from menu or manually, ping from pfSense CLI on monitoring 1.1.1.1 / 8.8.8.8 are ok, but ping / traceroute on other sites - no, LAN are working (surfing web as test), but sometimes some images not loaded...

Your suggestion?

P.S.

$ host -t srv _https._tcp.packages.netgate.com
;;connection timed out; no servers could be reached;

$ host files01.netgate.com.
;;connection timed out; no servers could be reached;

$ host files00.netgate.com.
;;connection timed out; no servers could be reached;

# pkg-static update -f
Updating pfSense-core repository catalogue...
pkg-static: https://packages-beta.netgate.com/packages/pfSense_master_amd64-core/meta.txz: No address record
repository pfSense-core has no meta file, using default settings
pkg-static: https://packages-beta.netgate.com/packages/pfSense_master_amd64-core/packagesite.pkg: No address record
pkg-static: https://packages-beta.netgate.com/packages/pfSense_master_amd64-core/packagesite.txz: No address record
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
pkg-static: https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel/meta.txz: No address record
repository pfSense has no meta file, using default settings
pkg-static: https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.pkg: No address record
pkg-static: https://packages-beta.netgate.com/packages/pfSense_master_amd64-pfSense_devel/packagesite.txz: No address record
Unable to update repository pfSense
Error updating repositories!

stephenw10

Are you installing 2.7?

The config you are restoring is setting the repo to next-dev-version so if you are installing 2.6 and then restoring that it will cause it to try to use the wrong repo and fail.

You don't need to install packages before restoring is will pull them in at the first boot after the install.

But it looks like you have some more general connectivity issue happening there. Those SRV records should resolve:

steve@steve-MMLP7AP-00 ~ $ host -t srv _https._tcp.packages.netgate.com
_https._tcp.packages.netgate.com has SRV record 10 10 443 pkg00-atx.netgate.com.
_https._tcp.packages.netgate.com has SRV record 10 10 443 pkg01-atx.netgate.com.
steve@steve-MMLP7AP-00 ~ $ host -t srv _https._tcp.packages-beta.netgate.com
_https._tcp.packages-beta.netgate.com has SRV record 10 10 443 pkg01-atx.netgate.com.
_https._tcp.packages-beta.netgate.com has SRV record 10 10 443 pkg00-atx.netgate.com.

Steve

NollipfSense

Please see Jim's note here below and why it's important to post in development section.

https://forum.netgate.com/topic/171891/update-failure-2-7-0-development-amd64-built-on-tue-apr-26-06-13-40-utc-2022-freebsd-12-3-stable

Sergei_Shablovsky

@stephenw10 said in Unable to update and package install (ERROR: It was not possible to identify which pfSense kernel is installed):

Are you installing 2.7?

Yes, pfSense 2.7 CE

The config you are restoring is setting the repo to next-dev-version so if you are installing 2.6 and then restoring that it will cause it to try to use the wrong repo and fail.

You don't need to install packages before restoring is will pull them in at the first boot after the install.

I trying several times with and without installing packages before restoring from .xml backup file.
Result are the same.

But it looks like you have some more general connectivity issue happening there. Those SRV records should resolve:

steve@steve-MMLP7AP-00 ~ $ host -t srv _https._tcp.packages.netgate.com
_https._tcp.packages.netgate.com has SRV record 10 10 443 pkg00-atx.netgate.com.
_https._tcp.packages.netgate.com has SRV record 10 10 443 pkg01-atx.netgate.com.
steve@steve-MMLP7AP-00 ~ $ host -t srv _https._tcp.packages-beta.netgate.com
_https._tcp.packages-beta.netgate.com has SRV record 10 10 443 pkg01-atx.netgate.com.
_https._tcp.packages-beta.netgate.com has SRV record 10 10 443 pkg00-atx.netgate.com.

Because ping/traceroute not working for anything excluding LANs, and ping are ok only for 1.1.1.1/8.8.8.8 I come with the same conclusion.

But where is the source of problem and how to find them?

Sergei_Shablovsky

@nollipfsense said in Unable to update and package install (ERROR: It was not possible to identify which pfSense kernel is installed):

Please Jim's note here below and why it's important to post in development section.

https://forum.netgate.com/topic/171891/update-failure-2-7-0-development-amd64-built-on-tue-apr-26-06-13-40-utc-2022-freebsd-12-3-stable

Thank You that point me.

Hm, I never imagine that Netgate was making so unstable upgrade...

stephenw10

@sergei_shablovsky said in Unable to update and package install (ERROR: It was not possible to identify which pfSense kernel is installed):

I never imagine that Netgate was making so unstable upgrade...

What?!

We specifically stopped updating the snapshot servers in order to shield end users from any instability these significant changes might introduce. The snapshot servers are still there and you can still pull pkgs from them. They are just not being updated currently. It will not prevent you accessing the servers.

You might be seeing an IPv6 issue. Try:

host -4t srv _https._tcp.packages-beta.netgate.com

Steve