Unable to check for updates from dashboard

SteveITS

@callinectes said in Unable to check for updates from dashboard:

Shared object "libssl.so.30" not found, required by "pkg"

That error often means you have tried to install or update a package with the update branch set to a later pfSense version. (it defaults to Current) That tries to update any required libraries.

https://redmine.pfsense.org/issues/10464

There are a couple suggestions in that redmine but otherwise one can reinstall and restore from backup.

stephenw10

Seeing that lib error usually only means the version of pkg has been updated. That's normal as it checks for a new version every time it looks for pfSense updates.
It shouldn't be a problem since the pfSense pkg commands all use pkg-static.

@callinectes said in Unable to check for updates from dashboard:

Updating repositories metadata...
pkg-static: Warning: Major OS version upgrade detected. Running "pkg bootstrap -f" recommended
Updating pfSense-core repository catalogue...
pkg-static: An error occured while fetching package

Seeing that I would try running pkg-static -d update to see what error is shown there.

callinectes

@stephenw10 Thanks for the suggestion. I received a lot of the following (too many similar entries to paste in but happy to if it would help:

Hostname pkg00-atx.netgate.com was found in DNS cache
Trying 208.123.73.207:443...
Connected to pkg00-atx.netgate.com (208.123.73.207) port 443 (#9)
ALPN: offers http/1.1
CAfile: none
CApath: /etc/ssl/certs/
SSL certificate problem: self-signed certificate in certificate chain
Closing connection 9
DBG(1)[86277]> CURL> attempting to fetch from , left retry 2
Hostname pkg01-atx.netgate.com was found in DNS cache
Trying 208.123.73.209:443...
Connected to pkg01-atx.netgate.com (208.123.73.209) port 443 (#10)
ALPN: offers http/1.1
CAfile: none
CApath: /etc/ssl/certs/
SSL certificate problem: self-signed certificate in certificate chain
Closing connection 10
DBG(1)[86277]> CURL> attempting to fetch from , left retry 1
Hostname pkg00-atx.netgate.com was found in DNS cache
Trying 208.123.73.207:443...
Connected to pkg00-atx.netgate.com (208.123.73.207) port 443 (#11)
ALPN: offers http/1.1
CAfile: none
CApath: /etc/ssl/certs/
SSL certificate problem: self-signed certificate in certificate chain
Closing connection 11
pkg-static: An error occured while fetching package
Unable to update repository pfSense
Error updating repositories!

stephenw10

Ah, what pfSense version are you running? That output is from the new development pkg version.

callinectes

@stephenw10 Per the dashboard "Version 2.6.0-RELEASE (amd64)"

But it was set to watch the 2.7.0 devel versions

Gertjan

@callinectes said in Unable to check for updates from dashboard:

watch the 2.7.0 devel versions

Strange.
That version doesn't exist any more since the end of June, as 'dev' became 'Release'.
On the other hand, if it can't update any more, it also can't update the list with available versions.

What about the the easy way out :
Get an USB drive, goto pfsense download - download your version 2.7.0.
Build teh USB drive.
Export the current config.
Boot the pfSEnse device from USB, install pfSense, 'all clean'.
When it reboot ... check that it shows a 'future version' (or ancient 2.6.0)
Check that the available package list populates.
Now, import your config.
Reboot.
Be very patient, as packages get installed in the background.
When you are sure it's done, do a reboot for good manners.
After reboot : test and check most needed and important system functionalities.

Make a post-it, and paste it on the pfSense box : "stay away from bleeding edge technology".

stephenw10

It is possible to recover from there but it requires some command line shenanigans.

What update branches do you see offered in System > Update?

callinectes

@stephenw10 It appears the way I would assume it would if 2.7.0 was installed. The drop down shows:

Latest stable version (v2.7.0)
DEVEL version (devel)
PREVIOUS version (v2.6.0)

stephenw10

OK, that's good, it has the correct repo pkg. Make sure the branch is set to 'Latest stable version (v2.7.0)'.

Check what version of pkg you have:

pkg-static info pkg

Then check for older cached versions:

ls /var/cache/pkg/pkg*

You will probably have version 1.20.4 installed and hopefully have an older version available.

callinectes

@stephenw10 Output, in order:

pkg-static info pkg
pkg-1.20.2
Name           : pkg
Version        : 1.20.2
Installed on   : Tue Jul 11 22:53:08 2023 EDT
Origin         : ports-mgmt/pkg
Architecture   : FreeBSD:14:amd64
Prefix         : /usr/local
Categories     : ports-mgmt
Licenses       : BSD2CLAUSE
Maintainer     : pkg@FreeBSD.org
WWW            : https://github.com/freebsd/pkg
Comment        : Package manager
Options        :
        DOCS           : off
Shared Libs provided:
        libpkg.so.4
Annotations    :
        FreeBSD_version: 1400093
        build_timestamp: 2023-07-11T22:24:13+0000
        built_by       : poudriere-git-3.3.99.20220831
        port_checkout_unclean: no
        port_git_hash  : 09a785875a27
        ports_top_checkout_unclean: yes
        ports_top_git_hash: c959ae811528
        repo_type      : binary
        repository     : pfSense
Flat size      : 38.8MiB
Description    :
Package management tool

WWW: https://github.com/freebsd/pkg

ls /var/cache/pkg/pkg*
/var/cache/pkg/pkg-1.17.5_3.pkg
/var/cache/pkg/pkg-1.17.5_3~b8e15d34b2.pkg
/var/cache/pkg/pkg-1.19.1_1.pkg
/var/cache/pkg/pkg-1.19.1_1~d54ac0ca75.pkg
/var/cache/pkg/pkg-1.20.2.pkg
/var/cache/pkg/pkg-1.20.2~269d0c3235.pkg

stephenw10

Ok, good. Make sure the update branch is set to Latest stable (2.7) again. Then force the pkg downgrade at the command line with:

pkg-static add -f /var/cache/pkg/pkg-1.19.1_1.pkg

Then recheck for upgrades.

callinectes

@stephenw10 Awesome - appears that worked:
"

"
I can't run the upgrade right now but I believe you've helped me across the only hurdle. Greatly appreciated!

stephenw10

Nice

Nibblet

Any updates for me.. ?

stephenw10

You're still seeing the dash report 'unable to check' but everything working at the CLI?

Have you tried setting IPv4 as preferred?

Nibblet

@stephenw10 That is correct. Upgrade from 2.6.0 to 2.7.0 worked fine from the CLI, but the dashboard and the update (system -> update) menu item are still showing "Unable to check for updates"

Sadly I cannot change to IPv4 as preferred as - due to the need to resolve items internally, and the complete lack of 'internal' IPv4 - this is why I need to perform 'smoke and mirrors' tricks to emulate the netgate update system in order to then point to proxies in areas of our network that DO have IPv4 connectivity out.

If there is a way to tell the update check widget to ignore system resolvers and use an internet based system (similar to squids dns_nameservers directive) then I could do this.

This is why I was wanting to know the underlying mechanism that the update widget is employing to check - as, whatever it is seems different from the pkg/pkg-static command line utilities as these are working fine.
This is a diagram of the setup - the DNS server has a 'dummy' version of the netgate SVC records that return a DNS name for the pkg servers, and the proxy is used to reach these. The proxy is squid and has the afor-mentioned DNS directive such that the proxy can resolve internet entries and reach them, yet still be part of the internal domain.

This used to work for the widget, but broke somewhere during 2.6.0 and after the transition from files00 and files01 to the new domain name

alt text

SteveITS

@Nibblet re DNS for pfSense: https://docs.netgate.com/pfsense/en/latest/config/general.html#dns-resolution-behavior

Re: IPv4: https://docs.netgate.com/pfsense/en/latest/config/advanced-networking.html#prefer-ipv4-over-ipv6
“ this option causes the firewall itself to prefer sending traffic to IPv4 hosts instead of IPv6 hosts when a DNS query returns results for both.” Doesn’t affect LAN devices.

Nibblet

@SteveITS I guess the point here is - that the firewall should never actually get an IPv4 A record response.
The DNS server in this example is not recursive and doesn't actually have A records to return.

stephenw10

@Nibblet said in Unable to check for updates from dashboard:

this is why I need to perform 'smoke and mirrors' tricks to emulate the netgate update system in order to then point to proxies in areas of our network that DO have IPv4 connectivity out.

Ha well that seems likely to be involved in this error situation!

We are digging into a problem internally though, specifically when using external proxies. Do you have a proxy configured in pfSense?

If you run pkg-static -d update via Diag > Command Prompt does it fail there when it succeeds at the real command line?

Steve

Nibblet

@stephenw10 Really weirdly, it has all started working again. Nothing changed from the perspective of DNS/proxy settings - very strange.

Also, running the update -d command would have worked, except of course that these servers were already updated..

Of note - these are in production with around 40-50 people actively connecting through them so I REALLY don't like running arbitrary commands that update packages. Hopefully this will put to bed that whatever pkg (and pkg-static) and the 'check of updates' widget are doing is different, and the widget is not calling the pkg (or pkg-static) command but rather doing its own thing - do you know where the source code for this widget would be? is there a specific package I can look the sources up for?

DBG(1)[33734]> pkg initialized
Updating pfSense-core repository catalogue...
DBG(1)[33734]> PkgRepo: verifying update for pfSense-core
DBG(1)[33734]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense-core.sqlite'
DBG(1)[33734]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_0_amd64-core/meta.conf
DBG(1)[33734]> opening libfetch fetcher
DBG(1)[33734]> Fetch > libfetch: connecting
DBG(1)[33734]> Fetch: fetching from: https://pkg01-atx.netgate.com/pfSense_v2_7_0_amd64-core/meta.conf with opts "i"
DBG(1)[33734]> Fetch: fetcher chosen: https
DBG(1)[33734]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_0_amd64-core/packagesite.pkg
DBG(1)[33734]> opening libfetch fetcher
DBG(1)[33734]> Fetch > libfetch: connecting
DBG(1)[33734]> Fetch: fetching from: https://pkg01-atx.netgate.com/pfSense_v2_7_0_amd64-core/packagesite.pkg with opts "i"
DBG(1)[33734]> Fetch: fetcher chosen: https
DBG(1)[33734]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_0_amd64-core/packagesite.txz
DBG(1)[33734]> opening libfetch fetcher
DBG(1)[33734]> Fetch > libfetch: connecting
DBG(1)[33734]> Fetch: fetching from: https://pkg01-atx.netgate.com/pfSense_v2_7_0_amd64-core/packagesite.txz with opts "i"
DBG(1)[33734]> Fetch: fetcher chosen: https
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
DBG(1)[33734]> PkgRepo: verifying update for pfSense
DBG(1)[33734]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense.sqlite'
DBG(1)[33734]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_0_amd64-pfSense_v2_7_0/meta.conf
DBG(1)[33734]> opening libfetch fetcher
DBG(1)[33734]> Fetch > libfetch: connecting
DBG(1)[33734]> Fetch: fetching from: https://pkg00-atx.netgate.com/pfSense_v2_7_0_amd64-pfSense_v2_7_0/meta.conf with opts "i"
DBG(1)[33734]> Fetch: fetcher chosen: https
DBG(1)[33734]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_0_amd64-pfSense_v2_7_0/packagesite.pkg
DBG(1)[33734]> opening libfetch fetcher
DBG(1)[33734]> Fetch > libfetch: connecting
DBG(1)[33734]> Fetch: fetching from: https://pkg00-atx.netgate.com/pfSense_v2_7_0_amd64-pfSense_v2_7_0/packagesite.pkg with opts "i"
DBG(1)[33734]> Fetch: fetcher chosen: https
DBG(1)[33734]> Request to fetch pkg+https://pkg.pfsense.org/pfSense_v2_7_0_amd64-pfSense_v2_7_0/packagesite.txz
DBG(1)[33734]> opening libfetch fetcher
DBG(1)[33734]> Fetch > libfetch: connecting
DBG(1)[33734]> Fetch: fetching from: https://pkg00-atx.netgate.com/pfSense_v2_7_0_amd64-pfSense_v2_7_0/packagesite.txz with opts "i"
DBG(1)[33734]> Fetch: fetcher chosen: https
pfSense repository is up to date.
All repositories are up to date.