Unable to check for updates

bmeeks · Feb 25, 2023, 12:12 AM

@mission-ghost said in Unable to check for updates:

@bmeeks It'd be great if pfSense would give us more detailed messages about the updates. I spent four hours chasing the previous bug that stated my 1100 was unable to check for updates when it turns out it could check for updates, but the update had been withdrawn. A message to that effect would be very helpful. Google searches weren't returning this thread either. Could netgate's SQA use some invigorating, too?

I can't speak to any Netgate issues. But it may be that the pkg utility and associated infrastructure is limited in what you can display to users. That is not something Netgate created. I think all it can understand is "I can't get the info". I don't think it can then turn around and ask for a "why" and display that.

TheGushi · Feb 25, 2023, 12:37 AM

@bmeeks yeah, “service unavailable” sounds like a straight mapping of an http response code.

You’re welcome to put in a FreeBSD bug, but it might take a while before it makes it into pfsense, since most changes are applied to -CURRENT.

Dobby_ · Feb 25, 2023, 7:08 PM

PC Engines APU4D4
pfSense+ 23.01 Release
Option (13) Update from console
I got sometimes this answer;

ERROR: It was not possible to determine pkg remote version

If I have rebooted the pfSense I got even back this answer;

>>> Updating repositories metadata...
Updating pfSense-core repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: . done
Processing entries: .. done
pfSense-core repository update completed. 15 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
pfSense repository update completed. 537 packages processed.
All repositories are up to date.
Your packages are up to date
Netgate pfSense Plus - Netgate Device ID:xxxxxxxxxxx

But if I use the option (8) entering the Shell and I set up a

pfSense-upgrade

I got two things back, one is looking like that one from entrance post of that threat here but only for the arch amd64 and the other one gives me back nearly the same as before but without the Netgate pfSense Plus -Netgate Device ID: line at the end


[23.01-RELEASE][root@xxxx.xxxx]/root: pfSense-upgrade
>>> Updating repositories metadata...
Updating pfSense-core repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: . done
Processing entries: .. done
pfSense-core repository update completed. 15 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: .......... done
pfSense repository update completed. 537 packages processed.
All repositories are up to date.
Your packages are up to date

So I personally think this problem is not hitting alone the 1100 and 2100 series repositorries alone.

rosadog · Feb 25, 2023, 8:08 PM

Wondering if anyone is getting this message.

Messages "We could not connect to Netgate servers. Please try again later."

I'm not able to register Pfsense Plus

TheGushi · Feb 25, 2023, 9:35 PM

@rosadog You should probably post your own thread on this, as it's related to something completely different. (Also, it's not relevant here, as it's not about installation or upgrades).

jonathan.johnson · Feb 25, 2023, 9:55 PM

Found the following forum first when attempting to resolve this issue.
https://forum.netgate.com/topic/174768/unable-to-check-for-updates-solved/9

*Tried recommendation by @bingo600

Attempted to update pfSense.conf with "url: pkg+https://firmware.netgate.com..." instead of current "url: pkg+https://repo.netgate.com..."

Then ran pkg-static -d update.

This did not fix the issue although it seemed it would.

When running pkg-static -d update it fetches from https://repo01.atx.netgate.com.

When looking up that address or https://repo.netgate.com in a browser it doesn't resolve. When browsing firmware location https://firmware.netgate.com you do get to a site in your browser. Not sure if this is all expected/normal behavior, but thought I'd mention.

Dobby_ · Feb 25, 2023, 10:00 PM

@jonathan-johnson

pkg-static -d update

It was working well for me, thanks.

@TheGushi

Have a look on my output now, perhaps it will work for you too if the repo is up again;

[23.01-RELEASE][root@xxxx.xxxx]/root: pkg-static -d update
DBG(1)[19878]> pkg initialized
Updating pfSense-core repository catalogue...
DBG(1)[19878]> PkgRepo: verifying update for pfSense-core
DBG(1)[19878]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense-core.sqlite'
DBG(1)[19878]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_01_amd64-core/meta.conf
DBG(1)[19878]> opening libfetch fetcher
DBG(1)[19878]> Fetch > libfetch: connecting
DBG(1)[19878]> Fetch: fetching from: https://pfsense-plus-pkg00.atx.netgate.com/pfSense_plus-v23_01_amd64-core/meta.conf with opts "i"
DBG(1)[19878]> Fetch: fetcher chosen: https
DBG(1)[19878]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_01_amd64-core/packagesite.pkg
DBG(1)[19878]> opening libfetch fetcher
DBG(1)[19878]> Fetch > libfetch: connecting
DBG(1)[19878]> Fetch: fetching from: https://pfsense-plus-pkg00.atx.netgate.com/pfSense_plus-v23_01_amd64-core/packagesite.pkg with opts "i"
DBG(1)[19878]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_01_amd64-core/packagesite.txz
DBG(1)[19878]> opening libfetch fetcher
DBG(1)[19878]> Fetch > libfetch: connecting
DBG(1)[19878]> Fetch: fetching from: https://pfsense-plus-pkg00.atx.netgate.com/pfSense_plus-v23_01_amd64-core/packagesite.txz with opts "i"
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
DBG(1)[19878]> PkgRepo: verifying update for pfSense
DBG(1)[19878]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense.sqlite'
DBG(1)[19878]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_01_amd64-pfSense_plus_v23_01/meta.conf
DBG(1)[19878]> opening libfetch fetcher
DBG(1)[19878]> Fetch > libfetch: connecting
DBG(1)[19878]> Fetch: fetching from: https://pfsense-plus-pkg01.atx.netgate.com/pfSense_plus-v23_01_amd64-pfSense_plus_v23_01/meta.conf with opts "i"
DBG(1)[19878]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_01_amd64-pfSense_plus_v23_01/packagesite.pkg
DBG(1)[19878]> opening libfetch fetcher
DBG(1)[19878]> Fetch > libfetch: connecting
DBG(1)[19878]> Fetch: fetching from: https://pfsense-plus-pkg01.atx.netgate.com/pfSense_plus-v23_01_amd64-pfSense_plus_v23_01/packagesite.pkg with opts "i"
DBG(1)[19878]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_01_amd64-pfSense_plus_v23_01/packagesite.txz
DBG(1)[19878]> opening libfetch fetcher
DBG(1)[19878]> Fetch > libfetch: connecting
DBG(1)[19878]> Fetch: fetching from: https://pfsense-plus-pkg01.atx.netgate.com/pfSense_plus-v23_01_amd64-pfSense_plus_v23_01/packagesite.txz with opts "i"
pfSense repository is up to date.
All repositories are up to date.

TheGushi · Feb 25, 2023, 10:09 PM

@dobby_

The configured update was disabled by Netgate to stop people from bricking their devices due to a problem with the update. This was covered by a netgate staffer earlier.

Presumably, once they've resolved that, the main site will no longer report unavailable.

If you want to try and outsmart them by running shell commands, which may find a method that they haven't updated, because their devices don't try it by default, please do so at your own risk.

jonathan.johnson · Feb 25, 2023, 10:18 PM

@dobby_

Thanks for the information.

I updated my pfSense.conf again to account for your url of "...https://pfsense-plus-pkg.netgate.com..."

This resulted in the following:

DBG(1)[28253]> pkg initialized
Updating pfSense-core repository catalogue...
DBG(1)[28253]> PkgRepo: verifying update for pfSense-core
DBG(1)[28253]> PkgRepo: need forced update of pfSense-core
DBG(1)[28253]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense-core.sqlite'
DBG(1)[28253]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pkg/pfSense_plus-v23_01_aarch64-core/meta.conf
DBG(1)[28253]> opening libfetch fetcher
DBG(1)[28253]> Fetch > libfetch: connecting
DBG(1)[28253]> Fetch: fetching from: https://pfsense-plus-pkg00.atx.netgate.com/pkg/pfSense_plus-v23_01_aarch64-core/meta.conf with opts "i"
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/OU=pfSense Plus/CN=pfsense-plus-pkg00.atx.netgate.com
1086423040:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBSD-src-plus-RELENG_23_01/crypto/openssl/ssl/statem/statem_clnt.c:1921:
DBG(1)[28253]> Fetch: fetching from: https://pfsense-plus-pkg00.atx.netgate.com/pkg/pfSense_plus-v23_01_aarch64-core/meta.conf with opts "i"
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/OU=pfSense Plus/CN=pfsense-plus-pkg00.atx.netgate.com
1086423040:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBSD-src-plus-RELENG_23_01/crypto/openssl/ssl/statem/statem_clnt.c:1921:
DBG(1)[28253]> Fetch: fetching from: https://pfsense-plus-pkg00.atx.netgate.com/pkg/pfSense_plus-v23_01_aarch64-core/meta.conf with opts "i"
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/OU=pfSense Plus/CN=pfsense-plus-pkg00.atx.netgate.com
1086423040:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBSD-src-plus-RELENG_23_01/crypto/openssl/ssl/statem/statem_clnt.c:1921:
DBG(1)[28253]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pkg/pfSense_plus-v23_01_aarch64-core/meta.txz
...
Unable to update repository pfSense
Error updating repositories!

The certificate error may be the issue?

I think the main difference being you're on amd64 and I'm on arm64 for a Netgate2100 which as above commenters have been stating that it's currently problematic for this branch.

jonathan.johnson
Netgate 2100
pfSense+ 22.05 (arm64)

Dobby_ · Feb 26, 2023, 8:08 PM

@jonathan-johnson

I only want to say to @TheGushi that not only the arm repo is not available, nothing more.

rcoleman-netgate · Mar 6, 2023, 2:38 PM

@jonathan-johnson said in Unable to check for updates:

I think the main difference being you're on amd64 and I'm on arm64 for a Netgate2100 which as above commenters have been stating that it's currently problematic for this branch.

Not problematic. The upgrade path is turned off while our engineers work on a solution to the storage space issue faced on some devices.

To upgrade to 23.01 right now on the 1100 and 2100 you have to request the image from TAC, back up your config, write the new image to your device, restore your backup config and reboot.

If you wish to maintain ZFS snapshots and roll-back capability you will have to wait for the online upgrade to be released.

tshaw256 · Feb 26, 2023, 8:06 PM

Sorry if I missed it but has there been any resolution found?

There's no DNS issues that i've found, but I can say that https://repo.netgate.com/ could not resolve and https://firmware.netgate.com/ was able to populate.

Is the repo just down and my timing on the initial setup was just bad?

rcoleman-netgate · Feb 26, 2023, 8:07 PM

@tshaw256 said in Unable to check for updates:

Sorry if I missed it but has there been any resolution found?

At the moment there is no resolution. Please follow https://forum.netgate.com/topic/178049/pfsense-plus-23-01-updates-on-the-1100-and-2100-systems for any updates. Use the Bell icon on the top of any page to follow a topic.

tshaw256 · Feb 26, 2023, 8:08 PM

@dobby_ OHHH so it's down at the moment? I have a 2100. If so, any timeframe on this by chance?

thanks!

rcoleman-netgate · Mar 5, 2023, 3:53 PM

@tshaw256 said in Unable to check for updates:

If so, any timeframe on this by chance?

When it's done it will be done.

anak1n · Mar 5, 2023, 3:53 PM

@rcoleman-netgate "When it's done it will be done." ? I see this comment it's 7 days ago, at least netgate could send a banner about this issue. It cannot be that we're stuck in this, is there any update on this matter?

SteveITS · Mar 5, 2023, 4:02 PM

@anak1n https://forum.netgate.com/topic/178080/unable-to-check-for-updates/21
You can install from USB if you don’t want to wait. They need to make sure it will properly abort on affected devices.

tshaw256 · Mar 5, 2023, 4:23 PM

@anak1n Yeah it threw me off too, BUT, the v22 for the 2100 is working. I can vouch for that at least. v23 is not yet.

SteveITS · Mar 5, 2023, 4:35 PM

@tshaw256 That’s the other thing…unless someone needs a new feature 22.05 will continue to work and has the security patches backported. If they do need something new, or a package installed, the USB install works fine.

If Netgate had caught this before release* and just said 23.01 would be ready mid March then, no issues. :)

*at the time (at release) Netgate staff posted they couldn’t duplicate the problem which makes it hard to catch in testing. Obviously it affects a lot in the wild…my 2100 among them. And several of our clients judging by the partition sizes.

TheGushi · Mar 5, 2023, 5:45 PM

Having read more about this, I'm sort of fascinated by it (because nerd).

Netgate has a whole product team managing this that I'm sure will figure it out, but right now, there's a limitation in how pkg itself works that there's no way to just send a message, or say "there are updates, but you can't install them because of reasons".

Packages can't do things like require a set amount of disk space, or a set partition layout, or a set type of boot environment (uefi or bios). There's no mechanism to check these when pulling packages down.

I've thought of hacky ways around this, like installing a pkg that in turn ran a script that installed a "pseudo package" that met requirements if your root partition were big enough.

In production at work where I have distant machines with no remote hands, I've totally solved this issue by booting into a ramdisk with MFSbsd, ssh'ing back in, paving over the existing layout, then re-downloading the OS. Yes, this could be automated (but I didn't). Yes, this will brick your device if it goes wrong, but so will interrupting any update, really.

I've referred to this process (with a nod to ancient egypt) as "getting the brain out through the nose, then getting a new one back in the same way."

But again, making a package that does all that is way harder. Packages really can only have preinstall and postinstall scripts, and once the preinstall script is running, it can't really "nope out" and refuse to install.

That would be a very sidesteppy process that would have to be built into pfsense from an earlier version, and qa'd and tested. If we're at the point where we're telling people to restore from backup, telling them to break out their serial cables and run a script is not much better.

I am surprised that I can't check for updates in my existing release train, but that is perhaps just an overabundance of caution.