26.07: all pkg fetches fail cert verification, but certs validate manually against the shipped netgate-ca.pem

Zermus

Edited to correct an error in my original post. I said the Netgate CA was absent from the system. That was wrong. It is at /etc/ssl/netgate-ca.pem. My find missed it because I grepped for the literal string "Netgate CA" and PEM bodies are base64, so the subject is not in the file as plaintext. Corrected report below.

Two firewalls on 26.07-RELEASE, both third-party hardware, ZFS. Package management is dead on both. Every pkg fetch fails certificate verification against the Netgate package servers, on both repos, on both CDN nodes.

Certificate verification failed for .../CN=pfsense-plus-pkg00.atx.netgate.com
error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed
pkg: .../packagesite.tzst: Authentication error
Unable to update repository pfSense
Error updating repositories!

But the certs validate fine by hand against the CA bundle that ships with the firmware:

openssl s_client -connect pfsense-plus-pkg00.atx.netgate.com:443 \
  -CAfile /etc/ssl/netgate-ca.pem < /dev/null

Returns Verify return code: 0 for both pkg00 and pkg01. /etc/ssl/netgate-ca.pem is 9160 bytes, valid 2021 to 3021. Without -CAfile the same connection returns code 21, unable to verify the first certificate, which is expected since Netgate CA is a private root and the chain arrives incomplete.

So the certs are good, the CA is present, and the clock is right, yet pkg fails against those same endpoints.

Two more observations, offered without a theory attached:

The node leaf certs have no subjectAltName extension at all. openssl x509 -noout -ext subjectAltName reports no extensions in the certificate. Running s_client with -verify_hostname pfsense-plus-pkg.netgate.com gives code 62, hostname mismatch, while -verify_hostname pfsense-plus-pkg01.atx.netgate.com gives code 0. The repo config uses mirror_type: "srv" against the alias pfsense-plus-pkg.netgate.com. I do not know whether libfetch validates against the alias or the SRV-resolved node name, so I am not claiming this is the failing path, only that it is a difference I can measure.

Setting SSL_CA_CERT_FILE=/etc/ssl/netgate-ca.pem before running pkg changed nothing. I also cannot confirm libfetch reads that variable.

Some possibly-unrelated things from the same boxes:

pfSense-repoc -N regenerates the repo files fine and pfSense-repoc -nZD completes with no error and builds a well formed request, so this does not look like a registration problem.

pfSense-repoc -N produces different output per device. One firewall gets a 579 byte pfSense-repo-0000-cert.pem with no key and a .repo-version file. The other gets a 2094 byte cert plus a 3243 byte key and no .repo-version. Neither works.

/usr/local/etc/pkg/repos/pfSense.conf is a symlink into /usr/local/etc/pfSense/pkg/repos/. When that directory gets cleared the symlink dangles and pkg reports "No such file or directory", which reads like a deleted file but isn't. Possibly related to https://forum.netgate.com/topic/201140/26.07-upgrade-fails-pfsense.conf-file-missing though nobody there has reported a cert verification failure.

The GUI package list can show stale cached data. One firewall looked healthy with Available Packages populated normally, but fails identically from a shell. Worth checking from the CLI before writing off a host as unaffected.

The question: what is pkg's expected certificate verification path on 26.07? If it is supposed to use /etc/ssl/netgate-ca.pem, something is not reaching it. Happy to run whatever diagnostic would help.

stephenw10

@Zermus what output do you see from pkg -d update or pfSense-repoc -nZD?

Zermus

@stephenw10 Both below. This is one, but both behave identically.

pkg -d update, trimmed to one representative fetch (every fetch to both nodes repeats this pattern, then both repos fail):

DBG(1)> (fetch) Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v26_07_amd64-pfSense_plus_v26_07/packagesite.tzst
DBG(1)> (fetch) libfetch> connecting
DBG(1)> (fetch) libfetch> fetching from: https://pfsense-plus-pkg00.atx.netgate.com/pfSense_plus-v26_07_amd64-pfSense_plus_v26_07/packagesite.tzst with opts "i"
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/OU=pfSense Plus/CN=pfsense-plus-pkg00.atx.netgate.com
10B0A6B2F4290000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-Plus-snapshots-26_07-main/sources/FreeBSD-src-plus-RELENG_26_07/crypto/openssl/ssl/statem/statem_clnt.c:2125:
pkg: https://pfsense-plus-pkg00.atx.netgate.com/pfSense_plus-v26_07_amd64-pfSense_plus_v26_07/packagesite.tzst: Authentication error
Unable to update repository pfSense
Error updating repositories!

It also tries meta.conf, meta.txz, data.pkg, data.tzst and packagesite.pkg first, three attempts each, all failing the same way. repository pfSense has no meta file, using default settings appears after the meta.txz failures. Both pfSense-core and pfSense end with Unable to update repository.

pfSense-repoc -nZD, with identifiers redacted:

OS: FreeBSD
OS Version: 16.0-CURRENT
Platform: amd64
Product: Netgate pfSense Plus
Version: 26.07-RELEASE
FS type: zfs
Language: en_US
Model: unknown hardware
NDI: <redacted>
License Phrase: <redacted>
Package prefix: pfSense-pkg-
Serial: <redacted>
Repo path: /usr/local/etc/pfSense
Request query: {"platform":"unknown hardware","os":"FreeBSD","osver":"16.0-CURRENT","prod":"Netgate pfSense Plus","ver":"26.07-RELEASE","ed":"Plus","fstype":"zfs","pkgs":"[{\"name\":\"Cron\",\"ver\":\"0.3.8_8\"},{\"name\":\"acme\",\"ver\":\"1.3.2\"},{\"name\":\"Nexus\",\"ver\":\"26.07\"},{\"name\":\"snort\",\"ver\":\"4.1.10\"},{\"name\":\"haproxy\",\"ver\":\"0.65.7\"},{\"name\":\"Status_Traffic_Totals\",\"ver\":\"2.3.5_3\"}]"}

No error from repoc, and it writes the repo files successfully. Happy to send the unredacted version or the full untrimmed pkg log privately if that helps.

Zermus

Following up with a correction to my own post above. I said the repo endpoint looked like it was assigned per device. That was wrong. It varies per call.

The firewall that had update.netgate.com moved back to pfsense-plus-pkg.netgate.com after another pfSense-repoc -N, with nothing changed on my end. Both of my firewalls are now landing on the old endpoint and failing identically. I've run repoc -N several times since and it has come back pfsense-plus-pkg.netgate.com every time, so whatever produced the update.netgate.com config appears to have been transient.

For anyone else comparing notes, the two endpoints fail differently:

update.netgate.com / update01.atx.netgate.com gives clean TLS and then HTTP 400 on every file:

pkg: https://update01.atx.netgate.com/pfSense_plus-v26_07_amd64-core/meta.txz: Bad Request
pkg: https://update01.atx.netgate.com/pfSense_plus-v26_07_amd64-core/packagesite.tzst: Bad Request
Unable to update repository pfSense-core

pfsense-plus-pkg.netgate.com / pfsense-plus-pkg00 and 01 fails certificate verification:

Certificate verification failed for .../CN=pfsense-plus-pkg00.atx.netgate.com
error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed
pkg: .../packagesite.tzst: Authentication error

Clearing the catalog cache with rm -rf /var/db/pkg/repos/* and re-running pkg update makes no difference, which rules out stale local data.

If this is an in-progress migration to update.netgate.com I'm happy to sit it out, and I'd rather not keep poking at it if the answer is just to wait. It would help to know whether that's the case. If it isn't a migration, then something is handing out two different repo configurations for the same device and neither of them works.

Also flagging in case it saves someone time: the Package Manager GUI keeps showing Available Packages fully populated on both of these firewalls the entire time. It renders from cached catalog data and does not reflect repo health at all. Check from a shell with pkg update before assuming a host is fine.

stephenw10

An instance already on 26.0 should be using update.netgate.com. Running pfSense-repoc -N will replace the repos with the old endpoint which is incorrect. If you need to update the repos from the CLI in 26.07 you should run pfSense-repoc -NZ. Like:

[26.07-RELEASE][admin@6100.stevew.lan]/root: pfSense-repoc -NZ
[26.07-RELEASE][admin@6100.stevew.lan]/root: pkg -d update
Updating pfSense-core repository catalogue...
DBG(1)[2046]> PkgRepo: verifying update for pfSense-core
DBG(1)[2046]> Pkgrepo, begin update of '/var/db/pkg/repos/pfSense-core/db'
DBG(1)[2046]> (fetch) Request to fetch pkg+https://update.netgate.com/pfSense_plus-v26_07_amd64-core/meta.conf
DBG(1)[2046]> (fetch) libfetch> connecting
DBG(1)[2046]> (fetch) libfetch> fetching from: https://update00.atx.netgate.com/pfSense_plus-v26_07_amd64-core/meta.conf with opts "i"
DBG(1)[2046]> (fetch) Request to fetch pkg+https://update.netgate.com/pfSense_plus-v26_07_amd64-core/data.pkg
DBG(1)[2046]> (fetch) libfetch> connecting
DBG(1)[2046]> (fetch) libfetch> fetching from: https://update00.atx.netgate.com/pfSense_plus-v26_07_amd64-core/data.pkg with opts "i"
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
DBG(1)[2046]> PkgRepo: verifying update for pfSense
DBG(1)[2046]> Pkgrepo, begin update of '/var/db/pkg/repos/pfSense/db'
DBG(1)[2046]> (fetch) Request to fetch pkg+https://update.netgate.com/pfSense_plus-v26_07_amd64-pfSense_plus_v26_07/meta.conf
DBG(1)[2046]> (fetch) libfetch> connecting
DBG(1)[2046]> (fetch) libfetch> fetching from: https://update01.atx.netgate.com/pfSense_plus-v26_07_amd64-pfSense_plus_v26_07/meta.conf with opts "i"
DBG(1)[2046]> (fetch) Request to fetch pkg+https://update.netgate.com/pfSense_plus-v26_07_amd64-pfSense_plus_v26_07/data.pkg
DBG(1)[2046]> (fetch) libfetch> connecting
DBG(1)[2046]> (fetch) libfetch> fetching from: https://update01.atx.netgate.com/pfSense_plus-v26_07_amd64-pfSense_plus_v26_07/data.pkg with opts "i"
pfSense repository is up to date.
All repositories are up to date.

But since you're seeing 400 auth errors there's likely some license issue. Send me your NDI in chat and I'll check it.

Zermus

@stephenw10 That was it, thanks. pfSense-repoc -NZ fixed all three of my firewalls immediately. Both repos update cleanly against update.netgate.com, no 400s, no cert errors.

So there was no license issue luckily. That also explains the two things I reported as strange earlier in the thread: the endpoint appearing to differ between devices and then appearing to change per call was just which flag I had run last on which box, and the differing cert and key file sets came from the same thing. Sorry for the noise.

Two things worth flagging for anyone who lands here from a search.

The -N versus -NZ distinction is not obvious. -N looks like the natural "reinstall the repo config" flag and it is what turns up if you go looking for a way to regenerate pfSense.conf, but on 26.07 it writes the old endpoint and breaks a working install.

The Package Manager GUI showed Available Packages randomly. It renders from cached catalog data and does not reflect repo health at all. I spent a while treating one firewall as a known-good control because the GUI looked fine while pkg update was failing on it from a shell. Check from the CLI.

stephenw10

Ah good news.

Normally you shouldn't ever have to run either of those commands at the CLI. Logging into the webgui calls that as part of the update check.

The Z switch there calls the new backend that uses the license phrase as the primary identity.

luckman212

@stephenw10 I've seen the command written with a lowercase n as in the pfSense-repoc -nZD example given here.

But the -h helptext says -n is a dry-run so that seems wrong for a "repair". Is there a difference between that and pfSense-repoc -NZD ? Any use case for the lowercase variant?

stephenw10

Good question. Upper case -N was added to request a new format for the repo files. -Z makes it request against the new backend. The new backend only uses the new format so it I could imagine using -Z there overrides -N. But I always use -N.