Firmware/Package update slow, missing mirror?

JeGr

Hi,

we were just trying to update several customer installations and were surprised, that trying to get System/Updates to open took a VERY long time. Dashboard already showed error updating version informations, after almost 2min Update page loaded and listed another error. Switching to 25.07.1 and saving again too a very long time to complete.

Switching to console and trying to start the update with option 13 again took almost 2min before anything happened output-wise (from pkg). So it took almost 2min to show the ">>> updating repositories metadata" message normally popping up in seconds.

The only thing we observed were many log entries in DNS resolver about
failed to resolve host frafiles.netgate.com
so I'm guessing we have some DNS shenannigans with a mirror down? At least we get no response for frafiles anymore and it seems the devices here are trying that mirror HARD to download the update infos and not getting anywhere. atxfiles does respond with a legacy IPv4 (why no IPv6 though?!) but somehow it seems the boxes are trying to reach stuff that is either currently down or was being deprecated (no mirror in Europe anymore??).

Perhaps someone has more insights in that?

Cheers
\jens

Gertjan

@JeGr said in Firmware/Package update slow, missing mirror?:

... in Europe ....

I tried the same thing right now :
SSH, and option 13 :

It took a bit more then 60 seconds before the result of the command was shown :

Messages:
Your Netgate device has pfSense+ as part of your device purchase.
>>> Updating repositories metadata...
Updating pfSense-core repository catalogue...
Fetching meta.conf: . done
Fetching data.pkg: . done
Processing entries: . done
pfSense-core repository update completed. 5 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: . done
Fetching data.pkg: ......... done
Processing entries: .......... done
pfSense repository update completed. 734 packages processed.
All repositories are up to date.
>>> Setting vital flag on pfSense...done.
Your packages are up to date
Netgate 4100 - Serial: 2014221462 - Netgate Device ID: e57xxdc47dcdd5aef27a

edit : from France = Europe also.

JeGr

@Gertjan said in Firmware/Package update slow, missing mirror?:

It took a bit more then 60 seconds before the result of the command was shown :

Tried on another box of another customer - same thing. More then 60s, was more like 90 until it finally showed the first line.

Took a shot at a box in the US, there it worked almost instantly.

Somehow smells like DNS'y propagation?

Gertjan

@JeGr said in Firmware/Package update slow, missing mirror?:

Somehow smells like DNS'y propagation?

Or some one plowed through the fiber somewhere in the Atlantic ?

Edit : me stupid. That would impact my connection to this forum as well, and that doesn't seem to be the case. I down-vote myself ...

stephenw10

Hmm, this is coming from 25.07?

stephenw10

We are digging....

stephenw10

Can you try running pkg -d update to get the full debug output calling that mirror?

jimp

Might also help to know your DNS server configuration on the firewall, is it in resolver mode, forwarding somewhere specific (and where), that sort of stuff.

And what is shown in pkg -vv | grep ' url' (note the spacing there)

Also does it work if you check the option to prefer IPv4?

JeGr

@stephenw10 said in Firmware/Package update slow, missing mirror?:

Can you try running pkg -d update to get the full debug output calling that mirror?

Did one better.

pkg(-static) -d update runs through in seconds:

Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
[25.07.1-RELEASE][root@fwl01.office.nroute.de]/root:
[25.07.1-RELEASE][root@fwl01.office.nroute.de]/root: pkg-static -d update
DBG(1)[33878]> pkg initialized
Updating pfSense-core repository catalogue...
DBG(1)[33878]> PkgRepo: verifying update for pfSense-core
DBG(1)[33878]> Pkgrepo, begin update of '/var/db/pkg/repos/pfSense-core/db'
DBG(1)[33878]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_07_1_amd64-core/meta.conf
DBG(1)[33878]> curl_open
DBG(1)[33878]> Fetch: fetcher used: pkg+https
DBG(1)[33878]> curl> fetching https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_07_1_amd64-core/meta.conf

DBG(1)[33878]> CURL> attempting to fetch from , left retry 3

* Couldn't find host pfsense-plus-pkg00.atx.netgate.com in the .netrc file; using defaults
* Host pfsense-plus-pkg00.atx.netgate.com:443 was resolved.
* IPv6: 2610:160:11:18::207
* IPv4: 208.123.73.207
*   Trying [2610:160:11:18::207]:443...
*   Trying 208.123.73.207:443...
* Connected to pfsense-plus-pkg00.atx.netgate.com (208.123.73.207) port 443
* ALPN: curl offers http/1.1
*  CAfile: /etc/ssl/netgate-ca.pem
*  CApath: /etc/ssl/certs/
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: C=US; ST=Texas; L=Austin; O=Rubicon Communications, LLC (Netgate); OU=pfSense Plus; CN=pfsense-plus-pkg00.atx.netgate.com
*  start date: Mar 15 20:23:11 2022 GMT
*  expire date: Feb 19 20:23:11 2122 GMT
*  common name: pfsense-plus-pkg00.atx.netgate.com (matched)
*  issuer: C=US; ST=Texas; L=Austin; O=Rubicon Communications, LLC (Netgate); OU=Netgate CA; CN=Netgate CA
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/1.x
> GET /pfSense_plus-v25_07_1_amd64-core/meta.conf HTTP/1.1
Host: pfsense-plus-pkg00.atx.netgate.com
User-Agent: pkg/1.21.3
Accept: */*
If-Modified-Since: Fri, 15 Aug 2025 21:11:56 GMT

* Request completely sent off
< HTTP/1.1 304 Not Modified
< Server: nginx
< Date: Wed, 10 Sep 2025 14:14:19 GMT
< Last-Modified: Fri, 15 Aug 2025 21:11:56 GMT
< Connection: keep-alive
< ETag: "689fa29c-b3"
<
* Connection #0 to host pfsense-plus-pkg00.atx.netgate.com left intact
DBG(1)[33878]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_07_1_amd64-core/data.pkg
DBG(1)[33878]> curl_open
DBG(1)[33878]> Fetch: fetcher used: pkg+https
DBG(1)[33878]> curl> fetching https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_07_1_amd64-core/data.pkg

DBG(1)[33878]> CURL> attempting to fetch from , left retry 3

* Couldn't find host pfsense-plus-pkg00.atx.netgate.com in the .netrc file; using defaults
* Found bundle for host: 0x3e547f2258a0 [serially]
* Re-using existing connection with host pfsense-plus-pkg00.atx.netgate.com
> GET /pfSense_plus-v25_07_1_amd64-core/data.pkg HTTP/1.1
Host: pfsense-plus-pkg00.atx.netgate.com
User-Agent: pkg/1.21.3
Accept: */*
If-Modified-Since: Fri, 15 Aug 2025 21:11:56 GMT

* Request completely sent off
< HTTP/1.1 304 Not Modified
< Server: nginx
< Date: Wed, 10 Sep 2025 14:14:20 GMT
< Last-Modified: Fri, 15 Aug 2025 21:11:56 GMT
< Connection: keep-alive
< ETag: "689fa29c-6be"
<
* Connection #0 to host pfsense-plus-pkg00.atx.netgate.com left intact
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
DBG(1)[33878]> PkgRepo: verifying update for pfSense
DBG(1)[33878]> Pkgrepo, begin update of '/var/db/pkg/repos/pfSense/db'
DBG(1)[33878]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_07_1_amd64-pfSense_plus_v25_07_1/meta.conf
DBG(1)[33878]> curl_open
DBG(1)[33878]> Fetch: fetcher used: pkg+https
DBG(1)[33878]> curl> fetching https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_07_1_amd64-pfSense_plus_v25_07_1/meta.conf

DBG(1)[33878]> CURL> attempting to fetch from , left retry 3

* Couldn't find host pfsense-plus-pkg01.atx.netgate.com in the .netrc file; using defaults
* Host pfsense-plus-pkg01.atx.netgate.com:443 was resolved.
* IPv6: 2610:160:11:18::209
* IPv4: 208.123.73.209
*   Trying [2610:160:11:18::209]:443...
*   Trying 208.123.73.209:443...
* Connected to pfsense-plus-pkg01.atx.netgate.com (208.123.73.209) port 443
* ALPN: curl offers http/1.1
*  CAfile: /etc/ssl/netgate-ca.pem
*  CApath: /etc/ssl/certs/
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: C=US; ST=Texas; L=Austin; O=Rubicon Communications, LLC (Netgate); OU=pfSense Plus; CN=pfsense-plus-pkg01.atx.netgate.com
*  start date: Mar 15 20:23:37 2022 GMT
*  expire date: Feb 19 20:23:37 2122 GMT
*  common name: pfsense-plus-pkg01.atx.netgate.com (matched)
*  issuer: C=US; ST=Texas; L=Austin; O=Rubicon Communications, LLC (Netgate); OU=Netgate CA; CN=Netgate CA
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/1.x
> GET /pfSense_plus-v25_07_1_amd64-pfSense_plus_v25_07_1/meta.conf HTTP/1.1
Host: pfsense-plus-pkg01.atx.netgate.com
User-Agent: pkg/1.21.3
Accept: */*
If-Modified-Since: Tue, 09 Sep 2025 15:44:52 GMT

* Request completely sent off
< HTTP/1.1 304 Not Modified
< Server: nginx
< Date: Wed, 10 Sep 2025 14:14:20 GMT
< Last-Modified: Tue, 09 Sep 2025 15:44:52 GMT
< Connection: keep-alive
< ETag: "68c04b74-b3"
<
* Connection #0 to host pfsense-plus-pkg01.atx.netgate.com left intact
DBG(1)[33878]> Request to fetch pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_07_1_amd64-pfSense_plus_v25_07_1/data.pkg
DBG(1)[33878]> curl_open
DBG(1)[33878]> Fetch: fetcher used: pkg+https
DBG(1)[33878]> curl> fetching https://pfsense-plus-pkg.netgate.com/pfSense_plus-v25_07_1_amd64-pfSense_plus_v25_07_1/data.pkg

DBG(1)[33878]> CURL> attempting to fetch from , left retry 3

* Couldn't find host pfsense-plus-pkg01.atx.netgate.com in the .netrc file; using defaults
* Found bundle for host: 0x3e547f316da0 [serially]
* Re-using existing connection with host pfsense-plus-pkg01.atx.netgate.com
> GET /pfSense_plus-v25_07_1_amd64-pfSense_plus_v25_07_1/data.pkg HTTP/1.1
Host: pfsense-plus-pkg01.atx.netgate.com
User-Agent: pkg/1.21.3
Accept: */*
If-Modified-Since: Tue, 09 Sep 2025 15:44:52 GMT

* Request completely sent off
< HTTP/1.1 304 Not Modified
< Server: nginx
< Date: Wed, 10 Sep 2025 14:14:20 GMT
< Last-Modified: Tue, 09 Sep 2025 15:44:52 GMT
< Connection: keep-alive
< ETag: "68c04b74-3afde"
<
* Connection #0 to host pfsense-plus-pkg01.atx.netgate.com left intact
pfSense repository is up to date.
All repositories are up to date.

But: what didn't run in mere seconds was pfSense-upgrade command, the one that gets called from the webUI (or anywhere else) to check if updates exist:

[25.07.1-RELEASE][root@]/root: pfSense-upgrade -dC

# ... no output for almost a minute(!)... then very fast:

>>> Updating repositories metadata...
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Your system is up to date

That happened before and after the update to 25.07, so it was there before but only popped up recently. The systems ran for months doing update-checking and never had that problem. That showed up very recently. Also the "debug" parameter of the command seems to do nothing? Weird.

If I run the upgrade command with /bin/sh -x I can see it run up to the lockf command and then stall, then after not 5s but ~60s it continues:

+ dirname /usr/local/sbin/pfSense-upgrade
+ basename /usr/local/sbin/pfSense-upgrade
+ realpath -q /usr/local/sbin/../libexec/pfSense-upgrade
+ pfsense_upgrade=/usr/local/libexec/pfSense-upgrade
+ basename /usr/local/sbin/pfSense-upgrade
+ lockfile=/tmp/pfSense-upgrade.lock
+ lockf_timeout=5
+ [ ! -f /usr/local/libexec/pfSense-upgrade ]
+ id -u
+ [ 0 -ne 0 ]
+ unset boot_stage
+ getopts 46b:cCdfi:hp:l:nr:RT:uUy opt
+ [ -n '' ]
+ [ -z 5 ]
+ [ 5 -lt 2 ]
+ [ 5 -gt 600 ]
+ EX_TEMPFAIL=75
+ EX_CANTCREAT=73
+ EX_OSERR=71
+ EX_SOFTWARE=70
+ EX_UPGRADE=99
+ true
+ unset run_again
+ /usr/bin/lockf -s -t 5 /tmp/pfSense-upgrade.lock /usr/local/libexec/pfSense-upgrade



>>> Updating repositories metadata...
Updating pfSense-core repository catalogue...
Fetching meta.conf: . done
Fetching data.pkg: . done
Processing entries: . done
pfSense-core repository update completed. 5 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: . done
Fetching data.pkg: .......... done
Processing entries: .......... done
pfSense repository update completed. 734 packages processed.
All repositories are up to date.
Your packages are up to date
+ rc=0
+ unset error
+ [ -f /tmp/pfSense-upgrade.lock ]
+ [ -n '' ]
+ [ -n '' ]
+ exit 0

also removing the -s silent switch from lockf did nothing. There's no output at all indicating a problem with the file locking but it seems related to it, that those boxes get errors and timeouts while checking for updates.

As said, pkg itself runs pretty fast so DNS seems out of that picture at least.

SteveITS

@JeGr said in Firmware/Package update slow, missing mirror?:

after almost 2min Update page loaded and listed another error

What was the error? "another instance" is running perhaps?

Just asking because I've seen several posts about that here and elsewhere. In my experience upgrading several to 24.11 earlier this year and recently one on 25.07 I get the "another instance" error relatively quickly but then have to wait a while to get past it. Especially on ARM hardware I need to wait up to 10 minutes, give or take, before reloading the page. Faster CPUs are much shorter time as I recall.

stephenw10

Still digging.

The odd thing is that it was trying to reach frafiles which hasn't existed for a long while.

jimp

Sounds more like it isn't the lockf but it isn't seeing what is being executed inside /usr/local/libexec/pfSense-upgrade

You could also edit /usr/local/libexec/pfSense-upgrade and add -x to the shabang line, or change that lockf line to run it via sh -x /usr/local/libexec/pfSense-upgrade.

Before going that far, you may want to try seeing if running pfSense-repoc-static returns quickly or has a delay. It may be having issues connecting to ews.netgate.com over IPv6 and it would take ~75 seconds or so to fall back to IPv4.

JeGr

@jimp said in Firmware/Package update slow, missing mirror?:

pfSense-repoc-static

That one is an interesting one. On the system that threw the error, that call takes a long time (again around a minute), on another system to test it's done in about 2-3s. Both output nothing, so it runs but eerily slow.

jimp

It's checking for new repository metadata to see if there are new available branches and so on from our dynamic repositories, since that can't be done by pkg itself directly.

We are investigating a general IPv6 connectivity issue now, there is something going on there.

As for frafiles.netgate.com I bet you have a firewall alias with that hostname somewhere, and the failed to resolve host message is from filterdns. We retired that hostname back in July so it no longer exists and isn't referenced by anything in the current DNS zones.

Gertjan

As @JeGr JeGr above :

pkg -d update

was shown at the usual 'lightning' speed : all done in less then 5 seconds.

@SteveITS I did check - like this morning, if another instance was already running :

ps aux | grep 'pkg'

I'm using 25.07.1 and my resolver has 100 % native settings : I'm resolving.
No VPNs
Fully IPv4/Ipv6 - both work fine.

No !

But wait : all my "is IPv6 working sites in USA answer Ok : cnn; netflix, amazon.com, cdc etc etc ... except : forum.netgate.com" : I can't access forum.netgate.com using IPv6 anymore.
Normally I connect to forum.netgate.com with IPv6 and this works fine for many years already.
Not today ...
My pfSense is full stack ... and if my pfSense emits an update requests using IPv6 as it is available, but IPv6 on the 'netgate' (package server) side is down, then the 'wait' time is explained - it has to fall back to IPv4 before it continues.
And this has happened before.

JeGr

Dito the system that is running into waits is fullstack IPv6. The fast box ist theoretically DS but prefers IPv4 due to reasons* so IPv6 issue could be a good point. At least it seems, that running the pfsense-repoc-static manually has wiped the repos now thus pkg update no longer finds repos to update from. So that's crazy borked ;)

The frafiles stuff we've identified in an old alias like you guessed.

jimp

Everything should be back to normal now for IPv6 connectivity, there was an upstream issue that's been resolved.