Repo metadata updated fails from offline mirror

[globejack]

Hi,

I am facing an issue with loading repo metadata in an "offline" network setup. We have several pfSense firewalls in a dedicated MPLS network not connected to the outside world. We host an offline repo mirror in that network to be able to update the firewalls and it used to work perfectly fine (e.g. 2.4.4 -> 2.4.5). The mirror is setup using an IIS web server (sorry but that's what we have in that network anyway).

Updating one of the firewalls from 2.4.5_1 to 2.5.1 worked just fine as well. Though I figured the newly updated firewall has issues to pull from the repo now after the update. I need to figure out what's wrong to be able to update to newer versions in the future.

Now here are the details. After the update to 2.5.1 trying to check for new update the command just hangs:

[2.5.1-RELEASE][admin@...]/root: pfSense-upgrade -d -c
>>> Updating repositories metadata...
Updating pfSense-core repository catalogue...
Fetching meta.conf: . done

Trying to update the repo metadata with debug output I get this:

[2.5.1-RELEASE][admin@...]/root: pkg-static -d update -f
DBG(1)[71405]> pkg initialized
Updating pfSense-core repository catalogue...
DBG(1)[71405]> PkgRepo: verifying update for pfSense-core
DBG(1)[71405]> PkgRepo: need forced update of pfSense-core
DBG(1)[71405]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense-core.sqlite'
DBG(1)[71405]> Request to fetch pkg+http://10.x.x.x/pfsense/pfSense_v2_5_1_amd64-core/meta.conf
DBG(1)[71405]> opening libfetch fetcher
DBG(1)[71405]> Fetch > libfetch: connecting
DBG(1)[71405]> Fetch: fetching from: http://10.x.x.x/pfsense/pfSense_v2_5_1_amd64-core/meta.conf with opts "i"
DBG(1)[71405]> Fetch: fetcher chosen: http
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
DBG(1)[71405]> Request to fetch pkg+http://10.x.x.x/pfsense/pfSense_v2_5_1_amd64-core/packagesite.txz
DBG(1)[71405]> opening libfetch fetcher
DBG(1)[71405]> Fetch > libfetch: connecting
DBG(1)[71405]> Fetch: fetching from: http://10.x.x.x/pfsense/pfSense_v2_5_1_amd64-core/packagesite.txz with opts "i"

So it pulls the file meta.conf just fine but then sits there for ages when loading packagesite.txz. Looking at the packets in Wireshark I see both the second HTTP GET as well as HTTP OK answer from the webserver. After that pfSense sends a TCP Dup ACK followed by TCP Retransmissions from the webserver side.

To make sure it's not the IIS being at fault here I pulled up a simple small webserver service but ran into the exact same thing. As well I tried the same repo metadata update from one of our other firewalls that are still on 2.4.5:

[2.4.5-RELEASE][admin@...]/root: pkg-static -d update -f
DBG(1)[25439]> pkg initialized
pkg-static: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
Updating pfSense-core repository catalogue...
DBG(1)[25439]> PkgRepo: verifying update for pfSense-core
DBG(1)[25439]> PkgRepo: need forced update of pfSense-core
DBG(1)[25439]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense-core.sqlite'
DBG(1)[25439]> Fetch: fetching from: http://10.x.x.x/pfsense/pfSense_v2_5_1_amd64-core/meta.conf with opts "i"
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
DBG(1)[25439]> Fetch: fetching from: http://10.x.x.x/pfsense/pfSense_v2_5_1_amd64-core/packagesite.txz with opts "i"
Fetching packagesite.txz: 100%    2 KiB   1.8kB/s    00:01
DBG(1)[25439]> PkgRepo: extracting packagesite.yaml of repo pfSense-core
DBG(1)[25688]> PkgRepo: extracting signature of repo in a sandbox
DBG(1)[25439]> Pkgrepo, reading new packagesite.yaml for '/var/db/pkg/repo-pfSense-core.sqlite'
Processing entries: 100%
pfSense-core repository update completed. 7 packages processed.
Updating pfSense repository catalogue...
DBG(1)[25439]> PkgRepo: verifying update for pfSense
DBG(1)[25439]> PkgRepo: need forced update of pfSense
DBG(1)[25439]> Pkgrepo, begin update of '/var/db/pkg/repo-pfSense.sqlite'
DBG(1)[25439]> Fetch: fetching from: http://10.x.x.x/pfsense/pfSense_v2_5_1_amd64-pfSense_v2_5_1/meta.conf with opts "i"
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
DBG(1)[25439]> Fetch: fetching from: http://10.x.x.x/pfsense/pfSense_v2_5_1_amd64-pfSense_v2_5_1/packagesite.txz with opts "i"
Fetching packagesite.txz: 100%  141 KiB 144.0kB/s    00:01
DBG(1)[25439]> PkgRepo: extracting packagesite.yaml of repo pfSense
DBG(1)[26092]> PkgRepo: extracting signature of repo in a sandbox
DBG(1)[25439]> Pkgrepo, reading new packagesite.yaml for '/var/db/pkg/repo-pfSense.sqlite'
...

When updateing from 2.4.5_1 to 2.5.1 pkg was update from version 1.15.6 to 1.16.1. Anyone aware of changes between those versions that could explain the behaviour described above?

Any comment on how to debug this any further or hints on what could be wrong are highly apreciated. Thanks in advance!

SR

[globejack]

This issue is solved. Can't believe it but a reboot fixed it.

For several days we saw it behave exactly as described right after the update to 2.5.1 (including the subsequent reboot). Now after a manual reboot of the system we are back to normal. Repo information can be pulled and it's all happy now.

Great on the one hand but pretty awkward on the other I find. Would not expect that from a FreeBSD based system.

[globejack]

Unfortunately I am back on this issue when updating another firewall in that offline network.

Rebooting the firewall after the update did not help to fix the issue this time. Trying to debug this further I tried to call fetch manually:

[2.5.1-RELEASE][admin@...]/root: fetch -v http://10.x.x.x/pfsense/pfSense_v2_5_1_amd64-core/packagesite.txz
resolving server address: 10.x.x.x:80
requesting http://10.x.x.x/pfsense/pfSense_v2_5_1_amd64-core/packagesite.txz

It just hangs, same as when I call pkg-static update manually.

But if I do the same fetch on meta.conf it loads perfectly fine:

[2.5.1-RELEASE][admin@...]/root: fetch -v http://10.x.x.x/pfsense/pfSense_v2_5_1_amd64-core/meta.conf
resolving server address: 10.x.x.x:80
requesting http://10.x.x.x/pfsense/pfSense_v2_5_1_amd64-core/meta.conf
remote size / mtime: 163 / 1618334907
meta.conf                                              163  B  449 kBps    00s