Stuck upgrade of squid after pfsense upgrade 2.4.5

kevindd992002

I have two boxes that I just recently upgraded to 2.4.5. Just yesterday, there was an available upgrade for the Squid package so I went ahead and upgraded both boxes (they use the same set of packages). One successfully finished but the other was stuck at "Updating pfSense repository catalogue". So then I decided to go out of the update page and hoped to reinstall the package. But when I tried doing that, I cannot access System -> Package Manager anymore. The Installed Packages section just loads indefinitely and the Available Packages section says "Unable to retrieve package information.". So for some reason, it's not able to retrieve package information.

So thinking that I got a corrupted install, I backed up config.xml and reinstalled pfsense from scratch in my PCEngines APU2C4 box. Before even restoring config.xml, I tried going to System -> Package Manager -> Available Packages again and I get the same error. Why so?

I then proceeded in restoring the config.xml and as I expected it triggered the reinstallaton of the packages but after an hour of waiting, I get logs in the System Logs that says that the package is missing the configuration and have to be removed. The available packages section still errors out.

Also, will restoring a backup from the Auto Config Backup (ACB) service reinstall the packages as well? I tried restoring a 2.4.4p3 config (a backup one day before I upgraded to 2.4.5) but it did not even attempt to reinstall the packages. The console logs say it was trying to start the packages and this will naturally fail because they weren't installed in a fresh install of pfsense in the first place. So why would restoring a physical config.xml file reinstall the packages as expected but not when restoring using a backup from ACB? I cannot find any documentation about the differences between these two restore options.

kevindd992002

nslookup for pkg.pfsense.org to remove the possibility of DNS issue:

That pihole is pointed to the affected pfsense box which, in turn, uses DNS Resolver (unbound).

kevindd992002

I just tried again now and for some reason the Available Packages is listing the information I'm expecting. However, when I try to install freeradius, here's what I get:

>>> Installing pfSense-pkg-freeradius3... 
Updating pfSense-core repository catalogue...
pkg-static: https://pkg.pfsense.org/pfSense_v2_4_5_amd64-core/meta.txz: No route to host
repository pfSense-core has no meta file, using default settings
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
The following 9 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	pfSense-pkg-freeradius3: 0.15.7_11 [pfSense]
	bash: 5.0.11 [pfSense]
	freeradius3: 3.0.20 [pfSense]
	python27: 2.7.17_1 [pfSense]
	talloc: 2.3.0 [pfSense]
	postgresql11-client: 11.6 [pfSense]
	mysql57-client: 5.7.29 [pfSense]
	protobuf: 3.9.2,1 [pfSense]
	gdbm: 1.18.1_1 [pfSense]

Number of packages to be installed: 9

The process will require 166 MiB more space.
21 MiB to be downloaded.
[1/9] Fetching pfSense-pkg-freeradius3-0.15.7_11.txz: ........ done
pkg-static: https://pkg.pfsense.org/pfSense_v2_4_5_amd64-pfSense_v2_4_5/All/bash-5.0.11.txz: No route to host
Failed

So there's still something wrong here.

jimp

No route to host

Can you ping file00.netgate.com or files01.netgate.com ?

Are you sure it's using IPv4 and not IPv6? Try both ping and ping6 and see what happens.

kevindd992002

You mean files00.netgate.com, right?

If so, here's the result:

From the working pfsense box, I can ping files00.netgate.com though. That probably explains why the intermittent issue. Is this an ISP route problem?

jimp

It's possible. Try with traceroute (tracert on windows) or something like mtr and see where it stops along the way.

johnpoz

Those are in the same net block 162.208.119.40 and .41, I doubt its a routing issue... But yeah a trace to both would be interesting.

kevindd992002

@jimp said in Stuck upgrade of squid after pfsense upgrade 2.4.5:

tracert

johnpoz

Well from that you can tell for sure that is an upstream where you having the issue, because you make it to your gateway..

jimp

Certainly appears to be upstream from you, though I don't quite see why/how your ISP would be routing that any differently.

johnpoz

Yeah very odd.. since they are both in the same netblock

kevindd992002

Yeah, I agree. I guess I have to call them up again tomorrow to see why this is happening.

Do you guys have any ideas on my ACB restore question?

@kevindd992002 said in Stuck upgrade of squid after pfsense upgrade 2.4.5:

Also, will restoring a backup from the Auto Config Backup (ACB) service reinstall the packages as well? I tried restoring a 2.4.4p3 config (a backup one day before I upgraded to 2.4.5) but it did not even attempt to reinstall the packages. The console logs say it was trying to start the packages and this will naturally fail because they weren't installed in a fresh install of pfsense in the first place. So why would restoring a physical config.xml file reinstall the packages as expected but not when restoring using a backup from ACB? I cannot find any documentation about the differences between these two restore options.

jimp

Not currently. The only method which triggers a package reinstall is using Diag > Backup/Restore.

kevindd992002

Ok, thanks for the confirmation. If there are any corruption in a previous install, would backing up the config.xml of that corrupted install carry over any type of corruption when restoring to a new install?

johnpoz

From your trace and my trace that both go through that level 3 network 4/9 network, I would "guess" has something going on there..

kevindd992002

I have a site-to-site VPN connection to the working box so I can route traffic to files00.netgate.com to the WAN connection of that box. How do you create a rule that would direct this traffic to the openvpn gateway? I mean, how do you create rules with a source of the pfsense box itself? Floating rules?

jimp

You can't policy route from the firewall itself but since the destination is static, you can add the package server IP addresses to IPv4 Remote network(s) for the OpenVPN Client. For example, add 162.208.119.40/32,162.208.119.41/32

jimp

And FYI- I had our IT staff check and both package servers can trace a route all the way back to you (or at least one hop out) so it does appear to be a problem with the traffic from your location coming this way.

Hopefully nudging that to use your VPN works around it.

johnpoz

Oh a completely unrelated topic, I think you are the first one I have seen using the home.arpa name space.. Nice! I guess you are a rfc reader.. You just didn't accidentally decide to use that - did you? Curious.

kevindd992002

@jimp said in Stuck upgrade of squid after pfsense upgrade 2.4.5:

You can't policy route from the firewall itself but since the destination is static, you can add the package server IP addresses to IPv4 Remote network(s) for the OpenVPN Client. For example, add 162.208.119.40/32,162.208.119.41/32

@jimp said in Stuck upgrade of squid after pfsense upgrade 2.4.5:

And FYI- I had our IT staff check and both package servers can trace a route all the way back to you (or at least one hop out) so it does appear to be a problem with the traffic from your location coming this way.

Hopefully nudging that to use your VPN works around it.

Ok, that seems to have worked! Though not optimal, I can settle with it for now until I can report it to my ISP tomorrow. The weird thing is that these two boxes use the same ISP. The affected box uses a public static IP assigned directly to pfsense's WAN interface and the working box is behind CGNAT. For what it's worth, here's the traceroute from the working box:

1  192.168.100.1 (192.168.100.1)  0.548 ms  0.371 ms  0.304 ms
 2  10.188.16.1 (10.188.16.1)  3.142 ms  3.594 ms  3.606 ms
 3  172.20.20.93 (172.20.20.93)  3.832 ms  5.295 ms  3.420 ms
 4  181.1.49.161-rev.convergeict.com (161.49.1.181)  2.794 ms  2.719 ms  2.727 ms
 5  69.1.49.161-rev.convergeict.com (161.49.1.69)  5.239 ms  10.829 ms  2.981 ms
 6  161.49.1.250 (161.49.1.250)  15.138 ms  4.752 ms  4.578 ms
 7  * * *
 8  * * *
 9  * * *
10  THE-NEW-YOR.ear1.Newark1.Level3.net (4.15.150.218)  205.096 ms  205.100 ms  205.004 ms
11  cs90.cs99new.v.ewr.nyinternet.net (96.47.77.218)  205.165 ms  205.674 ms  205.380 ms
12  162.208.119.41 (162.208.119.41)  205.012 ms  205.114 ms  204.837 ms

1  192.168.100.1 (192.168.100.1)  0.674 ms  0.494 ms  0.322 ms
 2  10.188.16.1 (10.188.16.1)  2.990 ms  2.920 ms  3.217 ms
 3  172.20.20.93 (172.20.20.93)  60.350 ms  15.737 ms  3.726 ms
 4  181.1.49.161-rev.convergeict.com (161.49.1.181)  3.000 ms  2.895 ms  2.916 ms
 5  13.1.49.161-rev.convergeict.com (161.49.1.13)  10.875 ms  3.152 ms  3.102 ms
 6  246.1.49.161-rev.convergeict.com (161.49.1.246)  4.571 ms  4.453 ms  4.556 ms
 7  * * *
 8  * * *
 9  * * *
10  THE-NEW-YOR.ear1.Newark1.Level3.net (4.15.150.218)  208.253 ms  208.109 ms  208.061 ms
11  cs90.cs99new.v.ewr.nyinternet.net (96.47.77.218)  208.437 ms  208.363 ms  208.219 ms
12  162.208.119.40 (162.208.119.40)  207.967 ms  208.587 ms  208.425 ms

So I guess it's time I edit my config.xml now to include the edits I made for the OpenVPN remote networks and start from scratch by restoring again. In case you missed my question about this:

If there are any corruption in a previous install, would backing up the config.xml of that corrupted install carry over any type of corruption when restoring to a new install?

@johnpoz said in Stuck upgrade of squid after pfsense upgrade 2.4.5:

Oh a completely unrelated topic, I think you are the first one I have seen using the home.arpa name space.. Nice! I guess you are a rfc reader.. You just didn't accidentally decide to use that - did you? Curious.

You got it! I work as an MS Exchange sysad so I read up on RFC's from time to time. When I was optimizing my home networks, I researched on what the proper local domain name is to avoid any potential conflicts with any system and stumbled upon the arpa domain.