[Solved][But bug still present] Available Packages empty

[zippydan]

Seems to be repeat of this:

Jan 2019: https://forum.netgate.com/topic/139903/available-packages-is-empty-in-package-manager
Apr 2018: https://forum.netgate.com/topic/129818/no-available-packages/
Sep 2017: https://forum.netgate.com/topic/120711/no-available-packages-empty-list

I've two pretty new VM installs of 2.4.5 (on two different physical machines) and both are experiencing the same symptoms.

1. Dashboard -> Version, often hangs on Obtaining update status.
2. System -> Package Manager -> Installed Packages, often hangs on Please wait while the list of packages is retrieved and formatted. I only have one package installed. Sometimes after a long wait, it shows the one package. Sometimes it fails with Unable to retrieve package information.
3. System -> Package Manager -> Available Packages, shows Search field and Packages field with Name, Version, Description headers, but the list is empty.

It's been this way for about 3 days straight. I've tried rebooting with no effect. In System -> Update -> Update Settings I've tried disabling Dashboard auto-update check (as per one of the threads linked above).

No go.

If I go to Diagnostics -> System Activity, all four cores are near 100% idle (these pfSense installs are not actually performing any task yet), unlike the reports in the above links that the pkg process was pegged with high CPU usage.

This issue is over a year old and still occurring? Or this is a new occurrence?

[Gertjan]

@zippydan said in Available Packages empty:

Or this is a new occurrence?

Two possibilities :

The classic one : DNS (for pfSense itself) is broken.
Both lists, "Installed" and "Available", check the remote "pfsense-netgate" server for updates.
They are contacted using an URL that has to be resolved first. It can't, and after a 60 sec (or so) it fails ... lists stay empty.

The other one : the package database locally is 'not in a goed shape'.
Have it rebuild.
The manual handles the series of commands to type to get things working again.

These 2 cover 99,x %of all cases.
The first one is very popular.

[zippydan]

If DNS were broken, then why would the dashboard update checker sometimes confirm I am running the latest version?

If DNS were broken, then why would the Installed list sometimes (sometimes instantly, sometimes after a minute delay) confirm I have open-vm-tools installed?

If I go to Diagnostics -> DNS Lookup, I can resolve the A record for netgate.com nearly instantly.

I'll try the second one, but it seems strange that both installs would fail with bad package databases at the same time.

I assume this is the guide I should follow for repairing the package database:

https://docs.netgate.com/pfsense/en/latest/packages/fixing-a-broken-pkg-database.html

[zippydan]

@zippydan said in Available Packages empty:

I assume this is the guide I should follow for repairing the package database:

https://docs.netgate.com/pfsense/en/latest/packages/fixing-a-broken-pkg-database.html

I'm failing on the second step:

[2.4.5-RELEASE][root@server]/: /usr/sbin/pkg-static update -f
/usr/sbin/pkg-static: Command not found.
[2.4.5-RELEASE][root@server]/: cd /usr/sbin/
[2.4.5-RELEASE][root@server]/: ls pkg*
pkg

So it looks like pkg-static is not even available on this system?

For fun, I tried using pkg instead of pkg-static, and it seems to be working. Does this documentation need to be updated?

[zippydan]

Well, it seems there is definitely something wrong on the connection.

When doing:

[2.4.5-RELEASE][root@server]/: /usr/sbin/pkg update -f

I get the following errors:

pkg: https://pkg.pfsense.org/pfSense_v2_4_5_amd64-core/meta.txz : No route to host
pkg: https://pkg.pfsense.org/pfSense_v2_4_5_amd64-pfSense_v2_4_5/packagesite.txz : No route to host

Here's what weird about this:

1. A few days before, I was able to install open-vm-tools through the package manager without any problems. I haven't made any DNS changes since then.
2. DNS seems to resolve random test queries just fine.
3. I tried disabling the DNS Resolver, just for fun, and it made no difference. (Based on this mostly unhelpful reddit thread I found.)
4. Why is is this issue affecting both new installs? The installs were done independently from scratch for both VMs.
5. I have another older pfSense VM running on the same physical machine as one fo these pfSense installs. It's been running since version 2.2.x of pfSense and is now running 2.4.4. I double-checked the DNS setup on that install, and it seems identical. I have no trouble accessing package manager on this install.

[kiokoman]

No route to host:
The firewall cannot reach the update server because it cannot find a route there. Most likely, the firewall is missing its default route or the WAN with the default route is down.

maybe try to follow this
https://pfsense-docs.readthedocs.io/en/latest/install/upgrade-troubleshooting.html

[zippydan]

Both my gateways show as up in Status -> Gateways.

I can also ping any place on the internet just fine. I can imagine this is a DNS problem, but the internet connection itself is rock solid.

As for routes on the firewall:

1. This is a nearly fresh install, with only the LAN and WAN interfaces defined. No routes or special firewall rules are defined, other than whatever is part of a default install.
2. These are edge routers, plugged directly into the modem.
3. I was previously able to install open-vm-tool package without a problem, and haeand have not made any routing, DNS, or firewall changes since.

Based on this thread:
https://forum.netgate.com/topic/126277/pkg-pfsense-org-dns-record-not-found/

I issued command:

[2.4.5-RELEASE][root@server]/: dig _https._tcp.pkg.pfsense.org SRV

I receive immediate results:

Got answer:
->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63741

Also:

;; ANSWER SECTION:
 _https._tcp.pkg.pfsense.org.  60  IN          SRV          10  10  443  files00.netgate.com.
 _https._tcp.pkg.pfsense.org.  60  IN          SRV          10  10  443  files01.netgate.com.

Also:

Query time: 82ms

I got the same result on both installs, one using DNS resolver (showing reply from 127.0.0.1) and the other using my ISPs' external DNS servers.

[Gertjan]

@zippydan said in Available Packages empty:

I issued command:
dig _https._tcp.pkg.pfsense.org SRV

I receive immediate results:
Got answer:
->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63741

Also:
;; ANSWER SECTION:
_https._tcp.pkg.pfsense.org. 60 IN SRV 10 10 443 files00.netgate.com.
_https._tcp.pkg.pfsense.org. 60 IN SRV 10 10 443 files01.netgate.com.

Definitely no DNS troubles

@kiokoman said in Available Packages empty:

Most likely, the firewall is missing its default route or the WAN with the default route is down.

Wouldn't that impact all (Internet) outbound traffic ?

@zippydan said in Available Packages empty:

5. ..... I have no trouble accessing package manager on this install.

So, from the same site, a pfSense install can access the package server of Netgate at https://pkg.pfsense.org/pfSense_v2.... , and another local install can not ?

I hope' this is related to a local upstream router, because "No route to host" issues upstream are not for you to debug ...

Of, at worst : filesxx.netgate.com isn't available at (all) the moments you were trying ? This could happen during massive upgrade times, but that's not the case right now.

[zippydan]

Solved!

Following the instructions here: https://pfsense-docs.readthedocs.io/en/latest/install/upgrade-troubleshooting.html#pkg-pfsense-org-has-no-a-aaaa-record

I double-checked for any DNS issues using the alternate host command. But my results were all as expected:

[2.4.5-RELEASE][root@server]/: host -t srv _https._tcp.pkg.pfsense.org
_https._tcp.pkg.pfsense.org has SRV record 10 10 443 files01.netgate.com.
_https._tcp.pkg.pfsense.org has SRV record 10 10 443 files00.netgate.com.

[2.4.5-RELEASE][root@server]/: host files01.netgate.com.
files01.netgate.com has address 162.208.119.40
files01.netgate.com has IPv6 address 2610:1c1:0:6::40

[2.4.5-RELEASE][root@server]/: host files00.netgate.com.
files00.netgate.com has address 162.208.119.41
files00.netgate.com has IPv6 address 2610:1c1:0:6::41

So, I followed up with the `curl test:

[2.4.5-RELEASE][root@server]/: curl --output /dev/null --silent --head --fail "https://files00.netgate.com/pfSense_v2_4_5_amd64-core/meta.txz"
[2.4.5-RELEASE][root@server]/: echo $?
7

A 7 result for curl basically indicates the same No route to host failure I was getting before. So DNS is definitely working, but for some reason I can't access the files on the remote server.

The next section of the troubleshooting guide gave me the clue I needed to fix the problem (thanks to @kiokoman for the original link): https://pfsense-docs.readthedocs.io/en/latest/install/upgrade-troubleshooting.html#ipv6-connectivity-problems

The answer isn't in the guide explicitly, but this idea that pfsense defaults to always trying IPv6 first was intriguing.

On initial setup, my WAN connection defaulted to DHCP. This is when I was able to successfully download the open-vm-tools package. Once I setup my LAN and WAN interfaces (which was basically the only thing I had configured), I input the static (IPv4) IPs that I use for my WAN, and I set none for the IPv6 configuration.

Upon checking my aforementioned older pfsense install, which is working fine, the WAN configuration (which uses the same modem uplink) has a static (IPv4) IP and is set to use DHCP6 for IPv6.

Sure enough, once I turned on DHCP6 for my WAN interface, even though none is defined for my default IPv6 gateway, the Available Packages page immediately started working.

This seems like a pretty bad bug if pfSense is unable to gracefully fallback to IPv4 from IPv6 for updates and packages on a system with no defined IPv6 interfaces.

[Gertjan]

@zippydan said in Available Packages empty:

The answer isn't in the guide explicitly, but this idea that pfsense defaults to always trying IPv6 first was intriguing.

Correct.
Good to see you nailed it.

Most - of not all - modern operating system switched their default IP stack to IPv6 a decade ago.
And I agree with you : it, at least, the GUI, should fall back to 'the other one' if IPv6 doesn't work out.

Take note that, using the curl command - which has many (no, more ...) command line options, it uses most of it's default options when command line option are not given by the operator/admin (also known as 'God' to the system) , like

curl -6  --output /dev/null.....

The -6 is probably default, and I'm sure you know what that means.

If you want to use IPv4, you have to

curl -4  --output /dev/null.....

Be aware that a command like curl will not "try one option" if something didn't work out.
It presumes that the guy who types the command is aware effects and side effects.
Command line command are : what you type is what you get ;)

Example :
I do not use the "-6" option, but I chose the "-v" because I'm wondering what happens.
-v stands for Verbose, of course.

And guess what ? :

[2.4.5-RELEASE][admin@pfsense.brit-hotel-fumel.net]/root: curl  -v --output /dev/null --silent --head --fail "https://files00.netgate.com/pfSense_v2_4_5_amd64-core/meta.txz"
*   Trying 2607:ee80:10::119:41:443...
* TCP_NODELAY set
* Connected to files00.netgate.com (2607:ee80:10::119:41) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
....

That's an solid IPv6 connection.

@zippydan said in Available Packages empty:

This seems like a pretty bad bug if pfSense is unable to gracefully downgrade from IPv6 to IPv4 for updates and packages on a system with no defined IPv6 interfaces.

I do have a IPv6 access, not via my WAN ISP interface, but a separated interface, from he.net - some sort of "IPv6 over IPv4 tunnel" scheme that permits my pfSense to have a IPv4 over WAN and IPv6 over some other interface.
Works great.
Both are listed :

If my IPv6 is down or not activated, I have this impression that my outgoing IPv6 requests eventually fall back to IPv4.

You probably had a - temporary - situation where pfSense was thinking - sorry for the expression - that it had working IPv6 uplink available and didn't recognize it wasn't the case.

This dual IPvX thing will haunt us for the years to come ;)

[kiokoman]

@Gertjan said in Available Packages empty:

Wouldn't that impact all (Internet) outbound traffic ?

not necessary, could be a isp issue,

could be related to this, maybe
https://forum.netgate.com/topic/153464/pfsense-updates-an-package-installation-very-slow-when-using-deutsche-telekom

anyway his ipv6 is missing a route to netgate

[zippydan]

@kiokoman said in [Solved][But bug still present] Available Packages empty:

anyway his ipv6 is missing a route to netgate

I wasn't missing a route. pfSense insists on trying IPv6 first, even if no IPv6 interfaces are defined. This is semi-understandable considering IPv6 is the standard of the present/future, but seems to me that pfSense could check to see if there is any IPv6 configuration defined before attempting to use that protocol.

The real problem, though, is that it seems to be incapable of graceful fallback to IPv4.

A secondary problem is that the webUI gives absolutely no useful feedback about what is going on. The Dashboard update indicator just spins endlessly. The Installed Packages page just spins endlessly. Available Packages just shows a blank list. That's a pretty obtuse and unhelpful way of communicating there is a problem.

Some error messages like:

Could not resolve SRV records for _https._tcp.pkg.pfsense.org
or
Could not resolve A/AAAA records for files01.netgate.com
or
Timeout attempting to contact files01.netgate.com at 2610:1c1:0:6::40
or
Timeout attempting to contact files01.netgate.com at 162.208.119.41

Would be much more useful error messages, indicating problems with DNS, IPv6 connections, or IPv4 connections, respectively, and would have saved me hours of troubleshooting time.

[tinman]

A downgrade then upgrade in the UI worked for me.