Unable to ping Ubuntu software repository

sgseidel

In a nutshell, my issue is I am unable to ping the Ubuntu software repository at ca.archive ubuntu.com from pfSense.

Background: I upgraded to pfSense 2.5.2 today from 2.5.0. However the issue I will describe pre-dates the pfSense upgrade and started some weeks ago with 2.5.0. The Dashboard shows the following services:
arpwatch, bandwidthd, darkstat, dhcpd, dpinger, iperf, lldpd, miniupnpd, ntopng, ntpd, pcscd, pfb_dnsbl (temporarily disabled), pfb_filter (temporarily disabled), sshd, suricata (temporarily disabled), syslogd, unbound.

I have an Ubuntu 20.04.2 LTS server running on a Virtualbox VM. This Ubuntu server has been running for years without incident, and patches have been applied successfully countless times. However within the past month or so, when I attempt to upgrade Ubuntu with the latest software patches, I get the error "Failed to download repository information". The software repository is at ca.archive.ubuntu.com.

I am unable to ping ca.archive.ubuntu.com from any computer on my local 172.16.0.0/24 network, including from the pfSense server at 172.16.0.1. However, I have two public IPs, 11.22.15.72 and 11.22.23.21 (anonymized IPs), both provided by the same ISP. 11.22.15.72 is protected by the pfSense firewall and 11.22.23.21 traverses a Calix router provided by my ISP. Note that altho' I can't ping ca.archive.ubuntu from the local 172.16.0.0/24 network connected to 11.22.15.72, I CAN ping that same IP from the Calix router on the 11.22.23.21 network with zero packet loss and 63ms average RTT (round trip time).

Note that the DNS servers listed on the pfSense Dashboard are: 1.1.1.1, 9.9.9.9, and 8.8.8.8. Here are nslookup results from the pfSense server at 172.16.0.1:

[2.5.2-RELEASE][admin@pfSense252.novuscom.net]/root: nslookup
> ca.archive.ubuntu.com
Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
ca.archive.ubuntu.com	canonical name = us.archive.ubuntu.com.
Name:	us.archive.ubuntu.com
Address: 91.189.91.38
Name:	us.archive.ubuntu.com
Address: 91.189.91.39
Name:	us.archive.ubuntu.com
Address: 2001:67c:1562::15
Name:	us.archive.ubuntu.com
Address: 2001:67c:1562::18
> server 9.9.9.9
Default server: 9.9.9.9
Address: 9.9.9.9#53
> ca.archive.ubuntu.com
Server:		9.9.9.9
Address:	9.9.9.9#53

Non-authoritative answer:
ca.archive.ubuntu.com	canonical name = us.archive.ubuntu.com.
Name:	us.archive.ubuntu.com
Address: 91.189.91.39
Name:	us.archive.ubuntu.com
Address: 91.189.91.38
Name:	us.archive.ubuntu.com
Address: 2001:67c:1562::18
Name:	us.archive.ubuntu.com
Address: 2001:67c:1562::15
> server 8.8.8.8
Default server: 8.8.8.8
Address: 8.8.8.8#53
> ca.archive.ubuntu.com
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
ca.archive.ubuntu.com	canonical name = us.archive.ubuntu.com.
Name:	us.archive.ubuntu.com
Address: 91.189.91.39
Name:	us.archive.ubuntu.com
Address: 91.189.91.38
Name:	us.archive.ubuntu.com
Address: 2001:67c:1562::15
Name:	us.archive.ubuntu.com
Address: 2001:67c:1562::18
> server 172.16.0.1
Default server: 172.16.0.1
Address: 172.16.0.1#53
> ca.archive.ubuntu.com
Server:		172.16.0.1
Address:	172.16.0.1#53

Non-authoritative answer:
ca.archive.ubuntu.com	canonical name = us.archive.ubuntu.com.
Name:	us.archive.ubuntu.com
Address: 91.189.91.39
Name:	us.archive.ubuntu.com
Address: 91.189.91.38
Name:	us.archive.ubuntu.com
Address: 2001:67c:1562::18
Name:	us.archive.ubuntu.com
Address: 2001:67c:1562::15
>

I am able to ping www.google.com from the pfSense server, but I'm getting "Permission denied" error when I try to ping ca.archive.ubuntu.com.

[2.5.2-RELEASE][admin@pfSense252.novuscom.net]/root: ping www.google.com
PING www.google.com (142.250.217.68): 56 data bytes
64 bytes from 142.250.217.68: icmp_seq=0 ttl=122 time=5.141 ms
64 bytes from 142.250.217.68: icmp_seq=1 ttl=122 time=5.121 ms
64 bytes from 142.250.217.68: icmp_seq=2 ttl=122 time=5.178 ms
64 bytes from 142.250.217.68: icmp_seq=3 ttl=122 time=5.260 ms
^C
--- www.google.com ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 5.121/5.175/5.260/0.053 ms
[2.5.2-RELEASE][admin@pfSense252.novuscom.net]/root: ping ca.archive.ubuntu.com
PING us.archive.ubuntu.com (91.189.91.38): 56 data bytes
ping: sendto: Permission denied
ping: sendto: Permission denied
ping: sendto: Permission denied
ping: sendto: Permission denied
^C
--- us.archive.ubuntu.com ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
[2.5.2-RELEASE][admin@pfSense252.novuscom.net]/root:

It seems pfSense is somehow blocking this ca.archive.ubuntu.com IP address, but I see no indication in the pfSense logs nor anywhere else. I'm at a loss about where to go from here. Any suggestions are appreciated.

DaddyGo

@sgseidel said in Unable to ping Ubuntu software repository:

unable to ping the Ubuntu software repository at ca.archive ubuntu.com from pfSense.

Hi,

Just an idea

This rule used to cause similar problems if you use Suricata (but I don't think so now)
Many our Ubuntu installations run behind pfSense so we suppressed it...

#ET INFO [eSentire] Possible Linux Updates
suppress gen_id 1, sig_id 2025627, track by_dst, ip

This is what it looks like for me now and there is no problem (with similar settings):

note that it's not enough to disable the IPS/IDS, you have to restart it

++++edit:

On host:

sgseidel

@daddygo Thanks for your reply.

Indeed, Suricata was the culprit. As I had mentioned in my original post, I had temporarily disabled both Suricata and pfBlocker. What I didn't say was that Suricata had been configured to show alerts and not do any blocking. Even with Suricata configured this way, I was unable to ping ca.archive.unbuntu.com.

Following your advice I backed up pfSense, and removed the package from the package manager. Even then I was unable to ping ca.archive.unbuntu.com. After removing the Suricata package, it was only after rebooting pfSense that I was able to ping ca.archive.unbuntu.com.

In closing, I should say that it was almost exactly one month since I was able to successfully connect to ca.archive.unbuntu.com and thus update the Ubuntu software. Prior to this time I had been able to connect to ca.archive.unbuntu.com with Suricata running in alert mode. So it's clear to me that something in Suricata was changed, perhaps during an update, which caused ca.archive.unbuntu.com to be unavailable. IMO, Suricata is, in general, a very time consuming pig to configure; my observation is that Suricata renders very many harmless IP addresses inaccessible and it was for that reason that I set it to alert-only mode some time ago.

DaddyGo

@sgseidel said in Unable to ping Ubuntu software repository:

my observation is that Suricata renders very many harmless IP addresses inaccessible and it was for that reason that I set it to alert-only mode some time ago.

Yes, the "world" of IDS/IPS is like that...

Bill always recommends to start with just alarm mode for a while and slowly build up your defenses. (connectivity / balanced / security / and I would not go any further).

I usually spend 3 - 6 months monitoring the installation, before "I light it up"...

bmeeks

Suricata (and Snort) are very often misunderstood tools. Many users seem to think you just install the package, enable ALL the rules, and then you don't need to touch it again. That is far, far from the truth. The IDS/IPS packages require a highly skilled security admin at the helm who understands the security exposures in their network, and only deploys the rules explicitly designed to defend just those exposures.

The ET INFO rules are a case in point. Those rules are designed to provide "information" about what kinds of traffic is coming and going on the network. Hence their title "Emerging Threats Information (info) rules". You almost never want those rules blocking, as they will probably block things that are really not bad. The rule exists to let you know that something on your network called out for a Linux update. There are similar ET INFO rules for Windows. 95% or more of the users out there never need to enable the ET INFO rules. They are noise. However, consider an enterprise setup where maybe you have your own internal repo for Linux RPMs (or a Windows Update Services Server) for internal clients, and the clients are to get updates from there instead instead of getting un-vetted updates from the mothership. In that scenario, the ET INFO rules that alert you to internal clients phoning out to the mothership for updates instead of using your approved internal server would be useful. The alerts would let you know you had a misconfigured client.

The correct way to use Suricata in Legacy Mode in my opinion is to enable the "Block on DROPs Only" option on the INTERFACE SETTINGS tab, then use the SID MGMT features to set only carefully selected rules to DROP (and thus "block" traffic). Leave all the other rules set to the default of ALERT. That way you get alerts on all traffic, but only the truly bad stuff is blocked by way of the rules you modifed to DROP using the features on the SID MGMT tab, or manually using the icons on the ALERTS and RULES tabs. Do as @DaddyGo suggested, let the IDS run in alerts-only mode for several weeks (or months even), so you can get a picture of what constitutes "normal traffic" on your network. Then carefully pick which rules you want to actually block stuff, and set those to DROP. Leave everything else set to ALERT, Suppressed or Disabled (if necessary).

When using Legacy Mode Blocking, Suricata actually blocks by putting the offender's IP address in a pfSense firewall table called snort2c. Any IP placed in that table is blocked by a hidden firewall rule created by pfSense at boot-up. The IP stays in the table until removed. Uninstalling the package will cause Suricata to clear the table. But simply stopping an interface will not, because the table is shared with all running instances. You can clear the table using the button on the BLOCKS tab, by using the option under DIAGNOSTICS > TABLES, or by rebooting the firewall.

sgseidel

@bmeeks Thank you so much for your post, which is very informative on the subject of Suricata. I am aware that there is a lot more to Suricata than simply installing the package and enabling all the rules. YouTube videos from Lawrence Systems got me started on Suricata and I did spend some weeks tuning Suricata, narrowing down the rules by trial and error. Of course it always seemed another website became inaccessible, and some time ago I decided to put Suricata in alert mode rather than block mode, and maybe wait for when I had more time to do additional tuning. Since Suricata was in alert mode, it didn't cross my mind that Suricata (or was it SNORT) would have been responsible for blocking access to Ubuntu, since I had had access to Ubuntu software previously, even with Suricata running.

Thanks again for taking the time to reply to my post.

bmeeks

@sgseidel said in Unable to ping Ubuntu software repository:

@bmeeks Thank you so much for your post, which is very informative on the subject of Suricata. I am aware that there is a lot more to Suricata than simply installing the package and enabling all the rules. YouTube videos from Lawrence Systems got me started on Suricata and I did spend some weeks tuning Suricata, narrowing down the rules by trial and error. Of course it always seemed another website became inaccessible, and some time ago I decided to put Suricata in alert mode rather than block mode, and maybe wait for when I had more time to do more tuning. Since Suricata was in alert mode, it didn't cross my mind that Suricata (or was it SNORT) would have been responsible for blocking access to Ubuntu, since I had had access to Ubuntu software previously, even with Suricata running.

Thanks again for taking the time to reply to my post.

But simply changing the mode is not enough. You need to also restart Suricata on the interface (if you did not do that) after saving the mode change. You would then also need to go to the BLOCKS tab and clear all the blocked hosts to remove the IP addresses from the firewall's snort2c table. Skipping one of these would likely be why you still were blocked from accessing the web site. Removing the package will clear the blocks table.

And my post was not meant to chastise you. Rather, I post more or less the same message in a number of the Suricata and Snort threads to spread the word to new users -- "it's not like deploying an anti-virus product".

sgseidel

@bmeeks Thanks. Yes, I did remove the Suricata package from pfSense with the result that ca.archive.ubuntu.com was no longer inaccessible.

DaddyGo

@sgseidel said in Unable to ping Ubuntu software repository:

got me started on Suricata and I did spend some weeks tuning Suricata,

I don't want to discourage you , but it took me at least 1 year of intensive use before I could say that I had some understanding of IDS/IPS. Since then I have been running in In-Line mode in several installations.
(then I started to look at the inside of the rules to understand what they wanted - so the things you described in your original post were familiar)

I'm glad you've started using it, as many people these days have started to neglect this great protection tool and / or implementation.

The forum here is full of Bill's great and very detailed descriptions, I don't know of any other package maintainer who documents his work so conscientiously!

Lawrence Systems can also be used at some points, but if you want to do well on this theme Bill @bmeeks is always helpful with advice.

sgseidel

@daddygo said in Unable to ping Ubuntu software repository:

The forum here is full of Bill's great and very detailed descriptions, I don't know of any other package maintainer who documents his work so conscientiously!

I wasn't aware that @bmeeks was a package maintainer for Suricata so thanks for alerting me.

Tho' I do understand that Suricata isn't plug and play, I have to admit that devoting one year of intensive use to understanding IDS/IPS is a bit daunting. I'm a retired computer professional running a home network so my requirements--even with twenty or so users who use my homebrew VoIP phone system, Plex, calibre, and so on--are most likely much less than many others who use pfSense. I've decided to leave Suricata uninstalled for the time being, but I am clear that it most certainly provides a necessary service and I will revisit my decision later when I can devote more time to the project.

This little posting of mine has been very helpful. Thanks to all who replied.

DaddyGo

@sgseidel said in Unable to ping Ubuntu software repository:

maintainer for Suricata so thanks for alerting me.

and also the Snort package

don't take it as intrusive (I know grandchildren are important too):

the 1 year I described as a deep learning, but don't let that put you off...

you say you're retired (IT spec.), so you might have more time to learn this great tool