Linux apt update/upgrade stopped working

johnpoz · May 2, 2022, 2:44 PM

@maddy_in65 can you access it in your browser?

Either http or https

login-to-view

Also that archive.raspberrypi.org is a cname that points to 2 other fqdn, and resolves to many different IPs.

login-to-view

I have a couple of pis and not having any issues updating them via apt..

login-to-view

maddy_in65 · May 2, 2022, 2:59 PM

@johnpoz
Yes I can access it my main PC (Linux mint) which is on another VLAN. i dont have any issue with apt update on this PC.
login-to-view
Dig Command also gives me similar results.

user@host:~ $ dig archive.raspberrypi.org

; <<>> DiG 9.16.27-Raspbian <<>> archive.raspberrypi.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19692
;; flags: qr rd ra; QUERY: 1, ANSWER: 14, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;archive.raspberrypi.org.	IN	A

;; ANSWER SECTION:
archive.raspberrypi.org. 183	IN	CNAME	lb.raspberrypi.org.
lb.raspberrypi.org.	183	IN	CNAME	lb.raspberrypi.com.
lb.raspberrypi.com.	183	IN	A	176.126.240.167
lb.raspberrypi.com.	183	IN	A	46.235.231.145
lb.raspberrypi.com.	183	IN	A	46.235.230.122
lb.raspberrypi.com.	183	IN	A	46.235.231.151
lb.raspberrypi.com.	183	IN	A	93.93.135.141
lb.raspberrypi.com.	183	IN	A	93.93.135.117
lb.raspberrypi.com.	183	IN	A	176.126.240.86
lb.raspberrypi.com.	183	IN	A	46.235.227.39
lb.raspberrypi.com.	183	IN	A	93.93.135.118
lb.raspberrypi.com.	183	IN	A	176.126.240.84
lb.raspberrypi.com.	183	IN	A	46.235.231.111
lb.raspberrypi.com.	183	IN	A	93.93.130.212

;; Query time: 9 msec
;; SERVER: 192.168.11.1#53(192.168.11.1)
;; WHEN: Mon May 02 20:27:11 IST 2022
;; MSG SIZE  rcvd: 293

maddy_in65 · May 2, 2022, 3:01 PM

@crowfather
Not sure about this but zabbix repo is working fine but raspberry pi/rock64 repos are not working.
login-to-view

stephenw10 · May 2, 2022, 3:21 PM

Can you fetch the actual file it's looking for?

[2.6.0-RELEASE][admin@pfsensemirror.stevew.lan]/root: fetch http://raspbian.raspberrypi.org/raspbian/dists/bullseye/InRelease
InRelease                                               14 kB   13 MBps    00s

Steve

johnpoz · May 2, 2022, 5:16 PM

@maddy_in65 said in Linux apt update/upgrade stopped working:

which is on another VLAN. i dont have any issue with apt update on this PC.

Well what is different on your vlans? Different rules? Different dns? IPS, Proxy? Forcing traffic out a vpn? Different internet connection being used?

If your devices on another vlan works - then clearly its not a pfsense internet connection sort of thing.. So what is different about vlan that works and vlan that doesn't?
What is odd is your traceroute.. Why is pfsense not seen as your first hop?

user@host:~ $ traceroute 46.235.231.151
traceroute to 46.235.231.151 (46.235.231.151), 30 hops max, 60 byte packets
 1  103.134.162.24 (103.134.162.24)  1.611 ms 45.112.0.37 (45.112.0.37)  1.638 ms 103.134.162.24 (103.134.162.24)  1.230 ms
 2  * 45.112.0.33 (45.112.0.33)  2.448 ms  2.927 ms

Your pfsense lan side IP for this vlan is a 103.x.x.x address?

maddy_in65 · May 3, 2022, 4:37 AM

@stephenw10 said in Linux apt update/upgrade stopped working:

Can you fetch the actual file it's looking for?

[2.6.0-RELEASE][admin@pfsensemirror.stevew.lan]/root: fetch http://raspbian.raspberrypi.org/raspbian/dists/bullseye/InRelease
InRelease                                               14 kB   13 MBps    00s

Steve

This is working fine on pfsense

[2.6.0-RELEASE][admin@PfSense]/home: fetch http://raspbian.raspberrypi.org/raspbian/dists/bullseye/InRelease
InRelease                                               14 kB  118 kBps    00s
[2.6.0-RELEASE][admin@PfSense]/home: ls -lah
drwxr-xr-x   2 root  wheel     3B May  3 09:21 .
drwxr-xr-x  22 root  wheel    28B Feb 26 16:47 ..
-rw-r--r--   1 root  wheel    15K May  3 04:02 InReleas

maddy_in65 · May 3, 2022, 4:42 AM

@johnpoz said in Linux apt update/upgrade stopped working:

Well what is different on your vlans? Different rules? Different dns? IPS, Proxy? Forcing traffic out a vpn? Different internet connection being used?

There is no separate DNS , IPS. All VLANs uses same resources. No Proxy, no VPN in my network.
I am having two internet connections and each VLAN is using load balancer gateway.

If your devices on another vlan works - then clearly its not a pfsense internet connection sort of thing.. So what is different about vlan that works and vlan that doesn't?

I am not sure why this is not working in server VLAN.

What is odd is your traceroute.. Why is pfsense not seen as your first hop?

No Idea why it is using ISP gateway as first hop. I have checked traceroute from my desktop system and it is showing similar results.

user@host:~ $ traceroute 46.235.231.151
traceroute to 46.235.231.151 (46.235.231.151), 30 hops max, 60 byte packets
1 103.134.162.24 (103.134.162.24) 1.611 ms 45.112.0.37 (45.112.0.37) 1.638 ms 103.134.162.24 (103.134.162.24) 1.230 ms
2 * 45.112.0.33 (45.112.0.33) 2.448 ms 2.927 ms

Your pfsense lan side IP for this vlan is a 103.x.x.x address?
No that is ISP gateway. my LAN side IP of this VLAN is 192.168.11.0/24

johnpoz · May 3, 2022, 11:23 AM

@maddy_in65 does where it work have IPv6 or not? Could see failing on IPv6 in your output try forcing it to just use IPv4

apt-get -o Acquire::ForceIPv4=true update

maddy_in65 · May 3, 2022, 12:54 PM

@johnpoz said in Linux apt update/upgrade stopped working:

@maddy_in65 does where it work have IPv6 or not? Could see failing on IPv6 in your output try forcing it to just use IPv4
apt-get -o Acquire::ForceIPv4=true update

I have IPv6 blocked in pfsense.

login-to-view

I have tested forcing apt-get Ipv4 but no luck. Checked on all 3 SBCs and its same issue. Docker/Zabbix/mongodb all these repos work fine but raspberry, port.ubuntu, launchpad all these don't work.
I am really banging my head against desk since last week to get this worked. This is really frustrating.

Get:1 https://repo.zabbix.com/zabbix/6.0/raspbian bullseye InRelease [7,126 B]                                                    
Get:2 https://repo.zabbix.com/zabbix/6.0/raspbian bullseye/main Sources [1,231 B]                                     
Get:3 https://repo.zabbix.com/zabbix/6.0/raspbian bullseye/main armhf Packages [5,133 B]                                                       
Err:4 http://raspbian.raspberrypi.org/raspbian bullseye InRelease                                                                                                   
  Could not connect to raspbian.raspberrypi.org:80 (93.93.128.193), connection timed out
Err:5 http://archive.raspberrypi.org/debian bullseye InRelease                                                      
  Could not connect to archive.raspberrypi.org:80 (46.235.231.145), connection timed out Could not connect to archive.raspberrypi.org:80 (93.93.130.212), connection timed out Could not connect to archive.raspberrypi.org:80 (93.93.135.118), connection timed out Could not connect to archive.raspberrypi.org:80 (176.126.240.84), connection timed out Could not connect to archive.raspberrypi.org:80 (46.235.231.111), connection timed out Could not connect to archive.raspberrypi.org:80 (93.93.135.141), connection timed out Could not connect to archive.raspberrypi.org:80 (46.235.227.39), connection timed out Could not connect to archive.raspberrypi.org:80 (46.235.231.151), connection timed out Could not connect to archive.raspberrypi.org:80 (176.126.240.167), connection timed out Could not connect to archive.raspberrypi.org:80 (93.93.135.117), connection timed out Could not connect to archive.raspberrypi.org:80 (46.235.230.122), connection timed out Could not connect to archive.raspberrypi.org:80 (176.126.240.86), connection timed out
Fetched 13.5 kB in 34s (392 B/s)                           
Reading package lists... Done
W: Failed to fetch http://raspbian.raspberrypi.org/raspbian/dists/bullseye/InRelease  Could not connect to raspbian.raspberrypi.org:80 (93.93.128.193), connection timed out

Rock64:

Get:1 https://download.docker.com/linux/debian buster InRelease [54.0 kB]
Hit:2 https://download.docker.com/linux/ubuntu focal InRelease                                                    
Ign:3 https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 InRelease                        
Hit:4 https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 Release                                                                       
Err:6 http://apt.armbian.com/region/EU focal InRelease                                                                                                           
  Could not connect to apt.armbian.com:80 (107.175.134.226), connection timed out
Err:7 http://ports.ubuntu.com focal InRelease

johnpoz · May 3, 2022, 1:01 PM

@maddy_in65 said in Linux apt update/upgrade stopped working:

I have IPv6 blocked in pfsense.

But from your output your devices were trying to use IPv6..

Your saying nothing is different between these vlans... You can call up the website (from this vlan?)

Pfsense unless you have different rules, or using IPS - pfsense doesn't know any different between lan or vlan X.. Just a different IP, that is natted to the same public IP, etc.

Could not connect to raspbian.raspberrypi.org:80 (93.93.128.193)

But your saying browser can go there - there is no difference to pfsense..

login-to-view

pfsense just knows traffic to that IP on port 80.. So why it works via browser, and not via your pi box and apt.. Not sure how that is a pfsense thing.. The only thing that could make sense is some sort of IPS filtering that doesn't like something different in the packets. But you say your not doing that..

But to base pfsense traffic to IP X on port Y is just that.. It doesn't know if browser or apt, or whatever it just knows IP and port - allowed or not allowed, oh nat this traffic to my wan IP..

So sniff on pfsense - you see the traffic to IP:80, do you see that go out the wan.. If so pfsense did its job.. If you see hit pfsense lan side interface but not go out.. Then you know your filtering when you think your not, IPS is only thing I could see filtering apt but not browser.

stephenw10 · May 3, 2022, 12:58 PM

Look for states to one of those IPs that's failing. Make sure you have states on WAN and the VLAN in question.

Get a packet capture on WAN for the same IP. See what's actually happening there, are any packets coming back at all?

Steve

maddy_in65 · May 3, 2022, 1:01 PM

@johnpoz said in Linux apt update/upgrade stopped working:

@maddy_in65 said in Linux apt update/upgrade stopped working:

I have IPv6 blocked in pfsense.

But from your output your devices were trying to use IPv6..

Your saying nothing is different between these vlans... You can call up the website (from this vlan?)

I put my laptop in server VLAN and guess what it also has same issue. Apt update stopped working.

Hit:1 https://packages.microsoft.com/repos/ms-teams stable InRelease           
Hit:2 https://repo.skype.com/deb stable InRelease                              
Err:3 http://dl.google.com/linux/chrome/deb stable InRelease                                                            
  Cannot initiate the connection to dl.google.com:80 (2404:6800:4009:812::200e). - connect (101: Network is unreachable) Could not connect to dl.google.com:80 (216.58.203.14), connection timed out
Err:4 http://ppa.launchpad.net/inkscape.dev/stable/ubuntu focal InRelease                                               
  Cannot initiate the connection to ppa.launchpad.net:80 (2001:67c:1560:8008::19). - connect (101: Network is unreachable) Could not connect to ppa.launchpad.net:80 (91.189.95.85), connection timed out
Err:5 http://ppa.launchpad.net/tomtomtom/woeusb/ubuntu focal InRelease                                                  
  Cannot initiate the connection to ppa.launchpad.net:80 (2001:67c:1560:8008::19). - connect (101: Network is unreachable)
Err:6 http://packages.linuxmint.com una InRelease                                                                       
  Could not connect to packages.linuxmint.com:80 (208.77.20.11), connection timed out Could not connect to packages.linuxmint.com:80 (68.235.39.11), connection timed out

While the browsing is working fine, in fact I am posting this from same laptop in same VLAN

Pfsense unless you have different rules, or using IPS - pfsense doesn't know any different between lan or vlan X.. Just a different IP, that is natted to the same public IP, etc.
pfsense just knows traffic to that IP on port 80.. So why it works via browser, and not via your pi box and apt.. Not sure how that is a pfsense thing.. The only thing that could make sense is some sort of IPS filtering that doesn't like something different in the packets. But you say your not doing that..

But to base pfsense traffic to IP X on port Y is just that.. It doesn't know if browser or apt, or whatever it just knows IP and port - allowed or not allowed, oh nat this traffic to my wan IP..

So sniff on pfsense - you see the traffic to IP:80, do you see that go out the wan.. If so pfsense did its job.. If you see hit pfsense lan side interface but not go out.. Then you know your filtering when you think your not, IPS is only thing I could see filtering apt but not browser.

I can access repo website and I can see connection in states
login-to-view

HOME_SERVERS 	tcp 	192.168.11.4:34742 -> 46.235.231.111:443 	CLOSING:ESTABLISHED 	13 / 16 	2 KiB / 2 KiB

As you suggested IPS may be blocking apt update, Can i test by complete removal of Snort.
If yes what would be best way to complete removal of Snort

stephenw10 · May 3, 2022, 1:07 PM

Just disabling it will be sufficient. Or even just put it in non-blocking mode.

Just make sure to clear any existing blocks.

Check Diag > Tables. Make sure the snort2c table is empty.

Steve

maddy_in65 · May 3, 2022, 3:28 PM

@stephenw10
Tested apt updates by disabling & uninstalling Snort but no luck

stephenw10 · May 3, 2022, 4:02 PM

But is the block still in the table?

bmeeks · May 4, 2022, 12:44 AM

@maddy_in65 said in Linux apt update/upgrade stopped working:

@stephenw10
Tested apt updates by disabling & uninstalling Snort but no luck

Snort (and Suricata), when running in Legacy Mode Blocking, utilizes the pfSense firewall engine for blocking an IP address. It does so by making a system call that results in placing the IP to be blocked into a pre-defined pf table called snort2c. This is the table @stephenw10 is asking you about. Even when you disable the Snort service, or if you remove the package, any IP addresses that have been previously added to that snort2c pf table remain there and the blocks will persist.

You can clear the table three ways: (1) when Snort is installed, go to the BLOCKS tab and click the button to remove all blocked hosts; (2) use the option under DIAGNOSTICS > TABLES in pfSense to display the snort2c table and remove any resident IP addresses; or (3) by rebooting the firewall. Rebooting clears the table of IP addresses because the table is purely a RAM construct.

BUT, I don't think Snort is your problem here. Way back up you showed a screenshot of the BLOCKS tab from Snort, and there were zero blocked hosts displayed. That means Snort is not blocking anything. It reads the snort2c table contents directly and displays them on that tab, so if the tab shows no blocks then the table is empty.

maddy_in65 · May 4, 2022, 3:18 AM

@stephenw10
There is no host entry in snort2c table.

login-to-view

stephenw10 · May 4, 2022, 10:23 AM

Mmm, not Snort then. It sure 'feels' like Snort though because it definitely blocks apt update traffic if you just enable all the signatures. I've seen that myself.

Something else is preventing it on that VLAN then. Is there anything else the traffic passes through? Filtering on an access point or similar?

Otherwise I would be looking at states and running packet captures to see what's actually happening.

Steve

dem · May 5, 2022, 4:50 PM

@maddy_in65 From what you've posted it seems like only outbound traffic to port 80 from the problem VLAN is failing. Maybe run grep ' 80 ' /tmp/rules.debug and look for something other than the standard "anti-lockout rule"?