Linux apt update/upgrade stopped working

maddy_in65 · May 2, 2022, 11:27 AM

@nogbadthebad
Yes I did it again but no luck. BTW I have not enabled Snort rule for Linux OS.

CrowFather · May 2, 2022, 2:09 PM

@maddy_in65 said in Linux apt update/upgrade stopped working:

46.235.231.151

Have you checked to make sure this IP is not blocked? What happens if you Ping it and what happens is you traceroute it?

maddy_in65 · May 2, 2022, 2:30 PM

@crowfather said in Linux apt update/upgrade stopped working:

@maddy_in65 said in Linux apt update/upgrade stopped working:

46.235.231.151

Have you checked to make sure this IP is not blocked? What happens if you Ping it and what happens is you traceroute it?

I have checked Firewall logs, but this IP is not mentioned here
Status>System Logs>Firewall

There is no entry in Snort blocks.
Also checked Pfblockerng>Report>Unified> Alters
There is no block here as well:

Ping and traceroute results

user@host:~ $ ping 46.235.231.151
PING 46.235.231.151 (46.235.231.151) 56(84) bytes of data.
64 bytes from 46.235.231.151: icmp_seq=1 ttl=46 time=206 ms
64 bytes from 46.235.231.151: icmp_seq=2 ttl=46 time=206 ms
64 bytes from 46.235.231.151: icmp_seq=3 ttl=46 time=206 ms
64 bytes from 46.235.231.151: icmp_seq=4 ttl=46 time=206 ms
64 bytes from 46.235.231.151: icmp_seq=5 ttl=46 time=206 ms
^C
--- 46.235.231.151 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 205.629/205.810/205.989/0.127 ms
user@host:~ $ traceroute 46.235.231.151
traceroute to 46.235.231.151 (46.235.231.151), 30 hops max, 60 byte packets
 1  103.134.162.24 (103.134.162.24)  1.611 ms 45.112.0.37 (45.112.0.37)  1.638 ms 103.134.162.24 (103.134.162.24)  1.230 ms
 2  * 45.112.0.33 (45.112.0.33)  2.448 ms  2.927 ms
 3  10.0.16.249 (10.0.16.249)  2.149 ms nsg-static-125.79.72.182.airtel.in (182.72.79.125)  2.816 ms 10.0.16.249 (10.0.16.249)  2.466 ms
 4  116.119.68.55 (116.119.68.55)  121.256 ms * 182.79.206.46 (182.79.206.46)  133.649 ms
 5  nsg-static-105.23.76.182-airtel.com (182.76.23.105)  11.560 ms lo.router-sov-a.mythic-beasts.com (93.93.133.0)  133.549 ms nsg-static-105.23.76.182-airtel.com (182.76.23.105)  11.291 ms
 6  lo.router-sov-b.mythic-beasts.com (93.93.133.1)  133.095 ms  132.357 ms 116.119.68.55 (116.119.68.55)  120.809 ms
 7  lo.router-cam-a.mythic-beasts.com (93.93.133.4)  125.473 ms  125.299 ms lo.router-sov-a.mythic-beasts.com (93.93.133.0)  195.915 ms
 8  lo.router-cam-b.mythic-beasts.com (93.93.133.5)  132.301 ms lo.router-sov-b.mythic-beasts.com (93.93.133.1)  197.807 ms lo.router-cam-b.mythic-beasts.com (93.93.133.5)  125.219 ms
 9  lo.router-cam-a.mythic-beasts.com (93.93.133.4)  190.699 ms lo.router-ams-b.mythic-beasts.com (93.93.133.9)  131.466 ms lo.router-cam-a.mythic-beasts.com (93.93.133.4)  199.026 ms
10  * lo.router-cam-b.mythic-beasts.com (93.93.133.5)  138.616 ms *
11  lo.router-ams-b.mythic-beasts.com (93.93.133.9)  126.750 ms *  133.806 ms
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

CrowFather · May 2, 2022, 2:44 PM

@maddy_in65 The packets are making it past the router but timing out at the end before they reach the end.

I've seen some posts about the raspbian archive being slow. Does this same thing happen when you use a mirror?

johnpoz · May 2, 2022, 2:44 PM

@maddy_in65 can you access it in your browser?

Either http or https

Also that archive.raspberrypi.org is a cname that points to 2 other fqdn, and resolves to many different IPs.

I have a couple of pis and not having any issues updating them via apt..

maddy_in65 · May 2, 2022, 2:59 PM

@johnpoz
Yes I can access it my main PC (Linux mint) which is on another VLAN. i dont have any issue with apt update on this PC.

Dig Command also gives me similar results.

user@host:~ $ dig archive.raspberrypi.org

; <<>> DiG 9.16.27-Raspbian <<>> archive.raspberrypi.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19692
;; flags: qr rd ra; QUERY: 1, ANSWER: 14, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;archive.raspberrypi.org.	IN	A

;; ANSWER SECTION:
archive.raspberrypi.org. 183	IN	CNAME	lb.raspberrypi.org.
lb.raspberrypi.org.	183	IN	CNAME	lb.raspberrypi.com.
lb.raspberrypi.com.	183	IN	A	176.126.240.167
lb.raspberrypi.com.	183	IN	A	46.235.231.145
lb.raspberrypi.com.	183	IN	A	46.235.230.122
lb.raspberrypi.com.	183	IN	A	46.235.231.151
lb.raspberrypi.com.	183	IN	A	93.93.135.141
lb.raspberrypi.com.	183	IN	A	93.93.135.117
lb.raspberrypi.com.	183	IN	A	176.126.240.86
lb.raspberrypi.com.	183	IN	A	46.235.227.39
lb.raspberrypi.com.	183	IN	A	93.93.135.118
lb.raspberrypi.com.	183	IN	A	176.126.240.84
lb.raspberrypi.com.	183	IN	A	46.235.231.111
lb.raspberrypi.com.	183	IN	A	93.93.130.212

;; Query time: 9 msec
;; SERVER: 192.168.11.1#53(192.168.11.1)
;; WHEN: Mon May 02 20:27:11 IST 2022
;; MSG SIZE  rcvd: 293

maddy_in65 · May 2, 2022, 3:01 PM

@crowfather
Not sure about this but zabbix repo is working fine but raspberry pi/rock64 repos are not working.

stephenw10 · May 2, 2022, 3:21 PM

Can you fetch the actual file it's looking for?

[2.6.0-RELEASE][admin@pfsensemirror.stevew.lan]/root: fetch http://raspbian.raspberrypi.org/raspbian/dists/bullseye/InRelease
InRelease                                               14 kB   13 MBps    00s

Steve

johnpoz · May 2, 2022, 5:16 PM

@maddy_in65 said in Linux apt update/upgrade stopped working:

which is on another VLAN. i dont have any issue with apt update on this PC.

Well what is different on your vlans? Different rules? Different dns? IPS, Proxy? Forcing traffic out a vpn? Different internet connection being used?

If your devices on another vlan works - then clearly its not a pfsense internet connection sort of thing.. So what is different about vlan that works and vlan that doesn't?
What is odd is your traceroute.. Why is pfsense not seen as your first hop?

user@host:~ $ traceroute 46.235.231.151
traceroute to 46.235.231.151 (46.235.231.151), 30 hops max, 60 byte packets
 1  103.134.162.24 (103.134.162.24)  1.611 ms 45.112.0.37 (45.112.0.37)  1.638 ms 103.134.162.24 (103.134.162.24)  1.230 ms
 2  * 45.112.0.33 (45.112.0.33)  2.448 ms  2.927 ms

Your pfsense lan side IP for this vlan is a 103.x.x.x address?

maddy_in65 · May 3, 2022, 4:37 AM

@stephenw10 said in Linux apt update/upgrade stopped working:

Can you fetch the actual file it's looking for?

[2.6.0-RELEASE][admin@pfsensemirror.stevew.lan]/root: fetch http://raspbian.raspberrypi.org/raspbian/dists/bullseye/InRelease
InRelease                                               14 kB   13 MBps    00s

Steve

This is working fine on pfsense

[2.6.0-RELEASE][admin@PfSense]/home: fetch http://raspbian.raspberrypi.org/raspbian/dists/bullseye/InRelease
InRelease                                               14 kB  118 kBps    00s
[2.6.0-RELEASE][admin@PfSense]/home: ls -lah
drwxr-xr-x   2 root  wheel     3B May  3 09:21 .
drwxr-xr-x  22 root  wheel    28B Feb 26 16:47 ..
-rw-r--r--   1 root  wheel    15K May  3 04:02 InReleas

maddy_in65 · May 3, 2022, 4:42 AM

@johnpoz said in Linux apt update/upgrade stopped working:

Well what is different on your vlans? Different rules? Different dns? IPS, Proxy? Forcing traffic out a vpn? Different internet connection being used?

There is no separate DNS , IPS. All VLANs uses same resources. No Proxy, no VPN in my network.
I am having two internet connections and each VLAN is using load balancer gateway.

If your devices on another vlan works - then clearly its not a pfsense internet connection sort of thing.. So what is different about vlan that works and vlan that doesn't?

I am not sure why this is not working in server VLAN.

What is odd is your traceroute.. Why is pfsense not seen as your first hop?

No Idea why it is using ISP gateway as first hop. I have checked traceroute from my desktop system and it is showing similar results.

user@host:~ $ traceroute 46.235.231.151
traceroute to 46.235.231.151 (46.235.231.151), 30 hops max, 60 byte packets
1 103.134.162.24 (103.134.162.24) 1.611 ms 45.112.0.37 (45.112.0.37) 1.638 ms 103.134.162.24 (103.134.162.24) 1.230 ms
2 * 45.112.0.33 (45.112.0.33) 2.448 ms 2.927 ms

Your pfsense lan side IP for this vlan is a 103.x.x.x address?
No that is ISP gateway. my LAN side IP of this VLAN is 192.168.11.0/24

johnpoz · May 3, 2022, 11:23 AM

@maddy_in65 does where it work have IPv6 or not? Could see failing on IPv6 in your output try forcing it to just use IPv4

apt-get -o Acquire::ForceIPv4=true update

maddy_in65 · May 3, 2022, 12:54 PM

@johnpoz said in Linux apt update/upgrade stopped working:

@maddy_in65 does where it work have IPv6 or not? Could see failing on IPv6 in your output try forcing it to just use IPv4
apt-get -o Acquire::ForceIPv4=true update

I have IPv6 blocked in pfsense.

I have tested forcing apt-get Ipv4 but no luck. Checked on all 3 SBCs and its same issue. Docker/Zabbix/mongodb all these repos work fine but raspberry, port.ubuntu, launchpad all these don't work.
I am really banging my head against desk since last week to get this worked. This is really frustrating.

Get:1 https://repo.zabbix.com/zabbix/6.0/raspbian bullseye InRelease [7,126 B]                                                    
Get:2 https://repo.zabbix.com/zabbix/6.0/raspbian bullseye/main Sources [1,231 B]                                     
Get:3 https://repo.zabbix.com/zabbix/6.0/raspbian bullseye/main armhf Packages [5,133 B]                                                       
Err:4 http://raspbian.raspberrypi.org/raspbian bullseye InRelease                                                                                                   
  Could not connect to raspbian.raspberrypi.org:80 (93.93.128.193), connection timed out
Err:5 http://archive.raspberrypi.org/debian bullseye InRelease                                                      
  Could not connect to archive.raspberrypi.org:80 (46.235.231.145), connection timed out Could not connect to archive.raspberrypi.org:80 (93.93.130.212), connection timed out Could not connect to archive.raspberrypi.org:80 (93.93.135.118), connection timed out Could not connect to archive.raspberrypi.org:80 (176.126.240.84), connection timed out Could not connect to archive.raspberrypi.org:80 (46.235.231.111), connection timed out Could not connect to archive.raspberrypi.org:80 (93.93.135.141), connection timed out Could not connect to archive.raspberrypi.org:80 (46.235.227.39), connection timed out Could not connect to archive.raspberrypi.org:80 (46.235.231.151), connection timed out Could not connect to archive.raspberrypi.org:80 (176.126.240.167), connection timed out Could not connect to archive.raspberrypi.org:80 (93.93.135.117), connection timed out Could not connect to archive.raspberrypi.org:80 (46.235.230.122), connection timed out Could not connect to archive.raspberrypi.org:80 (176.126.240.86), connection timed out
Fetched 13.5 kB in 34s (392 B/s)                           
Reading package lists... Done
W: Failed to fetch http://raspbian.raspberrypi.org/raspbian/dists/bullseye/InRelease  Could not connect to raspbian.raspberrypi.org:80 (93.93.128.193), connection timed out

Rock64:

Get:1 https://download.docker.com/linux/debian buster InRelease [54.0 kB]
Hit:2 https://download.docker.com/linux/ubuntu focal InRelease                                                    
Ign:3 https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 InRelease                        
Hit:4 https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 Release                                                                       
Err:6 http://apt.armbian.com/region/EU focal InRelease                                                                                                           
  Could not connect to apt.armbian.com:80 (107.175.134.226), connection timed out
Err:7 http://ports.ubuntu.com focal InRelease

johnpoz · May 3, 2022, 1:01 PM

@maddy_in65 said in Linux apt update/upgrade stopped working:

I have IPv6 blocked in pfsense.

But from your output your devices were trying to use IPv6..

Your saying nothing is different between these vlans... You can call up the website (from this vlan?)

Pfsense unless you have different rules, or using IPS - pfsense doesn't know any different between lan or vlan X.. Just a different IP, that is natted to the same public IP, etc.

Could not connect to raspbian.raspberrypi.org:80 (93.93.128.193)

But your saying browser can go there - there is no difference to pfsense..

pfsense just knows traffic to that IP on port 80.. So why it works via browser, and not via your pi box and apt.. Not sure how that is a pfsense thing.. The only thing that could make sense is some sort of IPS filtering that doesn't like something different in the packets. But you say your not doing that..

But to base pfsense traffic to IP X on port Y is just that.. It doesn't know if browser or apt, or whatever it just knows IP and port - allowed or not allowed, oh nat this traffic to my wan IP..

So sniff on pfsense - you see the traffic to IP:80, do you see that go out the wan.. If so pfsense did its job.. If you see hit pfsense lan side interface but not go out.. Then you know your filtering when you think your not, IPS is only thing I could see filtering apt but not browser.

stephenw10 · May 3, 2022, 12:58 PM

Look for states to one of those IPs that's failing. Make sure you have states on WAN and the VLAN in question.

Get a packet capture on WAN for the same IP. See what's actually happening there, are any packets coming back at all?

Steve

maddy_in65 · May 3, 2022, 1:01 PM

@johnpoz said in Linux apt update/upgrade stopped working:

@maddy_in65 said in Linux apt update/upgrade stopped working:

I have IPv6 blocked in pfsense.

But from your output your devices were trying to use IPv6..

Your saying nothing is different between these vlans... You can call up the website (from this vlan?)

I put my laptop in server VLAN and guess what it also has same issue. Apt update stopped working.

Hit:1 https://packages.microsoft.com/repos/ms-teams stable InRelease           
Hit:2 https://repo.skype.com/deb stable InRelease                              
Err:3 http://dl.google.com/linux/chrome/deb stable InRelease                                                            
  Cannot initiate the connection to dl.google.com:80 (2404:6800:4009:812::200e). - connect (101: Network is unreachable) Could not connect to dl.google.com:80 (216.58.203.14), connection timed out
Err:4 http://ppa.launchpad.net/inkscape.dev/stable/ubuntu focal InRelease                                               
  Cannot initiate the connection to ppa.launchpad.net:80 (2001:67c:1560:8008::19). - connect (101: Network is unreachable) Could not connect to ppa.launchpad.net:80 (91.189.95.85), connection timed out
Err:5 http://ppa.launchpad.net/tomtomtom/woeusb/ubuntu focal InRelease                                                  
  Cannot initiate the connection to ppa.launchpad.net:80 (2001:67c:1560:8008::19). - connect (101: Network is unreachable)
Err:6 http://packages.linuxmint.com una InRelease                                                                       
  Could not connect to packages.linuxmint.com:80 (208.77.20.11), connection timed out Could not connect to packages.linuxmint.com:80 (68.235.39.11), connection timed out

While the browsing is working fine, in fact I am posting this from same laptop in same VLAN

Pfsense unless you have different rules, or using IPS - pfsense doesn't know any different between lan or vlan X.. Just a different IP, that is natted to the same public IP, etc.
pfsense just knows traffic to that IP on port 80.. So why it works via browser, and not via your pi box and apt.. Not sure how that is a pfsense thing.. The only thing that could make sense is some sort of IPS filtering that doesn't like something different in the packets. But you say your not doing that..

But to base pfsense traffic to IP X on port Y is just that.. It doesn't know if browser or apt, or whatever it just knows IP and port - allowed or not allowed, oh nat this traffic to my wan IP..

So sniff on pfsense - you see the traffic to IP:80, do you see that go out the wan.. If so pfsense did its job.. If you see hit pfsense lan side interface but not go out.. Then you know your filtering when you think your not, IPS is only thing I could see filtering apt but not browser.

I can access repo website and I can see connection in states

HOME_SERVERS 	tcp 	192.168.11.4:34742 -> 46.235.231.111:443 	CLOSING:ESTABLISHED 	13 / 16 	2 KiB / 2 KiB

As you suggested IPS may be blocking apt update, Can i test by complete removal of Snort.
If yes what would be best way to complete removal of Snort

stephenw10 · May 3, 2022, 1:07 PM

Just disabling it will be sufficient. Or even just put it in non-blocking mode.

Just make sure to clear any existing blocks.

Check Diag > Tables. Make sure the snort2c table is empty.

Steve

maddy_in65 · May 3, 2022, 3:28 PM

@stephenw10
Tested apt updates by disabling & uninstalling Snort but no luck

stephenw10 · May 3, 2022, 4:02 PM

But is the block still in the table?

bmeeks · May 4, 2022, 12:44 AM

@maddy_in65 said in Linux apt update/upgrade stopped working:

@stephenw10
Tested apt updates by disabling & uninstalling Snort but no luck

Snort (and Suricata), when running in Legacy Mode Blocking, utilizes the pfSense firewall engine for blocking an IP address. It does so by making a system call that results in placing the IP to be blocked into a pre-defined pf table called snort2c. This is the table @stephenw10 is asking you about. Even when you disable the Snort service, or if you remove the package, any IP addresses that have been previously added to that snort2c pf table remain there and the blocks will persist.

You can clear the table three ways: (1) when Snort is installed, go to the BLOCKS tab and click the button to remove all blocked hosts; (2) use the option under DIAGNOSTICS > TABLES in pfSense to display the snort2c table and remove any resident IP addresses; or (3) by rebooting the firewall. Rebooting clears the table of IP addresses because the table is purely a RAM construct.

BUT, I don't think Snort is your problem here. Way back up you showed a screenshot of the BLOCKS tab from Snort, and there were zero blocked hosts displayed. That means Snort is not blocking anything. It reads the snort2c table contents directly and displays them on that tab, so if the tab shows no blocks then the table is empty.