Linux apt update/upgrade stopped working
-
I am having 3 SBc (1 Raspi/2 Rock64) which are running few apps. for home use. The Apt/update/upgrade was stopped working post an pfsense upgrade to 2.6.0.
i had faced similar problem earlier (pfsense 2.5.2) but I have added aliases in snort & floating FW rules. Now it seems it stopped working.I have checked System/pfblocker Logs but none shows any related IP blocked.
Any idea what is going wrong here. Please help.Failed to fetch http://raspbian.raspberrypi.org/raspbian/dists/bullseye/InRelease Cannot initiate the connection to raspbian.raspberrypi.org:80 (2a00:1098:0:80:1000:75:0:3). - connect (101: Network is unreachable) Could not connect to raspbian.raspberrypi.org:80 (93.93.128.193), connection timed out connection timed out Could not connect to archive.raspberrypi.org:80 (46.235.231.151), connection timed out Could not connect to archive.raspberrypi.org:80 (93.93.135.117), connection timed out Could not connect to archive.raspberrypi.org:80 (176.126.240.167), connection timed out Could not connect to archive.raspberrypi.org:80 (46.235.231.111),Troubleshooting steps:
- Disabled SNORT/Pfblockerng but it didn't work.
I can ping to internet servers (e.g. Google/cloudflare DNS).
DNS seems working fine.
I have Connected all 3 SBCs to my friend's home network yesterday and apt update worked fine. This means something is blocking on pfsense.
nslookup install archive.raspberrypi.org Server: archive.raspberrypi.orgAddress: 46.235.230.122#53 nslookup ppa.launchpad.net Non-authoritative answer:Name: ppa.launchpad.net Address: 91.189.95.85 Name: ppa.launchpad.net Address: 2001:67c:1560:8008::19 ping 1.1.1.1 PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data. 64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=4.74 ms 64 bytes from 1.1.1.1: icmp_seq=2 ttl=57 time=4.87 ms 64 bytes from 1.1.1.1: icmp_seq=3 ttl=57 time=5.10 ms 64 bytes from 1.1.1.1: icmp_seq=4 ttl=57 time=4.66 ms 64 bytes from 1.1.1.1: icmp_seq=5 ttl=57 time=4.89 ms ^C --- 1.1.1.1 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4006ms rtt min/avg/max/mdev = 4.661/4.851/5.099/0.148 ms dig google.com ; <<>> DiG 9.16.8-Ubuntu <<>> google.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55702 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;google.com. IN A ;; ANSWER SECTION: google.com. 300 IN A 142.250.76.174 ;; Query time: 215 msec ;; SERVER: 192.168.11.1#53(192.168.11.1) - Disabled SNORT/Pfblockerng but it didn't work.
-
@maddy_in65 The following ET-Policy rule maybe:-
ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management
Did you clear the blocked hosts after disabling SNORT?
-
@nogbadthebad
Yes I did it again but no luck. BTW I have not enabled Snort rule for Linux OS.
-
@maddy_in65 said in Linux apt update/upgrade stopped working:
46.235.231.151
Have you checked to make sure this IP is not blocked? What happens if you Ping it and what happens is you traceroute it?
-
@crowfather said in Linux apt update/upgrade stopped working:
@maddy_in65 said in Linux apt update/upgrade stopped working:
46.235.231.151
Have you checked to make sure this IP is not blocked? What happens if you Ping it and what happens is you traceroute it?
I have checked Firewall logs, but this IP is not mentioned here
Status>System Logs>Firewall
There is no entry in Snort blocks.
Also checked Pfblockerng>Report>Unified> Alters
There is no block here as well:
Ping and traceroute resultsuser@host:~ $ ping 46.235.231.151 PING 46.235.231.151 (46.235.231.151) 56(84) bytes of data. 64 bytes from 46.235.231.151: icmp_seq=1 ttl=46 time=206 ms 64 bytes from 46.235.231.151: icmp_seq=2 ttl=46 time=206 ms 64 bytes from 46.235.231.151: icmp_seq=3 ttl=46 time=206 ms 64 bytes from 46.235.231.151: icmp_seq=4 ttl=46 time=206 ms 64 bytes from 46.235.231.151: icmp_seq=5 ttl=46 time=206 ms ^C --- 46.235.231.151 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4006ms rtt min/avg/max/mdev = 205.629/205.810/205.989/0.127 ms user@host:~ $ traceroute 46.235.231.151 traceroute to 46.235.231.151 (46.235.231.151), 30 hops max, 60 byte packets 1 103.134.162.24 (103.134.162.24) 1.611 ms 45.112.0.37 (45.112.0.37) 1.638 ms 103.134.162.24 (103.134.162.24) 1.230 ms 2 * 45.112.0.33 (45.112.0.33) 2.448 ms 2.927 ms 3 10.0.16.249 (10.0.16.249) 2.149 ms nsg-static-125.79.72.182.airtel.in (182.72.79.125) 2.816 ms 10.0.16.249 (10.0.16.249) 2.466 ms 4 116.119.68.55 (116.119.68.55) 121.256 ms * 182.79.206.46 (182.79.206.46) 133.649 ms 5 nsg-static-105.23.76.182-airtel.com (182.76.23.105) 11.560 ms lo.router-sov-a.mythic-beasts.com (93.93.133.0) 133.549 ms nsg-static-105.23.76.182-airtel.com (182.76.23.105) 11.291 ms 6 lo.router-sov-b.mythic-beasts.com (93.93.133.1) 133.095 ms 132.357 ms 116.119.68.55 (116.119.68.55) 120.809 ms 7 lo.router-cam-a.mythic-beasts.com (93.93.133.4) 125.473 ms 125.299 ms lo.router-sov-a.mythic-beasts.com (93.93.133.0) 195.915 ms 8 lo.router-cam-b.mythic-beasts.com (93.93.133.5) 132.301 ms lo.router-sov-b.mythic-beasts.com (93.93.133.1) 197.807 ms lo.router-cam-b.mythic-beasts.com (93.93.133.5) 125.219 ms 9 lo.router-cam-a.mythic-beasts.com (93.93.133.4) 190.699 ms lo.router-ams-b.mythic-beasts.com (93.93.133.9) 131.466 ms lo.router-cam-a.mythic-beasts.com (93.93.133.4) 199.026 ms 10 * lo.router-cam-b.mythic-beasts.com (93.93.133.5) 138.616 ms * 11 lo.router-ams-b.mythic-beasts.com (93.93.133.9) 126.750 ms * 133.806 ms 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * -
@maddy_in65 The packets are making it past the router but timing out at the end before they reach the end.
I've seen some posts about the raspbian archive being slow. Does this same thing happen when you use a mirror?
-
@maddy_in65 can you access it in your browser?
Either http or https
Also that archive.raspberrypi.org is a cname that points to 2 other fqdn, and resolves to many different IPs.
I have a couple of pis and not having any issues updating them via apt..
-
@johnpoz
Yes I can access it my main PC (Linux mint) which is on another VLAN. i dont have any issue with apt update on this PC.
Dig Command also gives me similar results.user@host:~ $ dig archive.raspberrypi.org ; <<>> DiG 9.16.27-Raspbian <<>> archive.raspberrypi.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19692 ;; flags: qr rd ra; QUERY: 1, ANSWER: 14, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;archive.raspberrypi.org. IN A ;; ANSWER SECTION: archive.raspberrypi.org. 183 IN CNAME lb.raspberrypi.org. lb.raspberrypi.org. 183 IN CNAME lb.raspberrypi.com. lb.raspberrypi.com. 183 IN A 176.126.240.167 lb.raspberrypi.com. 183 IN A 46.235.231.145 lb.raspberrypi.com. 183 IN A 46.235.230.122 lb.raspberrypi.com. 183 IN A 46.235.231.151 lb.raspberrypi.com. 183 IN A 93.93.135.141 lb.raspberrypi.com. 183 IN A 93.93.135.117 lb.raspberrypi.com. 183 IN A 176.126.240.86 lb.raspberrypi.com. 183 IN A 46.235.227.39 lb.raspberrypi.com. 183 IN A 93.93.135.118 lb.raspberrypi.com. 183 IN A 176.126.240.84 lb.raspberrypi.com. 183 IN A 46.235.231.111 lb.raspberrypi.com. 183 IN A 93.93.130.212 ;; Query time: 9 msec ;; SERVER: 192.168.11.1#53(192.168.11.1) ;; WHEN: Mon May 02 20:27:11 IST 2022 ;; MSG SIZE rcvd: 293 -
@crowfather
Not sure about this but zabbix repo is working fine but raspberry pi/rock64 repos are not working.
-
Can you fetch the actual file it's looking for?
[2.6.0-RELEASE][admin@pfsensemirror.stevew.lan]/root: fetch http://raspbian.raspberrypi.org/raspbian/dists/bullseye/InRelease InRelease 14 kB 13 MBps 00sSteve
-
@maddy_in65 said in Linux apt update/upgrade stopped working:
which is on another VLAN. i dont have any issue with apt update on this PC.
Well what is different on your vlans? Different rules? Different dns? IPS, Proxy? Forcing traffic out a vpn? Different internet connection being used?
If your devices on another vlan works - then clearly its not a pfsense internet connection sort of thing.. So what is different about vlan that works and vlan that doesn't?
What is odd is your traceroute.. Why is pfsense not seen as your first hop?user@host:~ $ traceroute 46.235.231.151 traceroute to 46.235.231.151 (46.235.231.151), 30 hops max, 60 byte packets 1 103.134.162.24 (103.134.162.24) 1.611 ms 45.112.0.37 (45.112.0.37) 1.638 ms 103.134.162.24 (103.134.162.24) 1.230 ms 2 * 45.112.0.33 (45.112.0.33) 2.448 ms 2.927 msYour pfsense lan side IP for this vlan is a 103.x.x.x address?
-
@stephenw10 said in Linux apt update/upgrade stopped working:
Can you fetch the actual file it's looking for?
[2.6.0-RELEASE][admin@pfsensemirror.stevew.lan]/root: fetch http://raspbian.raspberrypi.org/raspbian/dists/bullseye/InRelease InRelease 14 kB 13 MBps 00sSteve
This is working fine on pfsense
[2.6.0-RELEASE][admin@PfSense]/home: fetch http://raspbian.raspberrypi.org/raspbian/dists/bullseye/InRelease InRelease 14 kB 118 kBps 00s [2.6.0-RELEASE][admin@PfSense]/home: ls -lah drwxr-xr-x 2 root wheel 3B May 3 09:21 . drwxr-xr-x 22 root wheel 28B Feb 26 16:47 .. -rw-r--r-- 1 root wheel 15K May 3 04:02 InReleas -
@johnpoz said in Linux apt update/upgrade stopped working:
Well what is different on your vlans? Different rules? Different dns? IPS, Proxy? Forcing traffic out a vpn? Different internet connection being used?
There is no separate DNS , IPS. All VLANs uses same resources. No Proxy, no VPN in my network.
I am having two internet connections and each VLAN is using load balancer gateway.If your devices on another vlan works - then clearly its not a pfsense internet connection sort of thing.. So what is different about vlan that works and vlan that doesn't?
I am not sure why this is not working in server VLAN.
What is odd is your traceroute.. Why is pfsense not seen as your first hop?
No Idea why it is using ISP gateway as first hop. I have checked traceroute from my desktop system and it is showing similar results.
user@host:~ $ traceroute 46.235.231.151
traceroute to 46.235.231.151 (46.235.231.151), 30 hops max, 60 byte packets
1 103.134.162.24 (103.134.162.24) 1.611 ms 45.112.0.37 (45.112.0.37) 1.638 ms 103.134.162.24 (103.134.162.24) 1.230 ms
2 * 45.112.0.33 (45.112.0.33) 2.448 ms 2.927 msYour pfsense lan side IP for this vlan is a 103.x.x.x address?
No that is ISP gateway. my LAN side IP of this VLAN is 192.168.11.0/24 -
@maddy_in65 does where it work have IPv6 or not? Could see failing on IPv6 in your output try forcing it to just use IPv4
apt-get -o Acquire::ForceIPv4=true update -
@johnpoz said in Linux apt update/upgrade stopped working:
@maddy_in65 does where it work have IPv6 or not? Could see failing on IPv6 in your output try forcing it to just use IPv4
apt-get -o Acquire::ForceIPv4=true updateI have IPv6 blocked in pfsense.
I have tested forcing apt-get Ipv4 but no luck. Checked on all 3 SBCs and its same issue. Docker/Zabbix/mongodb all these repos work fine but raspberry, port.ubuntu, launchpad all these don't work.
I am really banging my head against desk since last week to get this worked. This is really frustrating.Get:1 https://repo.zabbix.com/zabbix/6.0/raspbian bullseye InRelease [7,126 B] Get:2 https://repo.zabbix.com/zabbix/6.0/raspbian bullseye/main Sources [1,231 B] Get:3 https://repo.zabbix.com/zabbix/6.0/raspbian bullseye/main armhf Packages [5,133 B] Err:4 http://raspbian.raspberrypi.org/raspbian bullseye InRelease Could not connect to raspbian.raspberrypi.org:80 (93.93.128.193), connection timed out Err:5 http://archive.raspberrypi.org/debian bullseye InRelease Could not connect to archive.raspberrypi.org:80 (46.235.231.145), connection timed out Could not connect to archive.raspberrypi.org:80 (93.93.130.212), connection timed out Could not connect to archive.raspberrypi.org:80 (93.93.135.118), connection timed out Could not connect to archive.raspberrypi.org:80 (176.126.240.84), connection timed out Could not connect to archive.raspberrypi.org:80 (46.235.231.111), connection timed out Could not connect to archive.raspberrypi.org:80 (93.93.135.141), connection timed out Could not connect to archive.raspberrypi.org:80 (46.235.227.39), connection timed out Could not connect to archive.raspberrypi.org:80 (46.235.231.151), connection timed out Could not connect to archive.raspberrypi.org:80 (176.126.240.167), connection timed out Could not connect to archive.raspberrypi.org:80 (93.93.135.117), connection timed out Could not connect to archive.raspberrypi.org:80 (46.235.230.122), connection timed out Could not connect to archive.raspberrypi.org:80 (176.126.240.86), connection timed out Fetched 13.5 kB in 34s (392 B/s) Reading package lists... Done W: Failed to fetch http://raspbian.raspberrypi.org/raspbian/dists/bullseye/InRelease Could not connect to raspbian.raspberrypi.org:80 (93.93.128.193), connection timed outRock64:
Get:1 https://download.docker.com/linux/debian buster InRelease [54.0 kB] Hit:2 https://download.docker.com/linux/ubuntu focal InRelease Ign:3 https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 InRelease Hit:4 https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 Release Err:6 http://apt.armbian.com/region/EU focal InRelease Could not connect to apt.armbian.com:80 (107.175.134.226), connection timed out Err:7 http://ports.ubuntu.com focal InRelease -
@maddy_in65 said in Linux apt update/upgrade stopped working:
I have IPv6 blocked in pfsense.
But from your output your devices were trying to use IPv6..
Your saying nothing is different between these vlans... You can call up the website (from this vlan?)
Pfsense unless you have different rules, or using IPS - pfsense doesn't know any different between lan or vlan X.. Just a different IP, that is natted to the same public IP, etc.
Could not connect to raspbian.raspberrypi.org:80 (93.93.128.193)But your saying browser can go there - there is no difference to pfsense..
pfsense just knows traffic to that IP on port 80.. So why it works via browser, and not via your pi box and apt.. Not sure how that is a pfsense thing.. The only thing that could make sense is some sort of IPS filtering that doesn't like something different in the packets. But you say your not doing that..
But to base pfsense traffic to IP X on port Y is just that.. It doesn't know if browser or apt, or whatever it just knows IP and port - allowed or not allowed, oh nat this traffic to my wan IP..
So sniff on pfsense - you see the traffic to IP:80, do you see that go out the wan.. If so pfsense did its job.. If you see hit pfsense lan side interface but not go out.. Then you know your filtering when you think your not, IPS is only thing I could see filtering apt but not browser.
-
Look for states to one of those IPs that's failing. Make sure you have states on WAN and the VLAN in question.
Get a packet capture on WAN for the same IP. See what's actually happening there, are any packets coming back at all?
Steve
-
@johnpoz said in Linux apt update/upgrade stopped working:
@maddy_in65 said in Linux apt update/upgrade stopped working:
I have IPv6 blocked in pfsense.
But from your output your devices were trying to use IPv6..
Your saying nothing is different between these vlans... You can call up the website (from this vlan?)
I put my laptop in server VLAN and guess what it also has same issue. Apt update stopped working.
Hit:1 https://packages.microsoft.com/repos/ms-teams stable InRelease Hit:2 https://repo.skype.com/deb stable InRelease Err:3 http://dl.google.com/linux/chrome/deb stable InRelease Cannot initiate the connection to dl.google.com:80 (2404:6800:4009:812::200e). - connect (101: Network is unreachable) Could not connect to dl.google.com:80 (216.58.203.14), connection timed out Err:4 http://ppa.launchpad.net/inkscape.dev/stable/ubuntu focal InRelease Cannot initiate the connection to ppa.launchpad.net:80 (2001:67c:1560:8008::19). - connect (101: Network is unreachable) Could not connect to ppa.launchpad.net:80 (91.189.95.85), connection timed out Err:5 http://ppa.launchpad.net/tomtomtom/woeusb/ubuntu focal InRelease Cannot initiate the connection to ppa.launchpad.net:80 (2001:67c:1560:8008::19). - connect (101: Network is unreachable) Err:6 http://packages.linuxmint.com una InRelease Could not connect to packages.linuxmint.com:80 (208.77.20.11), connection timed out Could not connect to packages.linuxmint.com:80 (68.235.39.11), connection timed outWhile the browsing is working fine, in fact I am posting this from same laptop in same VLAN
Pfsense unless you have different rules, or using IPS - pfsense doesn't know any different between lan or vlan X.. Just a different IP, that is natted to the same public IP, etc.
pfsense just knows traffic to that IP on port 80.. So why it works via browser, and not via your pi box and apt.. Not sure how that is a pfsense thing.. The only thing that could make sense is some sort of IPS filtering that doesn't like something different in the packets. But you say your not doing that..But to base pfsense traffic to IP X on port Y is just that.. It doesn't know if browser or apt, or whatever it just knows IP and port - allowed or not allowed, oh nat this traffic to my wan IP..
So sniff on pfsense - you see the traffic to IP:80, do you see that go out the wan.. If so pfsense did its job.. If you see hit pfsense lan side interface but not go out.. Then you know your filtering when you think your not, IPS is only thing I could see filtering apt but not browser.
I can access repo website and I can see connection in states
HOME_SERVERS tcp 192.168.11.4:34742 -> 46.235.231.111:443 CLOSING:ESTABLISHED 13 / 16 2 KiB / 2 KiBAs you suggested IPS may be blocking apt update, Can i test by complete removal of Snort.
If yes what would be best way to complete removal of Snort -
Just disabling it will be sufficient. Or even just put it in non-blocking mode.
Just make sure to clear any existing blocks.
Check Diag > Tables. Make sure the snort2c table is empty.
Steve
-
@stephenw10
Tested apt updates by disabling & uninstalling Snort but no luck
