Linux apt update/upgrade stopped working
-
@stephenw10 said in Linux apt update/upgrade stopped working:
Can you fetch the actual file it's looking for?
[2.6.0-RELEASE][admin@pfsensemirror.stevew.lan]/root: fetch http://raspbian.raspberrypi.org/raspbian/dists/bullseye/InRelease InRelease 14 kB 13 MBps 00sSteve
This is working fine on pfsense
[2.6.0-RELEASE][admin@PfSense]/home: fetch http://raspbian.raspberrypi.org/raspbian/dists/bullseye/InRelease InRelease 14 kB 118 kBps 00s [2.6.0-RELEASE][admin@PfSense]/home: ls -lah drwxr-xr-x 2 root wheel 3B May 3 09:21 . drwxr-xr-x 22 root wheel 28B Feb 26 16:47 .. -rw-r--r-- 1 root wheel 15K May 3 04:02 InReleas -
@johnpoz said in Linux apt update/upgrade stopped working:
Well what is different on your vlans? Different rules? Different dns? IPS, Proxy? Forcing traffic out a vpn? Different internet connection being used?
There is no separate DNS , IPS. All VLANs uses same resources. No Proxy, no VPN in my network.
I am having two internet connections and each VLAN is using load balancer gateway.If your devices on another vlan works - then clearly its not a pfsense internet connection sort of thing.. So what is different about vlan that works and vlan that doesn't?
I am not sure why this is not working in server VLAN.
What is odd is your traceroute.. Why is pfsense not seen as your first hop?
No Idea why it is using ISP gateway as first hop. I have checked traceroute from my desktop system and it is showing similar results.
user@host:~ $ traceroute 46.235.231.151
traceroute to 46.235.231.151 (46.235.231.151), 30 hops max, 60 byte packets
1 103.134.162.24 (103.134.162.24) 1.611 ms 45.112.0.37 (45.112.0.37) 1.638 ms 103.134.162.24 (103.134.162.24) 1.230 ms
2 * 45.112.0.33 (45.112.0.33) 2.448 ms 2.927 msYour pfsense lan side IP for this vlan is a 103.x.x.x address?
No that is ISP gateway. my LAN side IP of this VLAN is 192.168.11.0/24 -
@maddy_in65 does where it work have IPv6 or not? Could see failing on IPv6 in your output try forcing it to just use IPv4
apt-get -o Acquire::ForceIPv4=true update -
@johnpoz said in Linux apt update/upgrade stopped working:
@maddy_in65 does where it work have IPv6 or not? Could see failing on IPv6 in your output try forcing it to just use IPv4
apt-get -o Acquire::ForceIPv4=true updateI have IPv6 blocked in pfsense.
I have tested forcing apt-get Ipv4 but no luck. Checked on all 3 SBCs and its same issue. Docker/Zabbix/mongodb all these repos work fine but raspberry, port.ubuntu, launchpad all these don't work.
I am really banging my head against desk since last week to get this worked. This is really frustrating.Get:1 https://repo.zabbix.com/zabbix/6.0/raspbian bullseye InRelease [7,126 B] Get:2 https://repo.zabbix.com/zabbix/6.0/raspbian bullseye/main Sources [1,231 B] Get:3 https://repo.zabbix.com/zabbix/6.0/raspbian bullseye/main armhf Packages [5,133 B] Err:4 http://raspbian.raspberrypi.org/raspbian bullseye InRelease Could not connect to raspbian.raspberrypi.org:80 (93.93.128.193), connection timed out Err:5 http://archive.raspberrypi.org/debian bullseye InRelease Could not connect to archive.raspberrypi.org:80 (46.235.231.145), connection timed out Could not connect to archive.raspberrypi.org:80 (93.93.130.212), connection timed out Could not connect to archive.raspberrypi.org:80 (93.93.135.118), connection timed out Could not connect to archive.raspberrypi.org:80 (176.126.240.84), connection timed out Could not connect to archive.raspberrypi.org:80 (46.235.231.111), connection timed out Could not connect to archive.raspberrypi.org:80 (93.93.135.141), connection timed out Could not connect to archive.raspberrypi.org:80 (46.235.227.39), connection timed out Could not connect to archive.raspberrypi.org:80 (46.235.231.151), connection timed out Could not connect to archive.raspberrypi.org:80 (176.126.240.167), connection timed out Could not connect to archive.raspberrypi.org:80 (93.93.135.117), connection timed out Could not connect to archive.raspberrypi.org:80 (46.235.230.122), connection timed out Could not connect to archive.raspberrypi.org:80 (176.126.240.86), connection timed out Fetched 13.5 kB in 34s (392 B/s) Reading package lists... Done W: Failed to fetch http://raspbian.raspberrypi.org/raspbian/dists/bullseye/InRelease Could not connect to raspbian.raspberrypi.org:80 (93.93.128.193), connection timed outRock64:
Get:1 https://download.docker.com/linux/debian buster InRelease [54.0 kB] Hit:2 https://download.docker.com/linux/ubuntu focal InRelease Ign:3 https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 InRelease Hit:4 https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 Release Err:6 http://apt.armbian.com/region/EU focal InRelease Could not connect to apt.armbian.com:80 (107.175.134.226), connection timed out Err:7 http://ports.ubuntu.com focal InRelease -
@maddy_in65 said in Linux apt update/upgrade stopped working:
I have IPv6 blocked in pfsense.
But from your output your devices were trying to use IPv6..
Your saying nothing is different between these vlans... You can call up the website (from this vlan?)
Pfsense unless you have different rules, or using IPS - pfsense doesn't know any different between lan or vlan X.. Just a different IP, that is natted to the same public IP, etc.
Could not connect to raspbian.raspberrypi.org:80 (93.93.128.193)But your saying browser can go there - there is no difference to pfsense..
pfsense just knows traffic to that IP on port 80.. So why it works via browser, and not via your pi box and apt.. Not sure how that is a pfsense thing.. The only thing that could make sense is some sort of IPS filtering that doesn't like something different in the packets. But you say your not doing that..
But to base pfsense traffic to IP X on port Y is just that.. It doesn't know if browser or apt, or whatever it just knows IP and port - allowed or not allowed, oh nat this traffic to my wan IP..
So sniff on pfsense - you see the traffic to IP:80, do you see that go out the wan.. If so pfsense did its job.. If you see hit pfsense lan side interface but not go out.. Then you know your filtering when you think your not, IPS is only thing I could see filtering apt but not browser.
-
Look for states to one of those IPs that's failing. Make sure you have states on WAN and the VLAN in question.
Get a packet capture on WAN for the same IP. See what's actually happening there, are any packets coming back at all?
Steve
-
@johnpoz said in Linux apt update/upgrade stopped working:
@maddy_in65 said in Linux apt update/upgrade stopped working:
I have IPv6 blocked in pfsense.
But from your output your devices were trying to use IPv6..
Your saying nothing is different between these vlans... You can call up the website (from this vlan?)
I put my laptop in server VLAN and guess what it also has same issue. Apt update stopped working.
Hit:1 https://packages.microsoft.com/repos/ms-teams stable InRelease Hit:2 https://repo.skype.com/deb stable InRelease Err:3 http://dl.google.com/linux/chrome/deb stable InRelease Cannot initiate the connection to dl.google.com:80 (2404:6800:4009:812::200e). - connect (101: Network is unreachable) Could not connect to dl.google.com:80 (216.58.203.14), connection timed out Err:4 http://ppa.launchpad.net/inkscape.dev/stable/ubuntu focal InRelease Cannot initiate the connection to ppa.launchpad.net:80 (2001:67c:1560:8008::19). - connect (101: Network is unreachable) Could not connect to ppa.launchpad.net:80 (91.189.95.85), connection timed out Err:5 http://ppa.launchpad.net/tomtomtom/woeusb/ubuntu focal InRelease Cannot initiate the connection to ppa.launchpad.net:80 (2001:67c:1560:8008::19). - connect (101: Network is unreachable) Err:6 http://packages.linuxmint.com una InRelease Could not connect to packages.linuxmint.com:80 (208.77.20.11), connection timed out Could not connect to packages.linuxmint.com:80 (68.235.39.11), connection timed outWhile the browsing is working fine, in fact I am posting this from same laptop in same VLAN
Pfsense unless you have different rules, or using IPS - pfsense doesn't know any different between lan or vlan X.. Just a different IP, that is natted to the same public IP, etc.
pfsense just knows traffic to that IP on port 80.. So why it works via browser, and not via your pi box and apt.. Not sure how that is a pfsense thing.. The only thing that could make sense is some sort of IPS filtering that doesn't like something different in the packets. But you say your not doing that..But to base pfsense traffic to IP X on port Y is just that.. It doesn't know if browser or apt, or whatever it just knows IP and port - allowed or not allowed, oh nat this traffic to my wan IP..
So sniff on pfsense - you see the traffic to IP:80, do you see that go out the wan.. If so pfsense did its job.. If you see hit pfsense lan side interface but not go out.. Then you know your filtering when you think your not, IPS is only thing I could see filtering apt but not browser.
I can access repo website and I can see connection in states
HOME_SERVERS tcp 192.168.11.4:34742 -> 46.235.231.111:443 CLOSING:ESTABLISHED 13 / 16 2 KiB / 2 KiBAs you suggested IPS may be blocking apt update, Can i test by complete removal of Snort.
If yes what would be best way to complete removal of Snort -
Just disabling it will be sufficient. Or even just put it in non-blocking mode.
Just make sure to clear any existing blocks.
Check Diag > Tables. Make sure the snort2c table is empty.
Steve
-
@stephenw10
Tested apt updates by disabling & uninstalling Snort but no luck
-
But is the block still in the table?
-
@maddy_in65 said in Linux apt update/upgrade stopped working:
@stephenw10
Tested apt updates by disabling & uninstalling Snort but no luckSnort (and Suricata), when running in Legacy Mode Blocking, utilizes the pfSense firewall engine for blocking an IP address. It does so by making a system call that results in placing the IP to be blocked into a pre-defined
pftable called snort2c. This is the table @stephenw10 is asking you about. Even when you disable the Snort service, or if you remove the package, any IP addresses that have been previously added to that snort2cpftable remain there and the blocks will persist.You can clear the table three ways: (1) when Snort is installed, go to the BLOCKS tab and click the button to remove all blocked hosts; (2) use the option under DIAGNOSTICS > TABLES in pfSense to display the snort2c table and remove any resident IP addresses; or (3) by rebooting the firewall. Rebooting clears the table of IP addresses because the table is purely a RAM construct.
BUT, I don't think Snort is your problem here. Way back up you showed a screenshot of the BLOCKS tab from Snort, and there were zero blocked hosts displayed. That means Snort is not blocking anything. It reads the snort2c table contents directly and displays them on that tab, so if the tab shows no blocks then the table is empty.
-
@stephenw10
There is no host entry in snort2c table.
-
Mmm, not Snort then. It sure 'feels' like Snort though because it definitely blocks apt update traffic if you just enable all the signatures. I've seen that myself.
Something else is preventing it on that VLAN then. Is there anything else the traffic passes through? Filtering on an access point or similar?
Otherwise I would be looking at states and running packet captures to see what's actually happening.
Steve
-
@maddy_in65 From what you've posted it seems like only outbound traffic to port 80 from the problem VLAN is failing. Maybe run
grep ' 80 ' /tmp/rules.debugand look for something other than the standard "anti-lockout rule"?
