Linux apt update/upgrade stopped working

johnpoz

@maddy_in65 said in Linux apt update/upgrade stopped working:

I have IPv6 blocked in pfsense.

But from your output your devices were trying to use IPv6..

Your saying nothing is different between these vlans... You can call up the website (from this vlan?)

Pfsense unless you have different rules, or using IPS - pfsense doesn't know any different between lan or vlan X.. Just a different IP, that is natted to the same public IP, etc.

Could not connect to raspbian.raspberrypi.org:80 (93.93.128.193)

But your saying browser can go there - there is no difference to pfsense..

pfsense just knows traffic to that IP on port 80.. So why it works via browser, and not via your pi box and apt.. Not sure how that is a pfsense thing.. The only thing that could make sense is some sort of IPS filtering that doesn't like something different in the packets. But you say your not doing that..

But to base pfsense traffic to IP X on port Y is just that.. It doesn't know if browser or apt, or whatever it just knows IP and port - allowed or not allowed, oh nat this traffic to my wan IP..

So sniff on pfsense - you see the traffic to IP:80, do you see that go out the wan.. If so pfsense did its job.. If you see hit pfsense lan side interface but not go out.. Then you know your filtering when you think your not, IPS is only thing I could see filtering apt but not browser.

stephenw10

Look for states to one of those IPs that's failing. Make sure you have states on WAN and the VLAN in question.

Get a packet capture on WAN for the same IP. See what's actually happening there, are any packets coming back at all?

Steve

maddy_in65

@johnpoz said in Linux apt update/upgrade stopped working:

@maddy_in65 said in Linux apt update/upgrade stopped working:

I have IPv6 blocked in pfsense.

But from your output your devices were trying to use IPv6..

Your saying nothing is different between these vlans... You can call up the website (from this vlan?)

I put my laptop in server VLAN and guess what it also has same issue. Apt update stopped working.

Hit:1 https://packages.microsoft.com/repos/ms-teams stable InRelease           
Hit:2 https://repo.skype.com/deb stable InRelease                              
Err:3 http://dl.google.com/linux/chrome/deb stable InRelease                                                            
  Cannot initiate the connection to dl.google.com:80 (2404:6800:4009:812::200e). - connect (101: Network is unreachable) Could not connect to dl.google.com:80 (216.58.203.14), connection timed out
Err:4 http://ppa.launchpad.net/inkscape.dev/stable/ubuntu focal InRelease                                               
  Cannot initiate the connection to ppa.launchpad.net:80 (2001:67c:1560:8008::19). - connect (101: Network is unreachable) Could not connect to ppa.launchpad.net:80 (91.189.95.85), connection timed out
Err:5 http://ppa.launchpad.net/tomtomtom/woeusb/ubuntu focal InRelease                                                  
  Cannot initiate the connection to ppa.launchpad.net:80 (2001:67c:1560:8008::19). - connect (101: Network is unreachable)
Err:6 http://packages.linuxmint.com una InRelease                                                                       
  Could not connect to packages.linuxmint.com:80 (208.77.20.11), connection timed out Could not connect to packages.linuxmint.com:80 (68.235.39.11), connection timed out

While the browsing is working fine, in fact I am posting this from same laptop in same VLAN

Pfsense unless you have different rules, or using IPS - pfsense doesn't know any different between lan or vlan X.. Just a different IP, that is natted to the same public IP, etc.
pfsense just knows traffic to that IP on port 80.. So why it works via browser, and not via your pi box and apt.. Not sure how that is a pfsense thing.. The only thing that could make sense is some sort of IPS filtering that doesn't like something different in the packets. But you say your not doing that..

But to base pfsense traffic to IP X on port Y is just that.. It doesn't know if browser or apt, or whatever it just knows IP and port - allowed or not allowed, oh nat this traffic to my wan IP..

So sniff on pfsense - you see the traffic to IP:80, do you see that go out the wan.. If so pfsense did its job.. If you see hit pfsense lan side interface but not go out.. Then you know your filtering when you think your not, IPS is only thing I could see filtering apt but not browser.

I can access repo website and I can see connection in states

HOME_SERVERS 	tcp 	192.168.11.4:34742 -> 46.235.231.111:443 	CLOSING:ESTABLISHED 	13 / 16 	2 KiB / 2 KiB

As you suggested IPS may be blocking apt update, Can i test by complete removal of Snort.
If yes what would be best way to complete removal of Snort

stephenw10

Just disabling it will be sufficient. Or even just put it in non-blocking mode.

Just make sure to clear any existing blocks.

Check Diag > Tables. Make sure the snort2c table is empty.

Steve

maddy_in65

@stephenw10
Tested apt updates by disabling & uninstalling Snort but no luck

stephenw10

But is the block still in the table?

bmeeks

@maddy_in65 said in Linux apt update/upgrade stopped working:

@stephenw10
Tested apt updates by disabling & uninstalling Snort but no luck

Snort (and Suricata), when running in Legacy Mode Blocking, utilizes the pfSense firewall engine for blocking an IP address. It does so by making a system call that results in placing the IP to be blocked into a pre-defined pf table called snort2c. This is the table @stephenw10 is asking you about. Even when you disable the Snort service, or if you remove the package, any IP addresses that have been previously added to that snort2c pf table remain there and the blocks will persist.

You can clear the table three ways: (1) when Snort is installed, go to the BLOCKS tab and click the button to remove all blocked hosts; (2) use the option under DIAGNOSTICS > TABLES in pfSense to display the snort2c table and remove any resident IP addresses; or (3) by rebooting the firewall. Rebooting clears the table of IP addresses because the table is purely a RAM construct.

BUT, I don't think Snort is your problem here. Way back up you showed a screenshot of the BLOCKS tab from Snort, and there were zero blocked hosts displayed. That means Snort is not blocking anything. It reads the snort2c table contents directly and displays them on that tab, so if the tab shows no blocks then the table is empty.

maddy_in65

@stephenw10
There is no host entry in snort2c table.

stephenw10

Mmm, not Snort then. It sure 'feels' like Snort though because it definitely blocks apt update traffic if you just enable all the signatures. I've seen that myself.

Something else is preventing it on that VLAN then. Is there anything else the traffic passes through? Filtering on an access point or similar?

Otherwise I would be looking at states and running packet captures to see what's actually happening.

Steve

dem

@maddy_in65 From what you've posted it seems like only outbound traffic to port 80 from the problem VLAN is failing. Maybe run grep ' 80 ' /tmp/rules.debug and look for something other than the standard "anti-lockout rule"?