Suricata blocking IPs on passlist, legacy mode blocking both

bmeeks

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

@bmeeks Well that was unexplainable. All I did was stop the suricata service, update, and start the interfaces again, which resulted in the above.

I stopped the service, reinstalled, then had to do a restart of pfsense and finally got everything working as expected. So far the interfaces seem to be blocking properly and yet to have a hyperscan issue. At least I'm back up and running. pfSense did not like doing that update for whatever reason.

Usually it is better to remove the package, then reinstall it instead of simply updating. Things can be left hanging around from the old version if every little thing does not go off to perfection.

I prefer to remove and then reinstall when performing an update of the package. I have tested it the other way, and it does work, but pfSense can now and then cause a hiccup when updating a package in-place.

If you had to restart pfSense itself, that makes me suspect you had multiple instances of Suricata running on the same interface.

sgnoc

@bmeeks Thanks, I'll try that next time. I thought the update did remove the package first, but if actually doing a removal then install is a cleaner process I'll stick with that going forward.

sgnoc

@bmeeks This morning I had one of my interfaces block an internal IP, belonging to a web server. I don't have the pass list debugging on, because I just got everything working with the new update that was pushed out.

Here is as much as I can provide for the meantime. The interface is on the subnet 10.10.31.0/29. I've enabled pass list debugging and restarted the interface, so now to just wait as long as it takes to generate another alert on that interface.

Interface block.log:

12/21/2023-06:38:51.223305  [Block Dst] [**] [1:2013057:5] ET WEB_SERVER Inbound PHP User-Agent [**] [Classification: Attempted Informati                                        on Leak] [Priority: 2] {TCP} 10.10.33.2:80

Interface default IP Pass List:

10.10.5.0/24
10.10.5.101/32
10.10.6.0/24
10.10.7.0/24
10.10.8.0/24
10.10.9.0/24
10.10.10.0/24
10.10.11.0/24
10.10.15.0/24
10.10.25.0/24
10.10.31.0/29
10.10.32.0/29
10.10.33.0/29
10.10.34.0/29
10.10.35.0/29
10.10.36.0/29
10.10.37.0/29
10.10.45.0/24
10.10.55.0/24
10.10.60.0/29
<WAN Gateway>/32
fe80:6::/64
fe80:7::/64
fe80:8::/64
fe80:9::/64
fe80:10::/64

sgnoc

Again I run into the same situation. I was seeing IPs on the pass list being blocked when alerts hit, but when I enable pass list debugging, everything starts working as normal. Here are similar logs on the same interface and subnet from the pass list debugger log.

12/21/2023-16:44:28.587065  Thread: W#03  SRC IP: 172.71.142.65 did not match any Pass List entry, so adding to block list.
12/21/2023-16:44:28.726957  Thread: W#03  Successfully added IP: 172.71.142.65 to pf table snort2c for blocking.
12/21/2023-16:44:28.814591  Thread: W#03  Successfully killed any open states for IP: 172.71.142.65, so any stateful traffic is blocked.
12/21/2023-16:44:28.587065  Thread: W#03  DST IP: 10.10.33.2 covered by Pass List entry 10.10.33.0/29 - not blocking.
12/21/2023-17:17:06.123239  Thread: W#02  SRC IP: 172.70.175.102 did not match any Pass List entry, so adding to block list.
12/21/2023-17:17:06.432804  Thread: W#02  Successfully added IP: 172.70.175.102 to pf table snort2c for blocking.
12/21/2023-17:17:06.520426  Thread: W#02  Successfully killed any open states for IP: 172.70.175.102, so any stateful traffic is blocked.
12/21/2023-17:17:06.123239  Thread: W#02  DST IP: 10.10.33.2 covered by Pass List entry 10.10.33.0/29 - not blocking.
12/21/2023-19:32:25.421183  Thread: W#01  SRC IP: 172.69.200.146 did not match any Pass List entry, so adding to block list.
12/21/2023-19:32:25.663962  Thread: W#01  Successfully added IP: 172.69.200.146 to pf table snort2c for blocking.
12/21/2023-19:32:25.751367  Thread: W#01  Successfully killed any open states for IP: 172.69.200.146, so any stateful traffic is blocked.
12/21/2023-19:32:25.421183  Thread: W#01  DST IP: 10.10.33.2 covered by Pass List entry 10.10.33.0/29 - not blocking.

No idea why things start working when I turn on pass list debugging, but I get blocks when I'm not running debugging. It is also sporadic. Some interfaces are working and others are blocking IPs on the pass list. I'll keep trying to test the best I'm able.

btspce

@sgnoc @bmeeks
Im also seeing this now after upgrade from 23.05.1 to 23.09.1 7.2.0_3.
We had our internal dns server blocked today despite being on the IP Pass List. We did not observe this before this upgrade.

This was also the reason why we were seeing wan failover on our 6100 HA setup. The CARP VIP WAN IP was blocked by suricata when it started.
No extra suricata processes is running. This must be another bug in this release.

sgnoc

@btspce I was only seeing IPs that were part of a netblock listed in the IP Pass List, but yesterday also had my DNS blocked, which brought the whole network down. The DNS IP was specifically listed, presumably because it is listed as the DNS for pfSense. So it isn't an issue with just the subnets listed in the IP Pass List, but also the individual IPs. That's a little different from my original thoughts on the bug.

btspce

@sgnoc The internal dns is also the primary dns for this pfsense install. And the WAN CARP VIP is also being blocked within seconds of suricata starting on wan which triggers a failover resulting in both firewalls becomes master and all traffic dies until both firewalls are rebooted. This is very severe if any random internal ip can be blocked at any moment as it could result in all sorts of issues including management lockout.

SteveITS

Are you two running Suricata on WAN? Have you tried moving it to LAN, where IIRC it’s not even going to see the WAN IP.

sgnoc

@SteveITS No issues running on WAN. I like running the bad reputation lists on my WAN. I'm having issues with internal interfaces blocking internal IPs within ranges or specifically listed on the default IP Pass List.

btspce

@SteveITS Yes WAN plus two other internal interfaces.
But due to suricata blocking the wan vip within seconds on start we have disabled suricata on wan.
But today one of our internal dns servers was blocked by suricata running since yesterday on an internal interface.
This is how we found out that the passlist was not working and sure enough the carp wan vip was blocked by suricata in the logs on wan.
This dns server is the primary pfsense dns server and is on the passlist.
So the conclusion is that there is still issues with the passlist not working randomly and there is a high risk that any random internal ip could be blocked at any moment with this version 7.0.2_3.

We are stopping suricata on all interfaces until this is resolved.

SteveITS

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

I like running the bad reputation lists on my WAN.

OK. We put those in a pfBlocker feed or alias and just create a rule.

I’ll watch on ours, AFAIK no internal IPs blocked yet. But thanks for the heads up. Threads like these recent Suricata threads are helpful to know what’s going on.

AlternateShadow

Suricata is running only on internal interfaces here and DST addresses on the pass list are getting blocked.

SteveITS

Is everyone using the Default pass list or a custom list? We're using a custom list with all the "Auto-Generated IP Addresses" boxes checked, and our "trusted" alias added.

btspce

@SteveITS Threads like these is why we waited with the suricata upgrade but unfortunately there were still bugs left. Still grateful for @bmeeks great work.
Keep an eye on your lan ip so it doesn't get randomly blocked (!) . IP's from both the default passlist and the IP Pass List can be blocked as it is now.

We are using a custom passlist with all the "Auto-Generated IP Addresses" boxes checked and trusted alias added aswell.

SteveITS

@btspce What rule is blocking your internal IPs? I'm wondering if it's not something we have enabled.

Our LAN IP has actually shown up but not been blocked... I suppressed a "SURICATA SSH invalid banner" alert yesterday from an internal network scanner/probe IP and it didn't block either.

I'd upgraded Suricata and set it back to Auto the day before.

btspce

@SteveITS Our WAN VIP and our DNS internal IP were both found in suricatas block list and was very much blocked until removed.
Suricata works very well in that regard :)

btspce

@SteveITS

WAN VIP
[Block Dst] [] [1:2402000:6860] ET DROP Dshield Block Listed Source group 1 [] [Classification: Misc Attack] [Priority: 2] {TCP}
[Block Dst] [] [1:2402000:6860] ET DROP Dshield Block Listed Source group 1 [] [Classification: Misc Attack] [Priority: 2] {TCP}

DNS
[Block Dst] [] [1:2035465:4] ET INFO Observed Discord Domain in DNS Lookup (discord .com) [] [Classification: Misc activity] [Priority: 3] {UDP}

SteveITS

@btspce FWIW we don't have either of those enabled...DShield is covered by the ET_Block feed in pfBlocker (so plain fw rule) and "info" is usually meant as informational/observation per Bill and we'd seen a lot of false positives so we don't have those enabled. So, small possibility it's rule related but I would think not.

"when I enable pass list debugging, everything starts working as normal"

Knowing absolutely nothing about the code, maybe thread/timing related?

btspce

@SteveITS Well @bmeeks already found and fixed two bugs related to the passlist randomly not working higher up in this thread which was included in the latest suricata version as I understands it so another one seems likely at this point. I'm waiting for Bill to chime in but it's weird you don't see any issues yet.

Anyway suricata should not be blocking whitelisted ip's.

sgnoc

@SteveITS I'm using the default pass list on all of my interfaces.