Suricata blocking IPs on passlist, legacy mode blocking both

sgnoc

@bmeeks Thanks, I'll try that next time. I thought the update did remove the package first, but if actually doing a removal then install is a cleaner process I'll stick with that going forward.

sgnoc

@bmeeks This morning I had one of my interfaces block an internal IP, belonging to a web server. I don't have the pass list debugging on, because I just got everything working with the new update that was pushed out.

Here is as much as I can provide for the meantime. The interface is on the subnet 10.10.31.0/29. I've enabled pass list debugging and restarted the interface, so now to just wait as long as it takes to generate another alert on that interface.

Interface block.log:

12/21/2023-06:38:51.223305  [Block Dst] [**] [1:2013057:5] ET WEB_SERVER Inbound PHP User-Agent [**] [Classification: Attempted Informati                                        on Leak] [Priority: 2] {TCP} 10.10.33.2:80

Interface default IP Pass List:

10.10.5.0/24
10.10.5.101/32
10.10.6.0/24
10.10.7.0/24
10.10.8.0/24
10.10.9.0/24
10.10.10.0/24
10.10.11.0/24
10.10.15.0/24
10.10.25.0/24
10.10.31.0/29
10.10.32.0/29
10.10.33.0/29
10.10.34.0/29
10.10.35.0/29
10.10.36.0/29
10.10.37.0/29
10.10.45.0/24
10.10.55.0/24
10.10.60.0/29
<WAN Gateway>/32
fe80:6::/64
fe80:7::/64
fe80:8::/64
fe80:9::/64
fe80:10::/64

sgnoc

Again I run into the same situation. I was seeing IPs on the pass list being blocked when alerts hit, but when I enable pass list debugging, everything starts working as normal. Here are similar logs on the same interface and subnet from the pass list debugger log.

12/21/2023-16:44:28.587065  Thread: W#03  SRC IP: 172.71.142.65 did not match any Pass List entry, so adding to block list.
12/21/2023-16:44:28.726957  Thread: W#03  Successfully added IP: 172.71.142.65 to pf table snort2c for blocking.
12/21/2023-16:44:28.814591  Thread: W#03  Successfully killed any open states for IP: 172.71.142.65, so any stateful traffic is blocked.
12/21/2023-16:44:28.587065  Thread: W#03  DST IP: 10.10.33.2 covered by Pass List entry 10.10.33.0/29 - not blocking.
12/21/2023-17:17:06.123239  Thread: W#02  SRC IP: 172.70.175.102 did not match any Pass List entry, so adding to block list.
12/21/2023-17:17:06.432804  Thread: W#02  Successfully added IP: 172.70.175.102 to pf table snort2c for blocking.
12/21/2023-17:17:06.520426  Thread: W#02  Successfully killed any open states for IP: 172.70.175.102, so any stateful traffic is blocked.
12/21/2023-17:17:06.123239  Thread: W#02  DST IP: 10.10.33.2 covered by Pass List entry 10.10.33.0/29 - not blocking.
12/21/2023-19:32:25.421183  Thread: W#01  SRC IP: 172.69.200.146 did not match any Pass List entry, so adding to block list.
12/21/2023-19:32:25.663962  Thread: W#01  Successfully added IP: 172.69.200.146 to pf table snort2c for blocking.
12/21/2023-19:32:25.751367  Thread: W#01  Successfully killed any open states for IP: 172.69.200.146, so any stateful traffic is blocked.
12/21/2023-19:32:25.421183  Thread: W#01  DST IP: 10.10.33.2 covered by Pass List entry 10.10.33.0/29 - not blocking.

No idea why things start working when I turn on pass list debugging, but I get blocks when I'm not running debugging. It is also sporadic. Some interfaces are working and others are blocking IPs on the pass list. I'll keep trying to test the best I'm able.

btspce

@sgnoc @bmeeks
Im also seeing this now after upgrade from 23.05.1 to 23.09.1 7.2.0_3.
We had our internal dns server blocked today despite being on the IP Pass List. We did not observe this before this upgrade.

This was also the reason why we were seeing wan failover on our 6100 HA setup. The CARP VIP WAN IP was blocked by suricata when it started.
No extra suricata processes is running. This must be another bug in this release.

sgnoc

@btspce I was only seeing IPs that were part of a netblock listed in the IP Pass List, but yesterday also had my DNS blocked, which brought the whole network down. The DNS IP was specifically listed, presumably because it is listed as the DNS for pfSense. So it isn't an issue with just the subnets listed in the IP Pass List, but also the individual IPs. That's a little different from my original thoughts on the bug.

btspce

@sgnoc The internal dns is also the primary dns for this pfsense install. And the WAN CARP VIP is also being blocked within seconds of suricata starting on wan which triggers a failover resulting in both firewalls becomes master and all traffic dies until both firewalls are rebooted. This is very severe if any random internal ip can be blocked at any moment as it could result in all sorts of issues including management lockout.

SteveITS

Are you two running Suricata on WAN? Have you tried moving it to LAN, where IIRC it’s not even going to see the WAN IP.

sgnoc

@SteveITS No issues running on WAN. I like running the bad reputation lists on my WAN. I'm having issues with internal interfaces blocking internal IPs within ranges or specifically listed on the default IP Pass List.

btspce

@SteveITS Yes WAN plus two other internal interfaces.
But due to suricata blocking the wan vip within seconds on start we have disabled suricata on wan.
But today one of our internal dns servers was blocked by suricata running since yesterday on an internal interface.
This is how we found out that the passlist was not working and sure enough the carp wan vip was blocked by suricata in the logs on wan.
This dns server is the primary pfsense dns server and is on the passlist.
So the conclusion is that there is still issues with the passlist not working randomly and there is a high risk that any random internal ip could be blocked at any moment with this version 7.0.2_3.

We are stopping suricata on all interfaces until this is resolved.

SteveITS

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

I like running the bad reputation lists on my WAN.

OK. We put those in a pfBlocker feed or alias and just create a rule.

I’ll watch on ours, AFAIK no internal IPs blocked yet. But thanks for the heads up. Threads like these recent Suricata threads are helpful to know what’s going on.

AlternateShadow

Suricata is running only on internal interfaces here and DST addresses on the pass list are getting blocked.

SteveITS

Is everyone using the Default pass list or a custom list? We're using a custom list with all the "Auto-Generated IP Addresses" boxes checked, and our "trusted" alias added.

btspce

@SteveITS Threads like these is why we waited with the suricata upgrade but unfortunately there were still bugs left. Still grateful for @bmeeks great work.
Keep an eye on your lan ip so it doesn't get randomly blocked (!) . IP's from both the default passlist and the IP Pass List can be blocked as it is now.

We are using a custom passlist with all the "Auto-Generated IP Addresses" boxes checked and trusted alias added aswell.

SteveITS

@btspce What rule is blocking your internal IPs? I'm wondering if it's not something we have enabled.

Our LAN IP has actually shown up but not been blocked... I suppressed a "SURICATA SSH invalid banner" alert yesterday from an internal network scanner/probe IP and it didn't block either.

I'd upgraded Suricata and set it back to Auto the day before.

btspce

@SteveITS Our WAN VIP and our DNS internal IP were both found in suricatas block list and was very much blocked until removed.
Suricata works very well in that regard :)

btspce

@SteveITS

WAN VIP
[Block Dst] [] [1:2402000:6860] ET DROP Dshield Block Listed Source group 1 [] [Classification: Misc Attack] [Priority: 2] {TCP}
[Block Dst] [] [1:2402000:6860] ET DROP Dshield Block Listed Source group 1 [] [Classification: Misc Attack] [Priority: 2] {TCP}

DNS
[Block Dst] [] [1:2035465:4] ET INFO Observed Discord Domain in DNS Lookup (discord .com) [] [Classification: Misc activity] [Priority: 3] {UDP}

SteveITS

@btspce FWIW we don't have either of those enabled...DShield is covered by the ET_Block feed in pfBlocker (so plain fw rule) and "info" is usually meant as informational/observation per Bill and we'd seen a lot of false positives so we don't have those enabled. So, small possibility it's rule related but I would think not.

"when I enable pass list debugging, everything starts working as normal"

Knowing absolutely nothing about the code, maybe thread/timing related?

btspce

@SteveITS Well @bmeeks already found and fixed two bugs related to the passlist randomly not working higher up in this thread which was included in the latest suricata version as I understands it so another one seems likely at this point. I'm waiting for Bill to chime in but it's weird you don't see any issues yet.

Anyway suricata should not be blocking whitelisted ip's.

sgnoc

@SteveITS I'm using the default pass list on all of my interfaces.

bmeeks

@btspce and @sgnoc:
I need some additional information from both of you to help narrow this down.

Post the full output of the suricata.log file for the impacted interface (or interfaces if several). You can easily view that file and copy its contents to the clipboard for pasting here on the forum under the LOGS VIEW tab in Suricata. To make reading the file easier, once you paste its contents into your post, highlight all the text you just pasted with your mouse and then click the "Code" icon at the top of the post submission dialog. That icon looks like this: </>.
Use the DIAGNOSTICS > EDIT FILE menu choice in pfSense and browse to the configuration directory for an impacted Suricata interface and paste the full contents of the pass_list file back here. You will find the file under /usr/local/etc/suricata/suricata_xxx_yyyyy on the firewall. Again, use the DIAGNOSTICS > EDIT FILE menu choice to browse to the file and open it. Paste the contents back here. To format the pasted text so it's easier to read, do the same thing as step #1 above: highlight all of the pasted in text and click the Code icon (</>) to format it.
Are you using VLANs on the impacted interfaces? If so, how many?

Turn on the pass list debugging option as described in this post of mine higher up in this thread: https://forum.netgate.com/topic/184858/suricata-blocking-ips-on-passlist-legacy-mode-blocking-both/8.

I examined the Pass List logic pretty much all day yesterday, but I am not finding anything obviously wrong. Whatever is happening is subtle because not all users are impacted.