Suricata blocking IPs on passlist, legacy mode blocking both

sgnoc

@bmeeks I've enabled pass list debugging. The down side is previously I would only get alerts on the affected interfaces periodically, so it could be a day or two. I've enabled some of the noisy alerts that were previously disabled to try and generate some more alert activity. I'll report back as soon as I can get any results.

Hopefully someone else on here can do the same if they have more frequent blocks.

bmeeks

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

@bmeeks I've enabled pass list debugging. The down side is previously I would only get alerts on the affected interfaces periodically, so it could be a day or two. I've enabled some of the noisy alerts that were previously disabled to try and generate some more alert activity. I'll report back as soon as I can get any results.

Hopefully someone else on here can do the same if they have more frequent blocks.

While working on the Hyperscan bug in Suricata that's reported in another thread, I stumbled upon two additional bugs in the custom blocking module. They both could affect the operation of the Pass List when testing whether or not to block IP addresses.

sgnoc

@bmeeks That would tend to make sense with what I've seen on my end. This issue seemed to pop up around the same time I started having the hyperscan issue, so it would make a lot more sense to me if they were related.

Hopefully the bug you found will resolve these! Thanks for all of your efforts on the Suricata package!

sgnoc

@bmeeks Is there any chance the debugging mode turned on would use different logic that might not be affected by the bug(s)? As soon as I went through the steps to enable the pass list debugging mode and restarted the interfaces, they started working properly.

Oddly enough, these 10.10.5.0/24 addresses were being blocked before I switched to debugging mode, and now it is working as expected. Here are the associated lines in the debug log:

12/19/2023-16:20:25.178129  Thread: W#01  DST IP: 10.10.5.110 covered by Pass List entry 10.10.5.0/24 - not blocking.
12/19/2023-17:22:53.688965  Thread: W#03  DST IP: 10.10.5.120 covered by Pass List entry 10.10.5.0/24 - not blocking.
12/19/2023-17:45:17.282641  Thread: W#01  SRC IP: 10.10.5.121 covered by Pass List entry 10.10.5.0/24 - not blocking.

I'm going to turn off debugging and see if the IPs start getting blocked again, otherwise I have no explanation why it suddenly started working. Of course this information is likely less than helpful on any troubleshooting steps.

bmeeks

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

@bmeeks Is there any chance the debugging mode turned on would use different logic that might not be affected by the bug(s)? As soon as I went through the steps to enable the pass list debugging mode and restarted the interfaces, they started working properly.

Oddly enough, these 10.10.5.0/24 addresses were being blocked before I switched to debugging mode, and now it is working as expected. Here are the associated lines in the debug log:
12/19/2023-16:20:25.178129  Thread: W#01  DST IP: 10.10.5.110 covered by Pass List entry 10.10.5.0/24 - not blocking.
12/19/2023-17:22:53.688965  Thread: W#03  DST IP: 10.10.5.120 covered by Pass List entry 10.10.5.0/24 - not blocking.
12/19/2023-17:45:17.282641  Thread: W#01  SRC IP: 10.10.5.121 covered by Pass List entry 10.10.5.0/24 - not blocking.
I'm going to turn off debugging and see if the IPs start getting blocked again, otherwise I have no explanation why it suddenly started working. Of course this information is likely less than helpful on any troubleshooting steps.

Testing again with Pass List debugging turned off will be of interest to me. I don't think the debugging would impact the logic as it simply prints things to a log file.

The two bugs I found this morning would produce random "results" from the Pass List logic when looking up the IP addresses in the Radix Tree pulled from a packet as they result in random memory corruption within the running Suricata binary.

It's possible the extra debugging steps would arrange memory such that the corruption is not surfacing. Or it could just be the random alignment of the moon and stars in how the Suricata data is arranged in memory and which specific addresses get overwritten by the heap buffer overflow bugs. In short, it very well could be the next time you start and run Suricata the addresses not being blocked now might get blocked then.

The fixes seem to have corrected the Hyperscan bug based on limited testing by one user. If his results hold out, I will submit a pull request to the Netgate developer team to incorporate these bug fixes into a Suricata package update. It is highly likely the bug fixes will improve the performance and accuracy of the Pass List logic, too.

sgnoc

@bmeeks As expected, running today I did not receive any issues with the internal private IP space getting blocked. Strange as I had issues for multiple days with the issue, then it just cleared up. I guess that validates what you were saying as the bugs expectedly caused random issues to pop up on the pass list rule comparisons.

I've just updated to the latest release of Suricata to fix the bug issues, so now to wait and see if the Hyperscan issues and the pass list issues clear up for me. I've reset all my rules to run how they were originally configured and have set the pattern matcher back to auto for the issue in the other topic. I'll post back to you in the appropriate topic if either issues recur.

Thanks again!

sgnoc

@bmeeks Looks like something is broken in the Suricata update. As soon as I updated and got the interfaces on Suricata restarted, now my WAN is immediately blocking my WAN IP as soon as an alert on the WAN interface triggers. I checked the default pass list and it has the WAN gateway IP, but not the WAN IP listed.

Would anything in the update not have the WAN IP listed in the default passlist?

I had to disable blocking on my WAN interface just to keep the internet connection open. Oddly, the Home Net shows the WAN IP and the WAN Gateway, but the default pass list is missing the WAN IP, and shows only the WAN Gateway.

WAN default pass list with blocking enabled:

10.10.5.0/24
10.10.5.101/32
10.10.6.0/24
10.10.7.0/24
10.10.8.0/24
10.10.9.0/24
10.10.10.0/24
10.10.11.0/24
10.10.15.0/24
10.10.25.0/24
10.10.31.0/29
10.10.32.0/29
10.10.33.0/29
10.10.34.0/29
10.10.35.0/29
10.10.36.0/29
10.10.37.0/29
10.10.45.0/24
10.10.55.0/24
10.10.60.0/29
<WAN Gateway>/32
fe80:6::/64
fe80:7::/64
fe80:8::/64
fe80:9::/64
fe80:10::/64

Home Net list from WAN Interface:

10.10.5.0/24
10.10.5.101/32
10.10.6.0/24
10.10.7.0/24
10.10.8.0/24
10.10.9.0/24
10.10.10.0/24
10.10.11.0/24
10.10.15.0/24
10.10.25.0/24
10.10.31.0/29
10.10.32.0/29
10.10.33.0/29
10.10.34.0/29
10.10.35.0/29
10.10.36.0/29
10.10.37.0/29
10.10.45.0/24
10.10.55.0/24
10.10.60.0/29
<WAN Gateway>/32
<WAN IP>/32
127.0.0.1/32
::1/128
fe80:6::/64
fe80:7::/64
fe80:8::/64
fe80:9::/64
fe80:10::/64
fe80::208:a2ff:fe0e:f6d8/128
fe80::208:a2ff:fe0e:f6d9/128
fe80::208:a2ff:fe0e:f6da/128

bmeeks

I'm not seeing the same issue. Here are some screenshots from a test I just now performed to prove it works as designed:

I installed the latest Suricata package version and configured an instance on the WAN of a pfSense virtual machine for easy testing. I then scanned the WAN IP address of the VM with a simple nmap stealth SYN scan.

Here is the STATUS > INTERFACES page showing the WAN IP of the pfSense virtual machine is 192.168.10.28.

Here is the ALERTS tab showing some of the nmap scan hits. I've drawn a red rectangle around the WAN interface IP to show it was a target address.

Here is the BLOCKS tab showing the current blocks. Notice the WAN IP is not present in the image below although it was the direct target of the nmap scan. It was not blocked (as expected, since it will be part of the internal automatic pass list).

The 192.168.10.125 IP in the list above is the Kali Linux host that performed the nmap scan of the pfSense firewall virtual machine.

Your WAN IP is not present because perhaps the interface was not up and fully configured when Suricata started. The firewall interface IPs are added automatically by an internal routine in the custom blocking module. They are no longer required in the user-supplied Pass List. But you can certainly add them if you want to. To see what firewall interface IPs were automatically added, examine the suricata.log file for the interface under the LOGS VIEW tab.

I also enabled Pass List debugging to show the WAN IP is present in the automatica Pass List and was checked (and not blocked). Here are the pertinent sections:

First, here is an excerpt from the suricata.log for the interface showing the Interface Monitoring thread starting and populating the automatic firewall interface Pass List. It does this by querying the operating system to obtain a list of currently active interface IP addresses. Notice it added all the firewall interface IPs to the internal list, including the WAN at 192.168.10.28. The WAN IP was the second one added to the automatic list.

[100602 - Suricata-Main] 2023-12-20 22:05:47 Info: alert-pf: Creating automatic firewall interface IP address Pass List.
[100602 - Suricata-Main] 2023-12-20 22:05:47 Info: alert-pf: Adding firewall interface em0 IPv6 address fe80:0000:0000:0000:020c:29ff:fee2:8064 to automatic interface IP Pass List.
[100602 - Suricata-Main] 2023-12-20 22:05:47 Info: alert-pf: Adding firewall interface em0 IPv4 address 192.168.10.28 to automatic interface IP Pass List.
[100602 - Suricata-Main] 2023-12-20 22:05:47 Info: alert-pf: Adding firewall interface em1 IPv6 address fe80:0000:0000:0000:020c:29ff:fee2:805a to automatic interface IP Pass List.
[100602 - Suricata-Main] 2023-12-20 22:05:47 Info: alert-pf: Adding firewall interface em1 IPv4 address 192.168.233.16 to automatic interface IP Pass List.
[100602 - Suricata-Main] 2023-12-20 22:05:47 Info: alert-pf: Adding firewall interface lo0 IPv6 address 0000:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List.
[100602 - Suricata-Main] 2023-12-20 22:05:47 Info: alert-pf: Adding firewall interface lo0 IPv6 address fe80:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List.
[100602 - Suricata-Main] 2023-12-20 22:05:47 Info: alert-pf: Adding firewall interface lo0 IPv4 address 127.0.0.1 to automatic interface IP Pass List.

Here is the Pass List debugging log output showing how the logic handled the Kali Linux machine (at 192.168.10.125) and the pfSense firewall that was the nmap target (at 192.168.10.28).

12/20/2023-22:06:36.934029  Thread: W#01  SRC IP: 192.168.10.125 did not match any Pass List entry, so adding to block list.
12/20/2023-22:06:37.257918  Thread: W#01  Successfully added IP: 192.168.10.125 to pf table snort2c for blocking.
12/20/2023-22:06:36.934004  Thread: W#03  SRC IP: 192.168.10.125 did not match any Pass List entry, so adding to block list.
12/20/2023-22:06:37.271567  Thread: W#03  Successfully killed any open states for IP: 192.168.10.125, so any stateful traffic is blocked.
12/20/2023-22:06:36.934004  Thread: W#03  DST IP: 192.168.10.28 covered by Pass List entry 192.168.10.28/32 - not blocking.

Notice the very last line shows the Pass List logic determined the 192.168.10.28 WAN IP address was in the Pass List and thus was not blocked.

sgnoc

@bmeeks Well that was unexplainable. All I did was stop the suricata service, update, and start the interfaces again, which resulted in the above.

I stopped the service, reinstalled, then had to do a restart of pfsense and finally got everything working as expected. So far the interfaces seem to be blocking properly and yet to have a hyperscan issue. At least I'm back up and running. pfSense did not like doing that update for whatever reason.

bmeeks

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

@bmeeks Well that was unexplainable. All I did was stop the suricata service, update, and start the interfaces again, which resulted in the above.

I stopped the service, reinstalled, then had to do a restart of pfsense and finally got everything working as expected. So far the interfaces seem to be blocking properly and yet to have a hyperscan issue. At least I'm back up and running. pfSense did not like doing that update for whatever reason.

Usually it is better to remove the package, then reinstall it instead of simply updating. Things can be left hanging around from the old version if every little thing does not go off to perfection.

I prefer to remove and then reinstall when performing an update of the package. I have tested it the other way, and it does work, but pfSense can now and then cause a hiccup when updating a package in-place.

If you had to restart pfSense itself, that makes me suspect you had multiple instances of Suricata running on the same interface.

sgnoc

@bmeeks Thanks, I'll try that next time. I thought the update did remove the package first, but if actually doing a removal then install is a cleaner process I'll stick with that going forward.

sgnoc

@bmeeks This morning I had one of my interfaces block an internal IP, belonging to a web server. I don't have the pass list debugging on, because I just got everything working with the new update that was pushed out.

Here is as much as I can provide for the meantime. The interface is on the subnet 10.10.31.0/29. I've enabled pass list debugging and restarted the interface, so now to just wait as long as it takes to generate another alert on that interface.

Interface block.log:

12/21/2023-06:38:51.223305  [Block Dst] [**] [1:2013057:5] ET WEB_SERVER Inbound PHP User-Agent [**] [Classification: Attempted Informati                                        on Leak] [Priority: 2] {TCP} 10.10.33.2:80

Interface default IP Pass List:

10.10.5.0/24
10.10.5.101/32
10.10.6.0/24
10.10.7.0/24
10.10.8.0/24
10.10.9.0/24
10.10.10.0/24
10.10.11.0/24
10.10.15.0/24
10.10.25.0/24
10.10.31.0/29
10.10.32.0/29
10.10.33.0/29
10.10.34.0/29
10.10.35.0/29
10.10.36.0/29
10.10.37.0/29
10.10.45.0/24
10.10.55.0/24
10.10.60.0/29
<WAN Gateway>/32
fe80:6::/64
fe80:7::/64
fe80:8::/64
fe80:9::/64
fe80:10::/64

sgnoc

Again I run into the same situation. I was seeing IPs on the pass list being blocked when alerts hit, but when I enable pass list debugging, everything starts working as normal. Here are similar logs on the same interface and subnet from the pass list debugger log.

12/21/2023-16:44:28.587065  Thread: W#03  SRC IP: 172.71.142.65 did not match any Pass List entry, so adding to block list.
12/21/2023-16:44:28.726957  Thread: W#03  Successfully added IP: 172.71.142.65 to pf table snort2c for blocking.
12/21/2023-16:44:28.814591  Thread: W#03  Successfully killed any open states for IP: 172.71.142.65, so any stateful traffic is blocked.
12/21/2023-16:44:28.587065  Thread: W#03  DST IP: 10.10.33.2 covered by Pass List entry 10.10.33.0/29 - not blocking.
12/21/2023-17:17:06.123239  Thread: W#02  SRC IP: 172.70.175.102 did not match any Pass List entry, so adding to block list.
12/21/2023-17:17:06.432804  Thread: W#02  Successfully added IP: 172.70.175.102 to pf table snort2c for blocking.
12/21/2023-17:17:06.520426  Thread: W#02  Successfully killed any open states for IP: 172.70.175.102, so any stateful traffic is blocked.
12/21/2023-17:17:06.123239  Thread: W#02  DST IP: 10.10.33.2 covered by Pass List entry 10.10.33.0/29 - not blocking.
12/21/2023-19:32:25.421183  Thread: W#01  SRC IP: 172.69.200.146 did not match any Pass List entry, so adding to block list.
12/21/2023-19:32:25.663962  Thread: W#01  Successfully added IP: 172.69.200.146 to pf table snort2c for blocking.
12/21/2023-19:32:25.751367  Thread: W#01  Successfully killed any open states for IP: 172.69.200.146, so any stateful traffic is blocked.
12/21/2023-19:32:25.421183  Thread: W#01  DST IP: 10.10.33.2 covered by Pass List entry 10.10.33.0/29 - not blocking.

No idea why things start working when I turn on pass list debugging, but I get blocks when I'm not running debugging. It is also sporadic. Some interfaces are working and others are blocking IPs on the pass list. I'll keep trying to test the best I'm able.

btspce

@sgnoc @bmeeks
Im also seeing this now after upgrade from 23.05.1 to 23.09.1 7.2.0_3.
We had our internal dns server blocked today despite being on the IP Pass List. We did not observe this before this upgrade.

This was also the reason why we were seeing wan failover on our 6100 HA setup. The CARP VIP WAN IP was blocked by suricata when it started.
No extra suricata processes is running. This must be another bug in this release.

sgnoc

@btspce I was only seeing IPs that were part of a netblock listed in the IP Pass List, but yesterday also had my DNS blocked, which brought the whole network down. The DNS IP was specifically listed, presumably because it is listed as the DNS for pfSense. So it isn't an issue with just the subnets listed in the IP Pass List, but also the individual IPs. That's a little different from my original thoughts on the bug.

btspce

@sgnoc The internal dns is also the primary dns for this pfsense install. And the WAN CARP VIP is also being blocked within seconds of suricata starting on wan which triggers a failover resulting in both firewalls becomes master and all traffic dies until both firewalls are rebooted. This is very severe if any random internal ip can be blocked at any moment as it could result in all sorts of issues including management lockout.

SteveITS

Are you two running Suricata on WAN? Have you tried moving it to LAN, where IIRC it’s not even going to see the WAN IP.

sgnoc

@SteveITS No issues running on WAN. I like running the bad reputation lists on my WAN. I'm having issues with internal interfaces blocking internal IPs within ranges or specifically listed on the default IP Pass List.

btspce

@SteveITS Yes WAN plus two other internal interfaces.
But due to suricata blocking the wan vip within seconds on start we have disabled suricata on wan.
But today one of our internal dns servers was blocked by suricata running since yesterday on an internal interface.
This is how we found out that the passlist was not working and sure enough the carp wan vip was blocked by suricata in the logs on wan.
This dns server is the primary pfsense dns server and is on the passlist.
So the conclusion is that there is still issues with the passlist not working randomly and there is a high risk that any random internal ip could be blocked at any moment with this version 7.0.2_3.

We are stopping suricata on all interfaces until this is resolved.

SteveITS

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

I like running the bad reputation lists on my WAN.

OK. We put those in a pfBlocker feed or alias and just create a rule.

I’ll watch on ours, AFAIK no internal IPs blocked yet. But thanks for the heads up. Threads like these recent Suricata threads are helpful to know what’s going on.