Suricata blocking IPs on passlist, legacy mode blocking both

SteveITS

One cause, which I ran into today, is zombie processes. Stop all interfaces. Run “ps aux|grep suricata” and see if any still show, and kill them. Zombies don’t restart so they don’t pick up config changes, yet still trigger alerts.

sgnoc

@SteveITS Thanks, I checked and don't have any extra suricata instances. I've tried restarting my pfsense and also reinstalled suricata to see if any of it would help, but it hasn't so far.

sgnoc

Another thing to note, it seems sporadic. I checked my logs and I have a web server on 10.10.33.2, which did get blocked previously. Today there were more alerts on that interface and only the external IPs were blocked, not the 10.10.33.2 address.

Also, my internal DNS is listed in the default pass list (presumably because it is listed as the DNS address), along with that interface's subnet. The address for my DNS had 2 alerts, and it didn't get blocked, where other addresses in that subnet have been blocked previously. So it seems a specific address rather than a subnet hasn't gotten blocked. IE: DNS is 10.10.5.101 and the subnet 10.10.5.0/24 are both listed. I've had 10.10.5.110 and 10.10.5.120 addresses blocked over the last several days, but 10.10.5.101 had alerts and only the external address was blocked.

AlternateShadow

There might be multiple issues causing these symptoms, but at least in my case, it seams like something changed with the latest release. The instance that I’m having trouble with had been running flawlessly for a couple of years prior to this.

bmeeks

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

Another thing to note, it seems sporadic. I checked my logs and I have a web server on 10.10.33.2, which did get blocked previously. Today there were more alerts on that interface and only the external IPs were blocked, not the 10.10.33.2 address.

Also, my internal DNS is listed in the default pass list (presumably because it is listed as the DNS address), along with that interface's subnet. The address for my DNS had 2 alerts, and it didn't get blocked, where other addresses in that subnet have been blocked previously. So it seems a specific address rather than a subnet hasn't gotten blocked. IE: DNS is 10.10.5.101 and the subnet 10.10.5.0/24 are both listed. I've had 10.10.5.110 and 10.10.5.120 addresses blocked over the last several days, but 10.10.5.101 had alerts and only the external address was blocked.

Do this for me. There is a hidden passlist debug switch that will print a log file of Pass List logic actions. Here is how to enable it:

Go to the DIAGNOSTICS > EDIT FILE menu and browse to and edit the following file: /usr/local/pkg/suricata/suricata_yaml_template.inc.
Find this section in the file (it's near the top):

# alert-pf custom blocking plugin for pfSense only
- alert-pf:
    enabled: {$suri_blockoffenders}
    kill-state: {$suri_killstates}
    block-drops-only: {$suri_blockdrops}
    pass-list: {$suri_passlist}
    block-ip: {$suri_blockip}
    pf-table: {$suri_pf_table}
    passlist-debugging: no   # Do not enable debugging on production systems!

Change the passlist-debugging: no line to read passlist-debugging: yes and save the edit.
Return to the Suricata INTERFACES tab and edit the interface where you have Pass List entries being blocked. On the INTERFACE EDIT page, click the Save button to regenerate the suricata.yaml file for the interface.
Go to the INTERFACES tab in Suricata and restart the interface you edited.

You will then find a text log in the Suricata logging directory for the interface. That will be /var/log/suricata/suricata_xxxx_yyyy where xxxx and yyyy are the physical interface name and a random UUID. On a busy network this log file will start to grow quite large over time, so keep an eye on it. Also, Suricata will be a little less performant - hence the warning about not enabling the debug option on production systems. But running for a short time to test is not going to cause a large problem.

When you notice a blocked IP that should be covered by a Pass List entry, look through the passlist debug log to see what is recorded there for the IP.

WARNING: it is likely the log file will fairly quickly grow too big to view in the pfSense GUI. You will need a method to transfer a copy off to a PC for viewing. WinSCP is an excellent free Windows application for doing this.

To return to normal non-debug mode for the Pass List, simply repeat the steps except change the passlist-debugging: yes to passlist-debugging: no.

sgnoc

@bmeeks I've enabled pass list debugging. The down side is previously I would only get alerts on the affected interfaces periodically, so it could be a day or two. I've enabled some of the noisy alerts that were previously disabled to try and generate some more alert activity. I'll report back as soon as I can get any results.

Hopefully someone else on here can do the same if they have more frequent blocks.

bmeeks

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

@bmeeks I've enabled pass list debugging. The down side is previously I would only get alerts on the affected interfaces periodically, so it could be a day or two. I've enabled some of the noisy alerts that were previously disabled to try and generate some more alert activity. I'll report back as soon as I can get any results.

Hopefully someone else on here can do the same if they have more frequent blocks.

While working on the Hyperscan bug in Suricata that's reported in another thread, I stumbled upon two additional bugs in the custom blocking module. They both could affect the operation of the Pass List when testing whether or not to block IP addresses.

sgnoc

@bmeeks That would tend to make sense with what I've seen on my end. This issue seemed to pop up around the same time I started having the hyperscan issue, so it would make a lot more sense to me if they were related.

Hopefully the bug you found will resolve these! Thanks for all of your efforts on the Suricata package!

sgnoc

@bmeeks Is there any chance the debugging mode turned on would use different logic that might not be affected by the bug(s)? As soon as I went through the steps to enable the pass list debugging mode and restarted the interfaces, they started working properly.

Oddly enough, these 10.10.5.0/24 addresses were being blocked before I switched to debugging mode, and now it is working as expected. Here are the associated lines in the debug log:

12/19/2023-16:20:25.178129  Thread: W#01  DST IP: 10.10.5.110 covered by Pass List entry 10.10.5.0/24 - not blocking.
12/19/2023-17:22:53.688965  Thread: W#03  DST IP: 10.10.5.120 covered by Pass List entry 10.10.5.0/24 - not blocking.
12/19/2023-17:45:17.282641  Thread: W#01  SRC IP: 10.10.5.121 covered by Pass List entry 10.10.5.0/24 - not blocking.

I'm going to turn off debugging and see if the IPs start getting blocked again, otherwise I have no explanation why it suddenly started working. Of course this information is likely less than helpful on any troubleshooting steps.

bmeeks

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

@bmeeks Is there any chance the debugging mode turned on would use different logic that might not be affected by the bug(s)? As soon as I went through the steps to enable the pass list debugging mode and restarted the interfaces, they started working properly.

Oddly enough, these 10.10.5.0/24 addresses were being blocked before I switched to debugging mode, and now it is working as expected. Here are the associated lines in the debug log:
12/19/2023-16:20:25.178129  Thread: W#01  DST IP: 10.10.5.110 covered by Pass List entry 10.10.5.0/24 - not blocking.
12/19/2023-17:22:53.688965  Thread: W#03  DST IP: 10.10.5.120 covered by Pass List entry 10.10.5.0/24 - not blocking.
12/19/2023-17:45:17.282641  Thread: W#01  SRC IP: 10.10.5.121 covered by Pass List entry 10.10.5.0/24 - not blocking.
I'm going to turn off debugging and see if the IPs start getting blocked again, otherwise I have no explanation why it suddenly started working. Of course this information is likely less than helpful on any troubleshooting steps.

Testing again with Pass List debugging turned off will be of interest to me. I don't think the debugging would impact the logic as it simply prints things to a log file.

The two bugs I found this morning would produce random "results" from the Pass List logic when looking up the IP addresses in the Radix Tree pulled from a packet as they result in random memory corruption within the running Suricata binary.

It's possible the extra debugging steps would arrange memory such that the corruption is not surfacing. Or it could just be the random alignment of the moon and stars in how the Suricata data is arranged in memory and which specific addresses get overwritten by the heap buffer overflow bugs. In short, it very well could be the next time you start and run Suricata the addresses not being blocked now might get blocked then.

The fixes seem to have corrected the Hyperscan bug based on limited testing by one user. If his results hold out, I will submit a pull request to the Netgate developer team to incorporate these bug fixes into a Suricata package update. It is highly likely the bug fixes will improve the performance and accuracy of the Pass List logic, too.

sgnoc

@bmeeks As expected, running today I did not receive any issues with the internal private IP space getting blocked. Strange as I had issues for multiple days with the issue, then it just cleared up. I guess that validates what you were saying as the bugs expectedly caused random issues to pop up on the pass list rule comparisons.

I've just updated to the latest release of Suricata to fix the bug issues, so now to wait and see if the Hyperscan issues and the pass list issues clear up for me. I've reset all my rules to run how they were originally configured and have set the pattern matcher back to auto for the issue in the other topic. I'll post back to you in the appropriate topic if either issues recur.

Thanks again!

sgnoc

@bmeeks Looks like something is broken in the Suricata update. As soon as I updated and got the interfaces on Suricata restarted, now my WAN is immediately blocking my WAN IP as soon as an alert on the WAN interface triggers. I checked the default pass list and it has the WAN gateway IP, but not the WAN IP listed.

Would anything in the update not have the WAN IP listed in the default passlist?

I had to disable blocking on my WAN interface just to keep the internet connection open. Oddly, the Home Net shows the WAN IP and the WAN Gateway, but the default pass list is missing the WAN IP, and shows only the WAN Gateway.

WAN default pass list with blocking enabled:

10.10.5.0/24
10.10.5.101/32
10.10.6.0/24
10.10.7.0/24
10.10.8.0/24
10.10.9.0/24
10.10.10.0/24
10.10.11.0/24
10.10.15.0/24
10.10.25.0/24
10.10.31.0/29
10.10.32.0/29
10.10.33.0/29
10.10.34.0/29
10.10.35.0/29
10.10.36.0/29
10.10.37.0/29
10.10.45.0/24
10.10.55.0/24
10.10.60.0/29
<WAN Gateway>/32
fe80:6::/64
fe80:7::/64
fe80:8::/64
fe80:9::/64
fe80:10::/64

Home Net list from WAN Interface:

10.10.5.0/24
10.10.5.101/32
10.10.6.0/24
10.10.7.0/24
10.10.8.0/24
10.10.9.0/24
10.10.10.0/24
10.10.11.0/24
10.10.15.0/24
10.10.25.0/24
10.10.31.0/29
10.10.32.0/29
10.10.33.0/29
10.10.34.0/29
10.10.35.0/29
10.10.36.0/29
10.10.37.0/29
10.10.45.0/24
10.10.55.0/24
10.10.60.0/29
<WAN Gateway>/32
<WAN IP>/32
127.0.0.1/32
::1/128
fe80:6::/64
fe80:7::/64
fe80:8::/64
fe80:9::/64
fe80:10::/64
fe80::208:a2ff:fe0e:f6d8/128
fe80::208:a2ff:fe0e:f6d9/128
fe80::208:a2ff:fe0e:f6da/128

bmeeks

I'm not seeing the same issue. Here are some screenshots from a test I just now performed to prove it works as designed:

I installed the latest Suricata package version and configured an instance on the WAN of a pfSense virtual machine for easy testing. I then scanned the WAN IP address of the VM with a simple nmap stealth SYN scan.

Here is the STATUS > INTERFACES page showing the WAN IP of the pfSense virtual machine is 192.168.10.28.

Here is the ALERTS tab showing some of the nmap scan hits. I've drawn a red rectangle around the WAN interface IP to show it was a target address.

Here is the BLOCKS tab showing the current blocks. Notice the WAN IP is not present in the image below although it was the direct target of the nmap scan. It was not blocked (as expected, since it will be part of the internal automatic pass list).

The 192.168.10.125 IP in the list above is the Kali Linux host that performed the nmap scan of the pfSense firewall virtual machine.

Your WAN IP is not present because perhaps the interface was not up and fully configured when Suricata started. The firewall interface IPs are added automatically by an internal routine in the custom blocking module. They are no longer required in the user-supplied Pass List. But you can certainly add them if you want to. To see what firewall interface IPs were automatically added, examine the suricata.log file for the interface under the LOGS VIEW tab.

I also enabled Pass List debugging to show the WAN IP is present in the automatica Pass List and was checked (and not blocked). Here are the pertinent sections:

First, here is an excerpt from the suricata.log for the interface showing the Interface Monitoring thread starting and populating the automatic firewall interface Pass List. It does this by querying the operating system to obtain a list of currently active interface IP addresses. Notice it added all the firewall interface IPs to the internal list, including the WAN at 192.168.10.28. The WAN IP was the second one added to the automatic list.

[100602 - Suricata-Main] 2023-12-20 22:05:47 Info: alert-pf: Creating automatic firewall interface IP address Pass List.
[100602 - Suricata-Main] 2023-12-20 22:05:47 Info: alert-pf: Adding firewall interface em0 IPv6 address fe80:0000:0000:0000:020c:29ff:fee2:8064 to automatic interface IP Pass List.
[100602 - Suricata-Main] 2023-12-20 22:05:47 Info: alert-pf: Adding firewall interface em0 IPv4 address 192.168.10.28 to automatic interface IP Pass List.
[100602 - Suricata-Main] 2023-12-20 22:05:47 Info: alert-pf: Adding firewall interface em1 IPv6 address fe80:0000:0000:0000:020c:29ff:fee2:805a to automatic interface IP Pass List.
[100602 - Suricata-Main] 2023-12-20 22:05:47 Info: alert-pf: Adding firewall interface em1 IPv4 address 192.168.233.16 to automatic interface IP Pass List.
[100602 - Suricata-Main] 2023-12-20 22:05:47 Info: alert-pf: Adding firewall interface lo0 IPv6 address 0000:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List.
[100602 - Suricata-Main] 2023-12-20 22:05:47 Info: alert-pf: Adding firewall interface lo0 IPv6 address fe80:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List.
[100602 - Suricata-Main] 2023-12-20 22:05:47 Info: alert-pf: Adding firewall interface lo0 IPv4 address 127.0.0.1 to automatic interface IP Pass List.

Here is the Pass List debugging log output showing how the logic handled the Kali Linux machine (at 192.168.10.125) and the pfSense firewall that was the nmap target (at 192.168.10.28).

12/20/2023-22:06:36.934029  Thread: W#01  SRC IP: 192.168.10.125 did not match any Pass List entry, so adding to block list.
12/20/2023-22:06:37.257918  Thread: W#01  Successfully added IP: 192.168.10.125 to pf table snort2c for blocking.
12/20/2023-22:06:36.934004  Thread: W#03  SRC IP: 192.168.10.125 did not match any Pass List entry, so adding to block list.
12/20/2023-22:06:37.271567  Thread: W#03  Successfully killed any open states for IP: 192.168.10.125, so any stateful traffic is blocked.
12/20/2023-22:06:36.934004  Thread: W#03  DST IP: 192.168.10.28 covered by Pass List entry 192.168.10.28/32 - not blocking.

Notice the very last line shows the Pass List logic determined the 192.168.10.28 WAN IP address was in the Pass List and thus was not blocked.

sgnoc

@bmeeks Well that was unexplainable. All I did was stop the suricata service, update, and start the interfaces again, which resulted in the above.

I stopped the service, reinstalled, then had to do a restart of pfsense and finally got everything working as expected. So far the interfaces seem to be blocking properly and yet to have a hyperscan issue. At least I'm back up and running. pfSense did not like doing that update for whatever reason.

bmeeks

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

@bmeeks Well that was unexplainable. All I did was stop the suricata service, update, and start the interfaces again, which resulted in the above.

I stopped the service, reinstalled, then had to do a restart of pfsense and finally got everything working as expected. So far the interfaces seem to be blocking properly and yet to have a hyperscan issue. At least I'm back up and running. pfSense did not like doing that update for whatever reason.

Usually it is better to remove the package, then reinstall it instead of simply updating. Things can be left hanging around from the old version if every little thing does not go off to perfection.

I prefer to remove and then reinstall when performing an update of the package. I have tested it the other way, and it does work, but pfSense can now and then cause a hiccup when updating a package in-place.

If you had to restart pfSense itself, that makes me suspect you had multiple instances of Suricata running on the same interface.

sgnoc

@bmeeks Thanks, I'll try that next time. I thought the update did remove the package first, but if actually doing a removal then install is a cleaner process I'll stick with that going forward.

sgnoc

@bmeeks This morning I had one of my interfaces block an internal IP, belonging to a web server. I don't have the pass list debugging on, because I just got everything working with the new update that was pushed out.

Here is as much as I can provide for the meantime. The interface is on the subnet 10.10.31.0/29. I've enabled pass list debugging and restarted the interface, so now to just wait as long as it takes to generate another alert on that interface.

Interface block.log:

12/21/2023-06:38:51.223305  [Block Dst] [**] [1:2013057:5] ET WEB_SERVER Inbound PHP User-Agent [**] [Classification: Attempted Informati                                        on Leak] [Priority: 2] {TCP} 10.10.33.2:80

Interface default IP Pass List:

10.10.5.0/24
10.10.5.101/32
10.10.6.0/24
10.10.7.0/24
10.10.8.0/24
10.10.9.0/24
10.10.10.0/24
10.10.11.0/24
10.10.15.0/24
10.10.25.0/24
10.10.31.0/29
10.10.32.0/29
10.10.33.0/29
10.10.34.0/29
10.10.35.0/29
10.10.36.0/29
10.10.37.0/29
10.10.45.0/24
10.10.55.0/24
10.10.60.0/29
<WAN Gateway>/32
fe80:6::/64
fe80:7::/64
fe80:8::/64
fe80:9::/64
fe80:10::/64

sgnoc

Again I run into the same situation. I was seeing IPs on the pass list being blocked when alerts hit, but when I enable pass list debugging, everything starts working as normal. Here are similar logs on the same interface and subnet from the pass list debugger log.

12/21/2023-16:44:28.587065  Thread: W#03  SRC IP: 172.71.142.65 did not match any Pass List entry, so adding to block list.
12/21/2023-16:44:28.726957  Thread: W#03  Successfully added IP: 172.71.142.65 to pf table snort2c for blocking.
12/21/2023-16:44:28.814591  Thread: W#03  Successfully killed any open states for IP: 172.71.142.65, so any stateful traffic is blocked.
12/21/2023-16:44:28.587065  Thread: W#03  DST IP: 10.10.33.2 covered by Pass List entry 10.10.33.0/29 - not blocking.
12/21/2023-17:17:06.123239  Thread: W#02  SRC IP: 172.70.175.102 did not match any Pass List entry, so adding to block list.
12/21/2023-17:17:06.432804  Thread: W#02  Successfully added IP: 172.70.175.102 to pf table snort2c for blocking.
12/21/2023-17:17:06.520426  Thread: W#02  Successfully killed any open states for IP: 172.70.175.102, so any stateful traffic is blocked.
12/21/2023-17:17:06.123239  Thread: W#02  DST IP: 10.10.33.2 covered by Pass List entry 10.10.33.0/29 - not blocking.
12/21/2023-19:32:25.421183  Thread: W#01  SRC IP: 172.69.200.146 did not match any Pass List entry, so adding to block list.
12/21/2023-19:32:25.663962  Thread: W#01  Successfully added IP: 172.69.200.146 to pf table snort2c for blocking.
12/21/2023-19:32:25.751367  Thread: W#01  Successfully killed any open states for IP: 172.69.200.146, so any stateful traffic is blocked.
12/21/2023-19:32:25.421183  Thread: W#01  DST IP: 10.10.33.2 covered by Pass List entry 10.10.33.0/29 - not blocking.

No idea why things start working when I turn on pass list debugging, but I get blocks when I'm not running debugging. It is also sporadic. Some interfaces are working and others are blocking IPs on the pass list. I'll keep trying to test the best I'm able.

btspce

@sgnoc @bmeeks
Im also seeing this now after upgrade from 23.05.1 to 23.09.1 7.2.0_3.
We had our internal dns server blocked today despite being on the IP Pass List. We did not observe this before this upgrade.

This was also the reason why we were seeing wan failover on our 6100 HA setup. The CARP VIP WAN IP was blocked by suricata when it started.
No extra suricata processes is running. This must be another bug in this release.

sgnoc

@btspce I was only seeing IPs that were part of a netblock listed in the IP Pass List, but yesterday also had my DNS blocked, which brought the whole network down. The DNS IP was specifically listed, presumably because it is listed as the DNS for pfSense. So it isn't an issue with just the subnets listed in the IP Pass List, but also the individual IPs. That's a little different from my original thoughts on the bug.