Suricata blocking IPs on passlist, legacy mode blocking both

[bmeeks]

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

The original issue I had with Suricata blocking IPs was not occurring on my WAN, but only the internal interfaces.

Don't get hung up focusing on just the WAN. I used it in my explanations simply as an example of what's happening in the big picture. If you look in the suricata.log for your instances you will see that apparently all of the firewall interfaces are exhibiting that behavior. I see OpenVPN IPs, a WG Tunnel, etc., in the log messages. Anytime you see a message logged from the "IM#01" interface monitoring thread, that's one from the thread receiving the kernel routing messages. That thread will receive notifications for any IP address assigned on the firewall. That includes interfaces such as WAN and LAN, but also things like Virtual IPs, OpenVPN interfaces, etc.

I see those kinds of messages for a ton of firewall interfaces, and they show the IPs being added, then deleted, then added back again. I don't know why that is happening, but I can certainly see how that behavior can lead to both the WAN and internal interface IPs getting blocked because for brief instances of time those addresses are indeed not present in the in-memory Radix Tree that comprises what we call the "pass list".

Suricata uses a large common packet queue for processing. Packets are obtained off the wire and placed into this queue by packet acquisition threads. Then running worker threads service that queue by pulling out a queued up packet and processing it against the signatures. The worker thread then either generates an alert- or not- for that packet, then it goes back and grabs another packet from the common queue to process. You can see how in this process a packet with one of your interface IPs could get placed into the queue, then shortly thereafter the kernel for some reason deletes the address from that interface (and then later adds it back). If one of the worker threads happens to pull out that interface's packet to process during the brief interval where the kernel had "deleted the IP", then the alert would cause a block because at that instant the interface's IP is not in the internal pass list. Later it gets put back and some other worker thread analyzing a packet would find the interface IP in the internal pass list and not block it. But then at some point, the process of deletion recurs and you are back to being blocked. This is what I see happening when I correlate the data from both the suricata.log and the Pass List Debug log.

Here is an example:

12/24/2023-19:11:38.942701  Thread: W#01  DST IP: <WAN IP> covered by Pass List entry <WAN IP>/32 - not blocking.

....

12/24/2023-19:11:53.817090  Thread: W#03  DST IP: <WAN IP> did not match any Pass List entry, so adding to block list.

Looking at the above log snippets, you will see that at 19:11:38 Worker thread W#01 tested the WAN IP address and found it covered by a Pass List entry and thus did not block it. But then at 19:11:53 Worker thread W#03 processed a packet containing your WAN IP as the destination IP address, and when that thread tested your WAN IP against the Pass List it found it not there so it blocked it. If you look through the suricata.log file, I suspect you will find an entry in that time window between 19:11:38 and 19:11:53 where the interface monitoring thread (#IM01) received a kernel notification to delete the WAN IP address from the pass list because it was being removed from the interface. I'm not saying the interface flapping stuff is the only problem, but it certainly is a major contributor to your firewall interface IPs getting blocked.

What does your pfSense system log show during this time interval (say between Suricata starting and your WAN getting blocked)? I am curious about what's getting logged about the LAGG interface and all those OpenVPN interfaces. It might simply be a CARP sensitivity thing that could be mitigated by relaxing some timing. I'm not a CARP expert, but I do recall reading something about a configuration parameter that essentially increased the length of "down time" allowed from the Primary before the Secondary decided the Primary must be dead and tries to assume the Master role. The Primary needs some time to get all those interfaces up and running and Suricata stable on them. Maybe the Secondary is not waiting long enough at startup before trying to assume the Master role?

To be honest, you seem to have a crap ton of things defined on that firewall between the GUI pass list and the kernel routing notifications. I think that's the most IP subnets I've ever seen in configurations shared by users in the past.

You are also using two configurations I have never tested with Suricata: LAGG interfaces and CARP. There comes a point where your needs may outweigh the capabilities of free software and it's time to move up to the big boys club and pay the piper for enterprise-level security .

[bmeeks]

@btspce said in Suricata blocking IPs on passlist, legacy mode blocking both:

Switching to suricata inline should overcome these issues shouldn't it ?

Inline IPS Mode does not use a pass list as one is not necessary. So, yes it would prevent permanent blocking of an interface IP. Individual packets might still get dropped for some particular rule.

Note that with Inline IPS Mode nothing will be blocked unless you explicitly change the rule action for the signatures that you want to block traffic to DROP. The default action for rule signatures is ALERT, and that will log an alert but will not block (or drop) traffic using Inline IPS Mode.

You can change rule actions either one-by-one on the RULES tab, or as groups using the features on the SID MGMT tab. There is Sticky Post or two at the top of this sub-forum explaining how to do that.

[btspce]

@bmeeks We are using traffic shaping (limiters) to combat bufferbloat and limit other traffic. Are there still issues running suricata in inline mode with limiters ?

[bmeeks]

@btspce said in Suricata blocking IPs on passlist, legacy mode blocking both:

@bmeeks We are using traffic shaping (limiters) to combat bufferbloat and limit other traffic. Are there still issues running suricata in inline mode with limiters ?

Yes, limiters and the netmap device used by Inline IPS Mode in Suricata and Snort do not like each other and will not play well together.

Do you have an asymmetrical ISP connection? Typically buffer bloat is not nearly as big an issue on symmetrical connections (same speed for both upload and download).

[btspce]

@bmeeks Better up.. dual asymmetrical ISP connections :)
These issues did not seem like an easy fix so I will try to migrate the suricata wan instance to an internal interface instead (were no limiters are placed at the moment) and switch to inline to overcome this.

[bmeeks]

@btspce said in Suricata blocking IPs on passlist, legacy mode blocking both:

@bmeeks Better up.. dual asymmetrical ISP connections :)
These issues did not seem like an easy fix so I will try to migrate the suricata wan instance to an internal interface instead (were no limiters are placed at the moment) and switch to inline to overcome this.

Like those of @sgnoc, your suricata.log file you sent me is full of what appears to be some type of interface flapping. I see messages repeatedly from the interface monitoring thread (IM#01) saying it received a kernel routing message to delete an IP address from the WAN (and VPN interfaces), and then a second or so later another kernel routing message saying it is adding it back.

That sort of activity is going to result in the same thing I described in a post up above whereby interface IPs on the firewall could get blocked during the interval between when the kernel deleted them from the interface and then later added them back. The interface monitoring thread (IM#01) is listening for those kernel routing messages and dutifully removes (or adds back) that interface IP from the internal Radix Tree (which is what the pass list logic uses to see if an IP address is covered by a pass list entry before blocking it).

It also seems this issue is rather limited in scope. There are users reporting in the thread that are NOT having the problem. I think at least one of those not having a problem is also using CARP (but I've read so many posts from so many different threads the past two weeks that I may not be remembering the correct info for the correct thread).

Two things changed in the recent update. First was the underlying pfSense kernel, and then of course Suricata went from version 6.x from upstream to version 7.x from upstream. Have you investigated what I suggested to @sgnoc about seeing if the CARP failover interval can be lengthened? Interfaces may be flapping because during startup Suricata and CARP are interfering with each other. Just a guess as CARP is not something I've ever tested with Suricata- and I don't have the equipment here at home to do that anyway.

[SteveITS]

@bmeeks FWIW on our HA setup, which has Suricata running on LAN not WAN, the only "IM#01" messages logged are the "Info: alert-pf: Firewall Interface IP Address Change monitoring thread IM#01 has successfully started" notice, one each on Dec. 22.

@sgnoc, @btspce There's not a log entry in System Logs about the interface actually flapping?

[bmeeks]

@SteveITS said in Suricata blocking IPs on passlist, legacy mode blocking both:

FWIW on our HA setup, which has Suricata running on LAN not WAN, the only "IM#01" messages logged are the "Info: alert-pf: Firewall Interface IP Address Change monitoring thread IM#01 has successfully started" notice, one each on Dec. 22.

This is what I would expect as normal. Perhaps also a late entry about an OpenVPN interface coming online. But the amount of deletions and additions I'm seeing in the logs provided me by @btspce and @sgnoc is extreme.

During Suricata startup, a dedicated function within the custom Legacy Blocking Module scans all the firewall interfaces, grabs the assigned IP addresses for each one, then puts those in the internal Radix Tree structure that holds the pass list info.

Then, a monitoring thread is started. That thread is responsible for keeping tabs on firewall interface IPs and adjusting membership in the pass list if a firewall interface IP changes later (after the initial startup scan). The monitoring thread does this by subscribing to kernel routing messages via a socket and then looping waiting on the kernel to send it a message. This monitoring thread logic was added several years ago in response to complaints from users whose PPPoE WAN interfaces would get blocked. This happened if the interface was cycled by the ISP and the user got a new WAN IP. Suricata would be still using the old original IP it scanned at package startup and thus might block the new WAN IP. The monitoring thread logic fixed that issue.

[sgnoc]

@bmeeks @SteveITS I don't see any logs or records of interfaces flapping anywhere outside of Suricata, which leads me to think it is something inside Suricata going on. No other services, or logs indicate any interface flapping. Each time I restart my Suricata instance, or install Suricata, it starts each of the interfaces, which is currently 7. It seems that every time one interface on Suricata is started or changed, it causes every interface to reload, which seems to be causing the flapping. At least that's the best that I can tell. So 7 interfaces causing all 7 to start would generate a whole lot of noise.

I did a complete fresh install of pfSense on my router. From what I'm best able to tell, every fresh install of Suricata now causes the WAN interface to block the WAN IP for all alerts. This continues, but is somehow fixed when I restart the pfSense router, then it begins to operate normally. There is still no WAN IP listed in the default pass list IPs, but must be on a hidden list as explained by @bmeeks in a previous post.

The internal interfaces continue to block internal IPs. I am having to manually unblock my web server, which blocks every time an alert is generated. However, when I start pass list debugging, the interface starts to block properly. It's done this 3 times now. That is my more noisy interface. My other interfaces are experiencing this same issue, but just far less frequent.

The only logs I'm finding generated are of Suricata.log which shows nothing around the time of the alert causing the block, and the block.log showing the block, as well as the alert.log showing the alert.

I'm now trying to do a manually added pass list with all of the default generated rules. My interfaces are all on a 10.10.x.0/29 or 10.10.x.0/24, so I added 10.10.0.0/16 manually to the custom pass list table. So far that has seemed to stop blocking the internal IPs, which leads me to believe there is something going on with the default pass list that is generated not functioning properly.

*** Correction, I haven't had an alert generate yet to see if adding the custom pass list subnet works. The alerts I had are all older than me making that change, so I don't know if it has made a difference or the same result.

[bmeeks]

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

There is still no WAN IP listed in the default pass list IPs

I've stated this several times -- you will NEVER see any firewall interface IP (including the WAN IP) listed in the Pass List you see in the GUI (default or otherwise unless you manually add it as a specific extra entry). That's not how the logic works since several years ago. The custom blocking module queries pfSense itself at startup to get a list of all the firewall interface IPs (including the WAN IP). It then adds those to the Radix Tree. A Radix Tree is a fast-search software structure that is used to store IP addresses and netblocks for quick lookup and testing. Here is some additional information about them: https://medium.com/@rakesht2499/ip-model-radix-tree-implementation-cc4485e68a9e. All IPs from the GUI pass list along with those queried directly from pfSense go into the Radix Tree. The Radix Tree becomes the pass list internally.

Your problem is the frequent deletions and additions of firewall interface IPs. Suricata does not do that by itself in a vacuum. It subscribes to pfSense kernel routing messages for that info as described here: https://man.freebsd.org/cgi/man.cgi?query=route&apropos=0&sektion=4&manpath=FreeBSD+14.0-RELEASE&arch=default&format=html. It then acts upon the routing messages sent to it by the kernel. I don't know why your configuration and that of @btspce is doing the repeated deletions and additions. I have never seen that before.

[bmeeks]

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

This continues, but is somehow fixed when I restart the pfSense router, then it begins to operate normally.

When you do this, are you still seeing the same of number of "deleted IP address xxxx" and "added IP address xxxx" in the suricata.log file for the interface?

I see basically about 1 per second in the previous logs you supplied. There should be zero normally - or 1 or 2 at most if a VPN or something comes online.

You comments lead me to believe you are missing the point here. Quit worrying about the pass list and fiddling around with changes there. Your problem is the frequent interface IP changes showing in the suricata.log. You need to find out why that is happening in your configuration. @SteveITS says he does not see that, and he has SG-6100s in HA mode if I understand him correctly. You and @btspce are reporting the same issue, but I see the same interface flapping stuff in both of your logs. Again, not seeing that behavior in reports from others.

[sgnoc]

@bmeeks I'm not missing the point, quite the contrary I have been trying to figure out how to find the root cause, since this has only started being an issue with the most recent versions of Suricata. Since I have owned my XG-7100 from 2019 I have not had any issues until now. This is the rest of the sentence you quoted, but left out:

but must be on a hidden list as explained by @bmeeks in a previous post.

The flapping interface ONLY happens when Suricata is initializing. At no other time do I see these deletions in the Suricata.log. I had 4 of these instances with the WAN IP when I restarted from scratch on my system. I don't see how your claim of the deletions being the issue, because when the WAN interface has the issues (only after a fresh Suricata install BEFORE rebooting the system after completing setup), the flapping ceases but the interface continues to block EVERY alert to include the SRC and DST including the WAN IP. If this was just the monitor threads catching the IP between deletion and addition, then it would not occur after the flapping stops once initialization is completed. Again, the flapping only occurs during the Suricata initialization. So if that were the case, the WAN IP would not longer be blocked because the deletions have halted. Also, after I have everything setup and complete a full system reboot, the WAN interface blocking the WAN IP issue goes away and it is just the other interfaces blocking internal IPs after that point. There are two issues I see here. The primary issue is internal IPs being blocked on my internal Suricata interfaces, despite the subnets being in the default pass list, and there are no logged flapping of interface IPs on the internal interface Suricata instances. The second issue is that currently with the latest Suricata package, after a fresh install of Suricata, the WAN interfaces is not behaving as expected, where ther are a larger number of the flapping WAN IP, and the WAN IP being blocked (even after the flapping stops). This is only resolved on a full system reboot AFTER the Suricata has finished setting up from being installed and pulling new updates, etc, After the reboot, the WAN interface acts normally again and there are a minimal (4 in this case) number of deleted WAN IPs logged during the initialization of the WAN interface.

Suricata.log:

Suricata_Log_Latest_ix0.txt

Further, my internal interface consisting of my public facing web server has zero deletions of any related subnets from that interface, which are 10.10.33.0/29. It also catches the 4 WAN interface deletions/additions, but again after the initialization of Suricata, it clears up. I get it that you say Suricata doesn't control that, but it has to be related to Suricata's initialization, because that is the only time it occurs. Only during interface initialization of Suricata. The fact that there are exactly 4, and 4 monitors for that interface, doesn't seem to be a coincidence to me.

Here is the Suricata.log file for the 10.10.33.0/29 interface on my XG-7100:
Suricata_Log_lag0.33.txt

The only reason I've been "fiddling" with my Suricata settings, is because this is only a Suricata issue. No other logs on my system indicate any flapping, and the flapping ONLY occurs on initialization of the Suricata interfaces. That also cannot be a coincidence.

Here is the block log for my lag0.33 interface:

12/25/2023-01:23:53.793033  [Block Src] [**] [1:2032979:2] ET SCAN Yandex Webcrawler User-Agent (YandexBot) [**] [Classification: Not Suspicious Traffic] [Priority: 3] {TCP} 162.158.238.72:45214
12/25/2023-01:23:53.793033  [Block Dst] [**] [1:2032979:2] ET SCAN Yandex Webcrawler User-Agent (YandexBot) [**] [Classification: Not Suspicious Traffic] [Priority: 3] {TCP} 10.10.33.2:80
12/25/2023-12:24:43.090673  [Block Src] [**] [1:2853548:1] ETPRO HUNTING Suspicious Empty Referer Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 141.101.105.12:28970
12/25/2023-12:24:43.090673  [Block Dst] [**] [1:2853548:1] ETPRO HUNTING Suspicious Empty Referer Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 10.10.33.2:80
12/25/2023-20:21:18.406299  [Block Src] [**] [1:2853568:1] ETPRO HUNTING Suspicious Empty Sec-CH-UA Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 162.158.159.59:64511
12/25/2023-20:21:18.406299  [Block Dst] [**] [1:2853568:1] ETPRO HUNTING Suspicious Empty Sec-CH-UA Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 10.10.33.2:80
12/25/2023-20:21:18.542709  [Block Src] [**] [1:2853568:1] ETPRO HUNTING Suspicious Empty Sec-CH-UA Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 162.158.158.220:45036
12/25/2023-20:21:18.400555  [Block Src] [**] [1:2853568:1] ETPRO HUNTING Suspicious Empty Sec-CH-UA Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 162.158.159.202:65230
12/25/2023-20:21:18.415277  [Block Src] [**] [1:2853568:1] ETPRO HUNTING Suspicious Empty Sec-CH-UA Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 162.158.159.197:31129
12/26/2023-03:45:26.912634  [Block Src] [**] [1:2049255:1] ET SCAN LeakIX Inbound User-Agent [**] [Classification: Misc activity] [Priority: 3] {TCP} 172.71.242.59:65252
12/26/2023-03:45:26.912634  [Block Dst] [**] [1:2049255:1] ET SCAN LeakIX Inbound User-Agent [**] [Classification: Misc activity] [Priority: 3] {TCP} 10.10.33.2:80

Here is the Default Pass List:

10.10.5.0/24
10.10.5.101/32
10.10.6.0/24
10.10.7.0/24
10.10.8.0/24
10.10.9.0/24
10.10.10.0/24
10.10.11.0/24
10.10.15.0/24
10.10.25.0/24
10.10.31.0/29
10.10.32.0/29
10.10.33.0/29
10.10.34.0/29
10.10.35.0/29
10.10.36.0/29
10.10.37.0/29
10.10.45.0/24
10.10.55.0/24
10.10.60.0/29
<WAN Gateway>/32
fe80:6::/64
fe80:7::/64
fe80:8::/64
fe80:9::/64
fe80:10::/64

With no other logs generated from pfSense that show any interfaces going down/up, I don't see where else I may be able to look outside of this being a specific Suricata phenomenon. If there is any guidance where else I may be able to look, I would certainly give it a try. This is netgate hardware that has been running for years without these issues until Suricata 7, and it is only affecting Suricata. No hardware changes have been made, and only regular maintenance of updating packages and software have occurred to keep things on the most recent stable releases.

[bmeeks]

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

This is netgate hardware that has been running for years without these issues until Suricata 7, and it is only affecting Suricata.

The pfSense kernel also changed with the Suricata update. The new Suricata version came as part of the 23.09 pfSense Plus and 2.7.1 CE rollouts. So, more than just Suricata has changed.

My advice at this point is to remove Suricata from your environment. That will definitely solve your unwanted blocking issues.

[sgnoc]

@bmeeks That's great, thanks for the help. Hopefully there are only a limited number of people with this problem and not a larger portion of the Suricata community. Thanks for your maintaining this package.

So does anyone else monitoring this topic have any advice on trying to track this issue down? I see zero evidence in any log anywhere outside of Suricata of flapping interfaces, so any advice on where/how to locate some would be welcome.

I also see no noticeable issues with pfSense or other packages outside of IPs that are being blocked only on internal interfaces where Suricata is configured as legacy blocking mode. I'm trying to add an all inclusive 10.10.0.0/16 to a manually created Pass List, with all the default generated IPs as well, but am waiting for an alert to trigger to see if there is any difference. All the default pass lists have the subnet of these interfaces added, but are still blocking internal IP addresses. IE: an interface on the subnet 10.10.33.0/29 with that subnet on the default pass list is blocking IP 10.10.33.2.

I have tried to enable pass list debugging for further details on troubleshooting, but every time I enable debugging, the interfaces start operating normally. While this does not make sense since to me, since debugging should only be printing additional lines to the log files, it has been tried 3 separate times with the same result. I would simply keep the debugging mode on, but don't know how to get that generated log rotated with the others to prevent my drive from filling up, otherwise that would be a solution to the problem, and each update of Suricata going forward would reset debugging back to no.

Willing to try any additional troubleshooting options that anyone has ideas or advice on. Thanks.

*** Edit *** Finally got an alert on the 10.10.33.0/29 interface and with the 10.10.0.0/16 added to a custom pass list, the internal 10.10.33.2 IP is still blocked, which matches with some previous poster's experience.

[SteveITS]

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

every time I enable debugging, the interfaces start operating normally

Some sort of threading or timing issue? As in, in code simulate the log write by adding a 10ms wait and see if it still happens? (totally brainstorming here)

[sgnoc]

@SteveITS Any idea if that is something I can do from my end either in gui or in a SSH session?

I'm willing to give anything a shot. It doesn't take me too long to restore the whole system with a base image if I have a bad enough failure. I also found in another topic a few years back where disabling hardware checksum offloading fixed the same issue that was popping up then. It only worked for a few, but I'm giving it a shot (especially since I thought I had it already disabled). I just checked to disable it and restarted the system.

Interesting that after the restart, I had zero logs in any of my interfaces where an interface IP was deleted, so nothing caused any interface flapping at all.

[SteveITS]

@sgnoc I’d expect it’d be in the code right where it’s writing the log file. Imagining:

If (passlist log)
write log
Else
If (passlist wait)
sleep
End

Re checksum offload, I am pretty sure we’ve had that off per our setup doc from years ago because of false positives.

[sgnoc]

@SteveITS @btspce I've had some luck with my config now. I have disabled the checksum offloading (System->Advanced->Networking->Check "Disable checksum offloading") and adding an all encompassing subnet (10.10.0.0/16 vs all of /29 and /24 10.10.x.0 subnets) to a custom pass list. Last night I had multiple blocks on the interface that has been blocking my web server, and the web server address was not blocked this round.

I'm now putting the pass list back to default and letting it run today to see if it was a combination of the two changes or just the disable of checksum offloading. I thought I had already disabled it (I know I had at some point in the past), but can't remember if I enabled it for one reason or another sometime later, but it hasn't been modified in at least a year or two.

I found the hint on disabling checksum offloading on a previous topic from about 2020 where it was helping some posters with almost an identical issue as what we are seeing here, so definitely an issue that has occurred before if it is the same, and they found it in an even older post from back then. Seems to be something that only affects a portion of pfSense/Suricata users and only periodically. This doesn't fix it for everyone, but it's worth a try if you're setting is enabled (I know you said you have had yours off for years @SteveITS).

And while I was writing this, I had some additional alerts on my interface, after setting it back to the default pass list. It appears that just disabling hardware checksum offloading has done the trick for me. I'll continue watching the interfaces, but I think my issue may now be resolved.

Seems to be a result of the pfSense and Suricata updates and having hardware checksum offloading enabled that caused the instability. Again, I haven't modified the offloading settings in at least a year or two, so the updates changed something on my end that caused the compatibility issue on netgate hardware (running XG-7100-1U).

Hope this helps someone with similar circumstances! I'll post again if I notice anything more.

[btspce]

@sgnoc Great find! We are also running with hardware checksum offloading on the Netgate 6100 pair as the only thing enabled.
My understanding was that hardware checksum offloading only has to be disabled when running inline (and it has been working until now through all versions).
This could perhaps be a nic driver issue in the newer version of pfsense. Will Disable Hardware Checksum Offloading later today and report back.

[bmeeks]

Here is the pertinent information about the various hardware offloading options available for NICs and how each impacts Suricata:

https://docs.suricata.io/en/suricata-7.0.2/performance/packet-capture.html#offloading

It's best if all offload options for NIC hardware are set to "off". Legacy Blocking Mode in Suricata (and Snort) uses PCAP to for packet acquisition. Inline IPS Mode uses Netmap. Note in the link above that for both methods all hardware offloading options should be disabled.