Suricata blocking IPs on passlist, legacy mode blocking both

sgnoc

One thought just came to mind. I believe this specific alert/block happened while I was loading suricata on the other interfaces, which were not completely up yet.

It seems that each time I load/reload a single suricata interface, it causes all of the other suricata instances to reload some portions. Could that be causing Suricata to think the IPs are being updated when they aren't?

Each time I reload a single interface, I see these logs in the main system log, and there is a set of these for ALL interfaces by a single interface restart/start. I never understood why restarting one interface did this for all interfaces.

2023-12-24 19:31:04.068717-05:00 	php 	86831 	[Suricata] Building new sid-msg.map file for 00_WAN...
2023-12-24 19:31:03.828809-05:00 	php 	86831 	[Suricata] Enabling any flowbit-required rules for: 00_WAN...
2023-12-24 19:30:57.681301-05:00 	php 	86831 	[Suricata] Updating rules configuration for: 00_WAN ...

I just checked my other interface, the lag0.33. Where it was blocking the internal 10.10.33.2 address before I turned on the pass list debugging, now that pass list debugging is on, the 10.10.33.2 address is no longer being blocked when an alert hits. I've had two alerts hit since I restarted the interface with the pass list debug enabled and it only blocked the external address, not the internal address. So something different is happening for me with the pass list debugging that is not triggering the internal IP address to be blocked, where it does before the debug was enabled. I'm baffled.

bmeeks

@sgnoc:
When you make changes to interfaces inside pfSense, then pfSense itself will send all packages a "restart all packages" command. This is done in the event IP addresses or subnets change on configured interfaces and packages might need to know they are changing. Even when nothing "really" changes, the message is still sent. Suricata will see this command from pfSense and dutifully restart itself. It uses the shell script in /usr/local/etc/rc.d/suricata.sh to do this. That shell script restarts all configured Suricata interfaces.

Also, a few versions back Suricata upstream made some changes in how the PCAP function works. The changes were to address a bug that cropped up occasionally when the underlying interface PCAP was running on restarted. I believe a side-effecct of this bug fix is that now when Suricata starts PCAP on some interface types it will cause a brief "reset" of the interface. That might cause it to appear to have gone down and come righ back up.

When I said your WAN IP was changing, I don't literally mean a different one. I mean the kernel is cycling the interface for some reason and sending the interface monitoring thread in Suricata's custom blocking module a notification of removing and adding an IP address to an interface. The kernel sends two messages: one deleting the IP address from the interface; and then another adding it back. 9 times out of 10 it adds back the same IP. But the monitoring thread can't assume that. So, when it gets the "DELETE ADDRESS" message it removes the IP from the internal Radix Tree. Later, when it gets the "ADD ADDRESS" message, it will put it back in the tree. But in the interval between those two messages on a very busy network it's entirely possible for one of the other Suricata packet processing threads to have received a packet with the WAN IP and go ahead and block it since for a brief second it is removed from the Radix Tree and would not be found on the "pass list" when looked up.

These are the messages I'm talking about where the automatic Interface Monitoring Thread (thread ID IM#01) is removing your WAN IP from the internal pass list then later adding it back. It appears to do this multiple times. During the interval when the WAN IP has been automatically removed, then it most certainly can be blocked (and it is, according to the Pass List debugging log).

[117623 - Suricata-IM#01] 2023-12-24 19:11:23 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface ix0.
[117623 - Suricata-IM#01] 2023-12-24 19:11:23 Info: alert-pf: Deleted address <WAN IP> from automatic firewall interface IP Pass List.
[117623 - Suricata-IM#01] 2023-12-24 19:11:24 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface ix0.
[117623 - Suricata-IM#01] 2023-12-24 19:11:24 Info: alert-pf: Added address <WAN IP> to automatic firewall interface IP Pass List.
[117623 - Suricata-IM#01] 2023-12-24 19:11:25 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface ix0.
[117623 - Suricata-IM#01] 2023-12-24 19:11:25 Info: alert-pf: Deleted address <WAN IP> from automatic firewall interface IP Pass List.
[117623 - Suricata-IM#01] 2023-12-24 19:11:26 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface ix0.
[117623 - Suricata-IM#01] 2023-12-24 19:11:26 Info: alert-pf: Added address <WAN IP> to automatic firewall interface IP Pass List.

It's getting removed because the pfSense kernel is sending the monitoring thread a notification telling it that. The first line of log text is from the kernel routing message subscription. It says an IP change happened on the ix0 interface. As I said earlier, here "change" does not necessarily mean a new IP. It might simply mean the interface was flapping. Shortly thereafter the kernel sent another message about ix0 and resulted in putting the IP address back. At that point the IM#01 monitoring thread would have put the WAN IP back in the Radix Tree (same as Pass List). But then shortly thereafter, the process repeats and the address is removed again. You need to find out what's going on that is causing all the interface flapping. That's why things are getting blocked, I think.

Because Suricata is multithreaded, this can be a bit hard to visualize in your head. But consider that in your case Suricata has 4 threads processing packets, comparing them to the rule signatures, and generating alerts when something matches. Those threads are labeled in the Pass List Debug Log as W#01 through W#04. You can follow them in the Pass List Debug Log. A totally separate thread is running at the same time listening for Kernel Routing messages from pfSense. That thread is automatically adding or removing firewall interface IPs from the pass list depending on whether the kernel says "I'm deleting an IP" or "I'm adding an IP". So that thread (called #IM01, for "interface monitoring thread #01) is constantly deleting and then adding back firewall interface IP addresses in your case because something appears to be flapping (interfaces going up and down). So, now it's a random race with the four packet processing worker threads. Does one of them see a packet with your WAN IP in it during the brief interval where the interface monitoring thread has deleted the WAN IP from the internal pass list? If so, then the WAN IP gets blocked.

Looking in your Pass List Debug log I see instances where a worker thread found the WAN IP in the pass list and did NOT block it, and then later a different thread checked and the WAN IP was NOT in the pass list at that instant and thus got blocked.

sgnoc

@bmeeks That makes sense with the multiple threads and one catching an alert between deleting and readding the IP to the list. I'm not really sure how to go about finding what would cause the interface to flap. I haven't seen any other logs on the system that would indicate that to be the case, so this is the first I have found that to be happening. The original issue I had with Suricata blocking IPs was not occurring on my WAN, but only the internal interfaces.

I only had the issue happen once on the WAN, when I reinstalled Suricata. You mentioned at the time to remove and then install rather than reinstall Suricata. I decided to start with a fresh install on Suricata this go around and the WAN IP being blocked popped back up. I can't restart again right now, but I'm planning to try and see if I can restart again with it completely set back up to see if the issue on the WAN goes away again or if it is still persistent.

I'll start going through different logs and see if anything jumps out to show what could be causing the flapping.

btspce

@sgnoc We have the same interface flapping in the suricata logs sent to @bmeeks since going to suricata 7.
But CARP/HA does not set off any warnings on the same interfaces when not running suricata and no issues are observed otherwise. Switch does not log any interface flapping either.
Suricata blocks WAN VIP within seconds of being started on WAN but it also blocks internal IPs randomly after running for hours without issue on an internal interface.

We are using CARP/HA on these interfaces and were not having any issues before suricata 7.
@bmeeks Switching to suricata inline should overcome these issues shouldn't it ?

bmeeks

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

The original issue I had with Suricata blocking IPs was not occurring on my WAN, but only the internal interfaces.

Don't get hung up focusing on just the WAN. I used it in my explanations simply as an example of what's happening in the big picture. If you look in the suricata.log for your instances you will see that apparently all of the firewall interfaces are exhibiting that behavior. I see OpenVPN IPs, a WG Tunnel, etc., in the log messages. Anytime you see a message logged from the "IM#01" interface monitoring thread, that's one from the thread receiving the kernel routing messages. That thread will receive notifications for any IP address assigned on the firewall. That includes interfaces such as WAN and LAN, but also things like Virtual IPs, OpenVPN interfaces, etc.

I see those kinds of messages for a ton of firewall interfaces, and they show the IPs being added, then deleted, then added back again. I don't know why that is happening, but I can certainly see how that behavior can lead to both the WAN and internal interface IPs getting blocked because for brief instances of time those addresses are indeed not present in the in-memory Radix Tree that comprises what we call the "pass list".

Suricata uses a large common packet queue for processing. Packets are obtained off the wire and placed into this queue by packet acquisition threads. Then running worker threads service that queue by pulling out a queued up packet and processing it against the signatures. The worker thread then either generates an alert- or not- for that packet, then it goes back and grabs another packet from the common queue to process. You can see how in this process a packet with one of your interface IPs could get placed into the queue, then shortly thereafter the kernel for some reason deletes the address from that interface (and then later adds it back). If one of the worker threads happens to pull out that interface's packet to process during the brief interval where the kernel had "deleted the IP", then the alert would cause a block because at that instant the interface's IP is not in the internal pass list. Later it gets put back and some other worker thread analyzing a packet would find the interface IP in the internal pass list and not block it. But then at some point, the process of deletion recurs and you are back to being blocked. This is what I see happening when I correlate the data from both the suricata.log and the Pass List Debug log.

Here is an example:

12/24/2023-19:11:38.942701  Thread: W#01  DST IP: <WAN IP> covered by Pass List entry <WAN IP>/32 - not blocking.

....

12/24/2023-19:11:53.817090  Thread: W#03  DST IP: <WAN IP> did not match any Pass List entry, so adding to block list.

Looking at the above log snippets, you will see that at 19:11:38 Worker thread W#01 tested the WAN IP address and found it covered by a Pass List entry and thus did not block it. But then at 19:11:53 Worker thread W#03 processed a packet containing your WAN IP as the destination IP address, and when that thread tested your WAN IP against the Pass List it found it not there so it blocked it. If you look through the suricata.log file, I suspect you will find an entry in that time window between 19:11:38 and 19:11:53 where the interface monitoring thread (#IM01) received a kernel notification to delete the WAN IP address from the pass list because it was being removed from the interface. I'm not saying the interface flapping stuff is the only problem, but it certainly is a major contributor to your firewall interface IPs getting blocked.

What does your pfSense system log show during this time interval (say between Suricata starting and your WAN getting blocked)? I am curious about what's getting logged about the LAGG interface and all those OpenVPN interfaces. It might simply be a CARP sensitivity thing that could be mitigated by relaxing some timing. I'm not a CARP expert, but I do recall reading something about a configuration parameter that essentially increased the length of "down time" allowed from the Primary before the Secondary decided the Primary must be dead and tries to assume the Master role. The Primary needs some time to get all those interfaces up and running and Suricata stable on them. Maybe the Secondary is not waiting long enough at startup before trying to assume the Master role?

To be honest, you seem to have a crap ton of things defined on that firewall between the GUI pass list and the kernel routing notifications. I think that's the most IP subnets I've ever seen in configurations shared by users in the past.

You are also using two configurations I have never tested with Suricata: LAGG interfaces and CARP. There comes a point where your needs may outweigh the capabilities of free software and it's time to move up to the big boys club and pay the piper for enterprise-level security .

bmeeks

@btspce said in Suricata blocking IPs on passlist, legacy mode blocking both:

Switching to suricata inline should overcome these issues shouldn't it ?

Inline IPS Mode does not use a pass list as one is not necessary. So, yes it would prevent permanent blocking of an interface IP. Individual packets might still get dropped for some particular rule.

Note that with Inline IPS Mode nothing will be blocked unless you explicitly change the rule action for the signatures that you want to block traffic to DROP. The default action for rule signatures is ALERT, and that will log an alert but will not block (or drop) traffic using Inline IPS Mode.

You can change rule actions either one-by-one on the RULES tab, or as groups using the features on the SID MGMT tab. There is Sticky Post or two at the top of this sub-forum explaining how to do that.

btspce

@bmeeks We are using traffic shaping (limiters) to combat bufferbloat and limit other traffic. Are there still issues running suricata in inline mode with limiters ?

bmeeks

@btspce said in Suricata blocking IPs on passlist, legacy mode blocking both:

@bmeeks We are using traffic shaping (limiters) to combat bufferbloat and limit other traffic. Are there still issues running suricata in inline mode with limiters ?

Yes, limiters and the netmap device used by Inline IPS Mode in Suricata and Snort do not like each other and will not play well together.

Do you have an asymmetrical ISP connection? Typically buffer bloat is not nearly as big an issue on symmetrical connections (same speed for both upload and download).

btspce

@bmeeks Better up.. dual asymmetrical ISP connections :)
These issues did not seem like an easy fix so I will try to migrate the suricata wan instance to an internal interface instead (were no limiters are placed at the moment) and switch to inline to overcome this.

bmeeks

@btspce said in Suricata blocking IPs on passlist, legacy mode blocking both:

@bmeeks Better up.. dual asymmetrical ISP connections :)
These issues did not seem like an easy fix so I will try to migrate the suricata wan instance to an internal interface instead (were no limiters are placed at the moment) and switch to inline to overcome this.

Like those of @sgnoc, your suricata.log file you sent me is full of what appears to be some type of interface flapping. I see messages repeatedly from the interface monitoring thread (IM#01) saying it received a kernel routing message to delete an IP address from the WAN (and VPN interfaces), and then a second or so later another kernel routing message saying it is adding it back.

That sort of activity is going to result in the same thing I described in a post up above whereby interface IPs on the firewall could get blocked during the interval between when the kernel deleted them from the interface and then later added them back. The interface monitoring thread (IM#01) is listening for those kernel routing messages and dutifully removes (or adds back) that interface IP from the internal Radix Tree (which is what the pass list logic uses to see if an IP address is covered by a pass list entry before blocking it).

It also seems this issue is rather limited in scope. There are users reporting in the thread that are NOT having the problem. I think at least one of those not having a problem is also using CARP (but I've read so many posts from so many different threads the past two weeks that I may not be remembering the correct info for the correct thread).

Two things changed in the recent update. First was the underlying pfSense kernel, and then of course Suricata went from version 6.x from upstream to version 7.x from upstream. Have you investigated what I suggested to @sgnoc about seeing if the CARP failover interval can be lengthened? Interfaces may be flapping because during startup Suricata and CARP are interfering with each other. Just a guess as CARP is not something I've ever tested with Suricata- and I don't have the equipment here at home to do that anyway.

SteveITS

@bmeeks FWIW on our HA setup, which has Suricata running on LAN not WAN, the only "IM#01" messages logged are the "Info: alert-pf: Firewall Interface IP Address Change monitoring thread IM#01 has successfully started" notice, one each on Dec. 22.

@sgnoc, @btspce There's not a log entry in System Logs about the interface actually flapping?

bmeeks

@SteveITS said in Suricata blocking IPs on passlist, legacy mode blocking both:

FWIW on our HA setup, which has Suricata running on LAN not WAN, the only "IM#01" messages logged are the "Info: alert-pf: Firewall Interface IP Address Change monitoring thread IM#01 has successfully started" notice, one each on Dec. 22.

This is what I would expect as normal. Perhaps also a late entry about an OpenVPN interface coming online. But the amount of deletions and additions I'm seeing in the logs provided me by @btspce and @sgnoc is extreme.

During Suricata startup, a dedicated function within the custom Legacy Blocking Module scans all the firewall interfaces, grabs the assigned IP addresses for each one, then puts those in the internal Radix Tree structure that holds the pass list info.

Then, a monitoring thread is started. That thread is responsible for keeping tabs on firewall interface IPs and adjusting membership in the pass list if a firewall interface IP changes later (after the initial startup scan). The monitoring thread does this by subscribing to kernel routing messages via a socket and then looping waiting on the kernel to send it a message. This monitoring thread logic was added several years ago in response to complaints from users whose PPPoE WAN interfaces would get blocked. This happened if the interface was cycled by the ISP and the user got a new WAN IP. Suricata would be still using the old original IP it scanned at package startup and thus might block the new WAN IP. The monitoring thread logic fixed that issue.

sgnoc

@bmeeks @SteveITS I don't see any logs or records of interfaces flapping anywhere outside of Suricata, which leads me to think it is something inside Suricata going on. No other services, or logs indicate any interface flapping. Each time I restart my Suricata instance, or install Suricata, it starts each of the interfaces, which is currently 7. It seems that every time one interface on Suricata is started or changed, it causes every interface to reload, which seems to be causing the flapping. At least that's the best that I can tell. So 7 interfaces causing all 7 to start would generate a whole lot of noise.

I did a complete fresh install of pfSense on my router. From what I'm best able to tell, every fresh install of Suricata now causes the WAN interface to block the WAN IP for all alerts. This continues, but is somehow fixed when I restart the pfSense router, then it begins to operate normally. There is still no WAN IP listed in the default pass list IPs, but must be on a hidden list as explained by @bmeeks in a previous post.

The internal interfaces continue to block internal IPs. I am having to manually unblock my web server, which blocks every time an alert is generated. However, when I start pass list debugging, the interface starts to block properly. It's done this 3 times now. That is my more noisy interface. My other interfaces are experiencing this same issue, but just far less frequent.

The only logs I'm finding generated are of Suricata.log which shows nothing around the time of the alert causing the block, and the block.log showing the block, as well as the alert.log showing the alert.

I'm now trying to do a manually added pass list with all of the default generated rules. My interfaces are all on a 10.10.x.0/29 or 10.10.x.0/24, so I added 10.10.0.0/16 manually to the custom pass list table. So far that has seemed to stop blocking the internal IPs, which leads me to believe there is something going on with the default pass list that is generated not functioning properly.

*** Correction, I haven't had an alert generate yet to see if adding the custom pass list subnet works. The alerts I had are all older than me making that change, so I don't know if it has made a difference or the same result.

bmeeks

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

There is still no WAN IP listed in the default pass list IPs

I've stated this several times -- you will NEVER see any firewall interface IP (including the WAN IP) listed in the Pass List you see in the GUI (default or otherwise unless you manually add it as a specific extra entry). That's not how the logic works since several years ago. The custom blocking module queries pfSense itself at startup to get a list of all the firewall interface IPs (including the WAN IP). It then adds those to the Radix Tree. A Radix Tree is a fast-search software structure that is used to store IP addresses and netblocks for quick lookup and testing. Here is some additional information about them: https://medium.com/@rakesht2499/ip-model-radix-tree-implementation-cc4485e68a9e. All IPs from the GUI pass list along with those queried directly from pfSense go into the Radix Tree. The Radix Tree becomes the pass list internally.

Your problem is the frequent deletions and additions of firewall interface IPs. Suricata does not do that by itself in a vacuum. It subscribes to pfSense kernel routing messages for that info as described here: https://man.freebsd.org/cgi/man.cgi?query=route&apropos=0&sektion=4&manpath=FreeBSD+14.0-RELEASE&arch=default&format=html. It then acts upon the routing messages sent to it by the kernel. I don't know why your configuration and that of @btspce is doing the repeated deletions and additions. I have never seen that before.

bmeeks

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

This continues, but is somehow fixed when I restart the pfSense router, then it begins to operate normally.

When you do this, are you still seeing the same of number of "deleted IP address xxxx" and "added IP address xxxx" in the suricata.log file for the interface?

I see basically about 1 per second in the previous logs you supplied. There should be zero normally - or 1 or 2 at most if a VPN or something comes online.

You comments lead me to believe you are missing the point here. Quit worrying about the pass list and fiddling around with changes there. Your problem is the frequent interface IP changes showing in the suricata.log. You need to find out why that is happening in your configuration. @SteveITS says he does not see that, and he has SG-6100s in HA mode if I understand him correctly. You and @btspce are reporting the same issue, but I see the same interface flapping stuff in both of your logs. Again, not seeing that behavior in reports from others.

sgnoc

@bmeeks I'm not missing the point, quite the contrary I have been trying to figure out how to find the root cause, since this has only started being an issue with the most recent versions of Suricata. Since I have owned my XG-7100 from 2019 I have not had any issues until now. This is the rest of the sentence you quoted, but left out:

but must be on a hidden list as explained by @bmeeks in a previous post.

The flapping interface ONLY happens when Suricata is initializing. At no other time do I see these deletions in the Suricata.log. I had 4 of these instances with the WAN IP when I restarted from scratch on my system. I don't see how your claim of the deletions being the issue, because when the WAN interface has the issues (only after a fresh Suricata install BEFORE rebooting the system after completing setup), the flapping ceases but the interface continues to block EVERY alert to include the SRC and DST including the WAN IP. If this was just the monitor threads catching the IP between deletion and addition, then it would not occur after the flapping stops once initialization is completed. Again, the flapping only occurs during the Suricata initialization. So if that were the case, the WAN IP would not longer be blocked because the deletions have halted. Also, after I have everything setup and complete a full system reboot, the WAN interface blocking the WAN IP issue goes away and it is just the other interfaces blocking internal IPs after that point. There are two issues I see here. The primary issue is internal IPs being blocked on my internal Suricata interfaces, despite the subnets being in the default pass list, and there are no logged flapping of interface IPs on the internal interface Suricata instances. The second issue is that currently with the latest Suricata package, after a fresh install of Suricata, the WAN interfaces is not behaving as expected, where ther are a larger number of the flapping WAN IP, and the WAN IP being blocked (even after the flapping stops). This is only resolved on a full system reboot AFTER the Suricata has finished setting up from being installed and pulling new updates, etc, After the reboot, the WAN interface acts normally again and there are a minimal (4 in this case) number of deleted WAN IPs logged during the initialization of the WAN interface.

Suricata.log:

Suricata_Log_Latest_ix0.txt

Further, my internal interface consisting of my public facing web server has zero deletions of any related subnets from that interface, which are 10.10.33.0/29. It also catches the 4 WAN interface deletions/additions, but again after the initialization of Suricata, it clears up. I get it that you say Suricata doesn't control that, but it has to be related to Suricata's initialization, because that is the only time it occurs. Only during interface initialization of Suricata. The fact that there are exactly 4, and 4 monitors for that interface, doesn't seem to be a coincidence to me.

Here is the Suricata.log file for the 10.10.33.0/29 interface on my XG-7100:
Suricata_Log_lag0.33.txt

The only reason I've been "fiddling" with my Suricata settings, is because this is only a Suricata issue. No other logs on my system indicate any flapping, and the flapping ONLY occurs on initialization of the Suricata interfaces. That also cannot be a coincidence.

Here is the block log for my lag0.33 interface:

12/25/2023-01:23:53.793033  [Block Src] [**] [1:2032979:2] ET SCAN Yandex Webcrawler User-Agent (YandexBot) [**] [Classification: Not Suspicious Traffic] [Priority: 3] {TCP} 162.158.238.72:45214
12/25/2023-01:23:53.793033  [Block Dst] [**] [1:2032979:2] ET SCAN Yandex Webcrawler User-Agent (YandexBot) [**] [Classification: Not Suspicious Traffic] [Priority: 3] {TCP} 10.10.33.2:80
12/25/2023-12:24:43.090673  [Block Src] [**] [1:2853548:1] ETPRO HUNTING Suspicious Empty Referer Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 141.101.105.12:28970
12/25/2023-12:24:43.090673  [Block Dst] [**] [1:2853548:1] ETPRO HUNTING Suspicious Empty Referer Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 10.10.33.2:80
12/25/2023-20:21:18.406299  [Block Src] [**] [1:2853568:1] ETPRO HUNTING Suspicious Empty Sec-CH-UA Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 162.158.159.59:64511
12/25/2023-20:21:18.406299  [Block Dst] [**] [1:2853568:1] ETPRO HUNTING Suspicious Empty Sec-CH-UA Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 10.10.33.2:80
12/25/2023-20:21:18.542709  [Block Src] [**] [1:2853568:1] ETPRO HUNTING Suspicious Empty Sec-CH-UA Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 162.158.158.220:45036
12/25/2023-20:21:18.400555  [Block Src] [**] [1:2853568:1] ETPRO HUNTING Suspicious Empty Sec-CH-UA Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 162.158.159.202:65230
12/25/2023-20:21:18.415277  [Block Src] [**] [1:2853568:1] ETPRO HUNTING Suspicious Empty Sec-CH-UA Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 162.158.159.197:31129
12/26/2023-03:45:26.912634  [Block Src] [**] [1:2049255:1] ET SCAN LeakIX Inbound User-Agent [**] [Classification: Misc activity] [Priority: 3] {TCP} 172.71.242.59:65252
12/26/2023-03:45:26.912634  [Block Dst] [**] [1:2049255:1] ET SCAN LeakIX Inbound User-Agent [**] [Classification: Misc activity] [Priority: 3] {TCP} 10.10.33.2:80

Here is the Default Pass List:

10.10.5.0/24
10.10.5.101/32
10.10.6.0/24
10.10.7.0/24
10.10.8.0/24
10.10.9.0/24
10.10.10.0/24
10.10.11.0/24
10.10.15.0/24
10.10.25.0/24
10.10.31.0/29
10.10.32.0/29
10.10.33.0/29
10.10.34.0/29
10.10.35.0/29
10.10.36.0/29
10.10.37.0/29
10.10.45.0/24
10.10.55.0/24
10.10.60.0/29
<WAN Gateway>/32
fe80:6::/64
fe80:7::/64
fe80:8::/64
fe80:9::/64
fe80:10::/64

With no other logs generated from pfSense that show any interfaces going down/up, I don't see where else I may be able to look outside of this being a specific Suricata phenomenon. If there is any guidance where else I may be able to look, I would certainly give it a try. This is netgate hardware that has been running for years without these issues until Suricata 7, and it is only affecting Suricata. No hardware changes have been made, and only regular maintenance of updating packages and software have occurred to keep things on the most recent stable releases.

bmeeks

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

This is netgate hardware that has been running for years without these issues until Suricata 7, and it is only affecting Suricata.

The pfSense kernel also changed with the Suricata update. The new Suricata version came as part of the 23.09 pfSense Plus and 2.7.1 CE rollouts. So, more than just Suricata has changed.

My advice at this point is to remove Suricata from your environment. That will definitely solve your unwanted blocking issues.

sgnoc

@bmeeks That's great, thanks for the help. Hopefully there are only a limited number of people with this problem and not a larger portion of the Suricata community. Thanks for your maintaining this package.

So does anyone else monitoring this topic have any advice on trying to track this issue down? I see zero evidence in any log anywhere outside of Suricata of flapping interfaces, so any advice on where/how to locate some would be welcome.

I also see no noticeable issues with pfSense or other packages outside of IPs that are being blocked only on internal interfaces where Suricata is configured as legacy blocking mode. I'm trying to add an all inclusive 10.10.0.0/16 to a manually created Pass List, with all the default generated IPs as well, but am waiting for an alert to trigger to see if there is any difference. All the default pass lists have the subnet of these interfaces added, but are still blocking internal IP addresses. IE: an interface on the subnet 10.10.33.0/29 with that subnet on the default pass list is blocking IP 10.10.33.2.

I have tried to enable pass list debugging for further details on troubleshooting, but every time I enable debugging, the interfaces start operating normally. While this does not make sense since to me, since debugging should only be printing additional lines to the log files, it has been tried 3 separate times with the same result. I would simply keep the debugging mode on, but don't know how to get that generated log rotated with the others to prevent my drive from filling up, otherwise that would be a solution to the problem, and each update of Suricata going forward would reset debugging back to no.

Willing to try any additional troubleshooting options that anyone has ideas or advice on. Thanks.

*** Edit *** Finally got an alert on the 10.10.33.0/29 interface and with the 10.10.0.0/16 added to a custom pass list, the internal 10.10.33.2 IP is still blocked, which matches with some previous poster's experience.

SteveITS

@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:

every time I enable debugging, the interfaces start operating normally

Some sort of threading or timing issue? As in, in code simulate the log write by adding a 10ms wait and see if it still happens? (totally brainstorming here)

sgnoc

@SteveITS Any idea if that is something I can do from my end either in gui or in a SSH session?

I'm willing to give anything a shot. It doesn't take me too long to restore the whole system with a base image if I have a bad enough failure. I also found in another topic a few years back where disabling hardware checksum offloading fixed the same issue that was popping up then. It only worked for a few, but I'm giving it a shot (especially since I thought I had it already disabled). I just checked to disable it and restarted the system.

Interesting that after the restart, I had zero logs in any of my interfaces where an interface IP was deleted, so nothing caused any interface flapping at all.