Suricata blocking IPs on passlist, legacy mode blocking both
-
@bmeeks I did some additional testing and got a block logged with pass list debugging on. This was on the WAN interface. I haven't gotten one logged on an internal interface, but this shows where the WAN IP was blocked.
Default IP Pass List:
10.10.5.0/24 10.10.5.101/32 10.10.6.0/24 10.10.7.0/24 10.10.8.0/24 10.10.9.0/24 10.10.10.0/24 10.10.11.0/24 10.10.15.0/24 10.10.25.0/24 10.10.31.0/29 10.10.32.0/29 10.10.33.0/29 10.10.34.0/29 10.10.35.0/29 10.10.36.0/29 10.10.37.0/29 10.10.45.0/24 10.10.55.0/24 10.10.60.0/29 <WAN Gateway>/32 fe80:6::/64 fe80:7::/64 fe80:8::/64 fe80:9::/64 fe80:10::/64passlist_debug.log:
12/24/2023-19:07:07.155889 Pass List debugging enabled. Processing file: /usr/local/etc/suricata/suricata_57861_ix0/passlist. 12/24/2023-19:07:07.156041 Added IPv4 netblock 10.10.5.0/24 to IPv4 Radix Tree created from Pass List entry 10.10.5.0/24. 12/24/2023-19:07:07.156064 Added IPv4 address 10.10.5.101/32 from Pass List. 12/24/2023-19:07:07.156095 Added IPv4 netblock 10.10.6.0/24 to IPv4 Radix Tree created from Pass List entry 10.10.6.0/24. 12/24/2023-19:07:07.156106 Added IPv4 netblock 10.10.7.0/24 to IPv4 Radix Tree created from Pass List entry 10.10.7.0/24. 12/24/2023-19:07:07.156114 Added IPv4 netblock 10.10.8.0/24 to IPv4 Radix Tree created from Pass List entry 10.10.8.0/24. 12/24/2023-19:07:07.156121 Added IPv4 netblock 10.10.9.0/24 to IPv4 Radix Tree created from Pass List entry 10.10.9.0/24. 12/24/2023-19:07:07.156129 Added IPv4 netblock 10.10.10.0/24 to IPv4 Radix Tree created from Pass List entry 10.10.10.0/24. 12/24/2023-19:07:07.156136 Added IPv4 netblock 10.10.11.0/24 to IPv4 Radix Tree created from Pass List entry 10.10.11.0/24. 12/24/2023-19:07:07.156144 Added IPv4 netblock 10.10.15.0/24 to IPv4 Radix Tree created from Pass List entry 10.10.15.0/24. 12/24/2023-19:07:07.156151 Added IPv4 netblock 10.10.25.0/24 to IPv4 Radix Tree created from Pass List entry 10.10.25.0/24. 12/24/2023-19:07:07.156158 Added IPv4 netblock 10.10.31.0/29 to IPv4 Radix Tree created from Pass List entry 10.10.31.0/29. 12/24/2023-19:07:07.156166 Added IPv4 netblock 10.10.32.0/29 to IPv4 Radix Tree created from Pass List entry 10.10.32.0/29. 12/24/2023-19:07:07.156173 Added IPv4 netblock 10.10.33.0/29 to IPv4 Radix Tree created from Pass List entry 10.10.33.0/29. 12/24/2023-19:07:07.156180 Added IPv4 netblock 10.10.34.0/29 to IPv4 Radix Tree created from Pass List entry 10.10.34.0/29. 12/24/2023-19:07:07.156187 Added IPv4 netblock 10.10.35.0/29 to IPv4 Radix Tree created from Pass List entry 10.10.35.0/29. 12/24/2023-19:07:07.156215 Added IPv4 netblock 10.10.36.0/29 to IPv4 Radix Tree created from Pass List entry 10.10.36.0/29. 12/24/2023-19:07:07.156224 Added IPv4 netblock 10.10.37.0/29 to IPv4 Radix Tree created from Pass List entry 10.10.37.0/29. 12/24/2023-19:07:07.156232 Added IPv4 netblock 10.10.45.0/24 to IPv4 Radix Tree created from Pass List entry 10.10.45.0/24. 12/24/2023-19:07:07.156239 Added IPv4 netblock 10.10.55.0/24 to IPv4 Radix Tree created from Pass List entry 10.10.55.0/24. 12/24/2023-19:07:07.156246 Added IPv4 netblock 10.10.60.0/29 to IPv4 Radix Tree created from Pass List entry 10.10.60.0/29. 12/24/2023-19:07:07.156253 Added IPv4 address <WAN Gateway>/32 from Pass List. 12/24/2023-19:07:07.156277 Added IPv6 netblock fe80:0006:0000:0000:0000:0000:0000:0000/64 to IPv6 Radix Tree created from Pass List entry fe80:6::/64. 12/24/2023-19:07:07.156291 Added IPv6 netblock fe80:0007:0000:0000:0000:0000:0000:0000/64 to IPv6 Radix Tree created from Pass List entry fe80:7::/64. 12/24/2023-19:07:07.156300 Added IPv6 netblock fe80:0008:0000:0000:0000:0000:0000:0000/64 to IPv6 Radix Tree created from Pass List entry fe80:8::/64. 12/24/2023-19:07:07.156310 Added IPv6 netblock fe80:0009:0000:0000:0000:0000:0000:0000/64 to IPv6 Radix Tree created from Pass List entry fe80:9::/64. 12/24/2023-19:07:07.156322 Added IPv6 netblock fe80:0010:0000:0000:0000:0000:0000:0000/64 to IPv6 Radix Tree created from Pass List entry fe80:10::/64. 12/24/2023-19:07:07.156340 Completed processing Pass List /usr/local/etc/suricata/suricata_57861_ix0/passlist. Total entries parsed: 26, Unique IP addresses/netblocks/aliases added to Radix Trees: 26, IP addresses/netblocks ignored because they were covered by existing Radix Tree entries: 0. 12/24/2023-19:11:38.942701 Thread: W#01 SRC IP: 194.26.135.109 did not match any Pass List entry, so adding to block list. 12/24/2023-19:11:38.964387 Thread: W#01 Successfully added IP: 194.26.135.109 to pf table snort2c for blocking. 12/24/2023-19:11:39.052857 Thread: W#01 Successfully killed any open states for IP: 194.26.135.109, so any stateful traffic is blocked. 12/24/2023-19:11:38.942701 Thread: W#01 DST IP: <WAN IP> covered by Pass List entry <WAN IP>/32 - not blocking. 12/24/2023-19:11:53.817090 Thread: W#03 SRC IP: 77.90.185.73 did not match any Pass List entry, so adding to block list. 12/24/2023-19:11:53.877467 Thread: W#03 Successfully added IP: 77.90.185.73 to pf table snort2c for blocking. 12/24/2023-19:11:53.964781 Thread: W#03 Successfully killed any open states for IP: 77.90.185.73, so any stateful traffic is blocked. 12/24/2023-19:11:53.817090 Thread: W#03 DST IP: <WAN IP> did not match any Pass List entry, so adding to block list. 12/24/2023-19:11:53.964897 Thread: W#03 Successfully added IP: <WAN IP> to pf table snort2c for blocking. 12/24/2023-19:11:54.052646 Thread: W#03 Successfully killed any open states for IP: <WAN IP>, so any stateful traffic is blocked. 12/24/2023-19:12:13.266186 Thread: W#04 SRC IP: 35.203.211.174 did not match any Pass List entry, so adding to block list. 12/24/2023-19:12:13.464580 Thread: W#04 Successfully added IP: 35.203.211.174 to pf table snort2c for blocking. 12/24/2023-19:12:13.551139 Thread: W#04 Successfully killed any open states for IP: 35.203.211.174, so any stateful traffic is blocked. 12/24/2023-19:12:13.266186 Thread: W#04 DST IP: <WAN IP> did not match any Pass List entry, so adding to block list. 12/24/2023-19:12:13.637498 Thread: W#04 Successfully killed any open states for IP: <WAN IP>, so any stateful traffic is blocked. 12/24/2023-19:12:18.425677 Thread: W#01 SRC IP: 35.203.211.7 did not match any Pass List entry, so adding to block list. 12/24/2023-19:12:18.514789 Thread: W#01 Successfully added IP: 35.203.211.7 to pf table snort2c for blocking. 12/24/2023-19:12:18.602906 Thread: W#01 Successfully killed any open states for IP: 35.203.211.7, so any stateful traffic is blocked. 12/24/2023-19:12:18.425677 Thread: W#01 DST IP: <WAN IP> did not match any Pass List entry, so adding to block list. 12/24/2023-19:12:18.689395 Thread: W#01 Successfully killed any open states for IP: <WAN IP>, so any stateful traffic is blocked. 12/24/2023-19:12:35.402347 Thread: W#04 SRC IP: 65.49.20.108 did not match any Pass List entry, so adding to block list. 12/24/2023-19:12:35.709159 Thread: W#04 Successfully added IP: 65.49.20.108 to pf table snort2c for blocking. 12/24/2023-19:12:35.794935 Thread: W#04 Successfully killed any open states for IP: 65.49.20.108, so any stateful traffic is blocked. 12/24/2023-19:12:35.402347 Thread: W#04 DST IP: <WAN IP> did not match any Pass List entry, so adding to block list. 12/24/2023-19:12:35.880853 Thread: W#04 Successfully killed any open states for IP: <WAN IP>, so any stateful traffic is blocked. 12/24/2023-19:12:40.452282 Thread: W#02 SRC IP: 167.94.145.80 did not match any Pass List entry, so adding to block list. 12/24/2023-19:12:40.774064 Thread: W#02 Successfully added IP: 167.94.145.80 to pf table snort2c for blocking. 12/24/2023-19:12:40.860822 Thread: W#02 Successfully killed any open states for IP: 167.94.145.80, so any stateful traffic is blocked. 12/24/2023-19:12:40.452282 Thread: W#02 DST IP: <WAN IP> did not match any Pass List entry, so adding to block list. 12/24/2023-19:12:40.947516 Thread: W#02 Successfully killed any open states for IP: <WAN IP>, so any stateful traffic is blocked. 12/24/2023-19:12:40.918625 Thread: W#03 SRC IP: 62.204.41.63 did not match any Pass List entry, so adding to block list. 12/24/2023-19:12:41.273533 Thread: W#03 Successfully added IP: 62.204.41.63 to pf table snort2c for blocking. 12/24/2023-19:12:41.360033 Thread: W#03 Successfully killed any open states for IP: 62.204.41.63, so any stateful traffic is blocked. 12/24/2023-19:12:40.918625 Thread: W#03 DST IP: <WAN IP> did not match any Pass List entry, so adding to block list. 12/24/2023-19:12:41.446572 Thread: W#03 Successfully killed any open states for IP: <WAN IP>, so any stateful traffic is blocked. 12/24/2023-19:12:51.957100 Thread: W#03 SRC IP: 77.90.185.166 did not match any Pass List entry, so adding to block list. 12/24/2023-19:12:52.415512 Thread: W#03 Successfully added IP: 77.90.185.166 to pf table snort2c for blocking. 12/24/2023-19:12:52.502160 Thread: W#03 Successfully killed any open states for IP: 77.90.185.166, so any stateful traffic is blocked. 12/24/2023-19:12:51.957100 Thread: W#03 DST IP: <WAN IP> did not match any Pass List entry, so adding to block list. 12/24/2023-19:12:52.588762 Thread: W#03 Successfully killed any open states for IP: <WAN IP>, so any stateful traffic is blocked. 12/24/2023-19:13:17.273336 Thread: W#04 SRC IP: 80.66.83.171 did not match any Pass List entry, so adding to block list. 12/24/2023-19:13:17.672138 Thread: W#04 Successfully added IP: 80.66.83.171 to pf table snort2c for blocking. 12/24/2023-19:13:17.758300 Thread: W#04 Successfully killed any open states for IP: 80.66.83.171, so any stateful traffic is blocked. 12/24/2023-19:13:17.273336 Thread: W#04 DST IP: <WAN IP> did not match any Pass List entry, so adding to block list. 12/24/2023-19:13:17.844843 Thread: W#04 Successfully killed any open states for IP: <WAN IP>, so any stateful traffic is blocked. 12/24/2023-19:13:33.413564 Thread: W#02 SRC IP: 77.90.185.127 did not match any Pass List entry, so adding to block list. 12/24/2023-19:13:33.854843 Thread: W#02 Successfully added IP: 77.90.185.127 to pf table snort2c for blocking. 12/24/2023-19:13:33.940725 Thread: W#02 Successfully killed any open states for IP: 77.90.185.127, so any stateful traffic is blocked. 12/24/2023-19:13:33.413564 Thread: W#02 DST IP: <WAN IP> did not match any Pass List entry, so adding to block list. 12/24/2023-19:13:34.026931 Thread: W#02 Successfully killed any open states for IP: <WAN IP>, so any stateful traffic is blocked. 12/24/2023-19:13:46.250811 Thread: W#04 SRC IP: 198.235.24.144 did not match any Pass List entry, so adding to block list. 12/24/2023-19:13:46.513589 Thread: W#04 Successfully added IP: 198.235.24.144 to pf table snort2c for blocking. 12/24/2023-19:13:46.600316 Thread: W#04 Successfully killed any open states for IP: 198.235.24.144, so any stateful traffic is blocked. 12/24/2023-19:13:46.250811 Thread: W#04 DST IP: <WAN IP> did not match any Pass List entry, so adding to block list. 12/24/2023-19:13:46.687333 Thread: W#04 Successfully killed any open states for IP: <WAN IP>, so any stateful traffic is blocked. 12/24/2023-19:14:06.084828 Thread: W#03 SRC IP: 31.220.1.83 did not match any Pass List entry, so adding to block list. 12/24/2023-19:14:06.197129 Thread: W#03 Successfully added IP: 31.220.1.83 to pf table snort2c for blocking. 12/24/2023-19:14:06.283613 Thread: W#03 Successfully killed any open states for IP: 31.220.1.83, so any stateful traffic is blocked. 12/24/2023-19:14:06.084828 Thread: W#03 DST IP: <WAN IP> did not match any Pass List entry, so adding to block list. 12/24/2023-19:14:06.370284 Thread: W#03 Successfully killed any open states for IP: <WAN IP>, so any stateful traffic is blocked. 12/24/2023-19:14:38.394887 Thread: W#04 SRC IP: 77.90.185.92 did not match any Pass List entry, so adding to block list. 12/24/2023-19:14:38.518387 Thread: W#04 Successfully added IP: 77.90.185.92 to pf table snort2c for blocking. 12/24/2023-19:14:38.605196 Thread: W#04 Successfully killed any open states for IP: 77.90.185.92, so any stateful traffic is blocked. 12/24/2023-19:14:38.394887 Thread: W#04 DST IP: <WAN IP> did not match any Pass List entry, so adding to block list. 12/24/2023-19:14:38.692102 Thread: W#04 Successfully killed any open states for IP: <WAN IP>, so any stateful traffic is blocked. 12/24/2023-19:15:00.905606 Thread: W#04 SRC IP: 77.90.185.14 did not match any Pass List entry, so adding to block list. 12/24/2023-19:15:01.321169 Thread: W#04 Successfully added IP: 77.90.185.14 to pf table snort2c for blocking. 12/24/2023-19:15:01.407791 Thread: W#04 Successfully killed any open states for IP: 77.90.185.14, so any stateful traffic is blocked. 12/24/2023-19:15:00.905606 Thread: W#04 DST IP: <WAN IP> did not match any Pass List entry, so adding to block list. 12/24/2023-19:15:01.494809 Thread: W#04 Successfully killed any open states for IP: <WAN IP>, so any stateful traffic is blocked. 12/24/2023-19:15:37.899907 Thread: W#02 SRC IP: 95.214.55.244 did not match any Pass List entry, so adding to block list. 12/24/2023-19:15:38.293385 Thread: W#02 Successfully added IP: 95.214.55.244 to pf table snort2c for blocking. 12/24/2023-19:15:38.379657 Thread: W#02 Successfully killed any open states for IP: 95.214.55.244, so any stateful traffic is blocked. 12/24/2023-19:15:37.899907 Thread: W#02 DST IP: <WAN IP> did not match any Pass List entry, so adding to block list. 12/24/2023-19:15:38.465667 Thread: W#02 Successfully killed any open states for IP: <WAN IP>, so any stateful traffic is blocked.Suricata.log:
-
@sgnoc:
To make some sense of this you will need to correlate times for your WAN IP from two different logs.First, find all the times in
suricata.logwhere the WAN IP address is deleted by the interface monitoring thread. You should be able to find those quickly by searching for your WAN IP as a string in that log.Second, find the times from the Pass List debugging log where the WAN IP was blocked.
Here is what I think is happening. Your WAN IP (and all of the firewall interface IP addresses) are not part of the default Pass List you see in the GUI. Instead, a separate monitoring thread within the custom blocking module runs in a continuous loop subscribed to the kernel routing messages. The kernel sends that thread a notice each time an IP address is removed from or added to a firewall interface. The thread in turn either adds that firewall interface IP to or removes it from the Radix Tree structure that stores the Pass List IP addresses and netblocks.
The blocking portion of the custom module receives alerting packets, pulls out the SRC and DST IP addresses, searches for those in the Radix Tree structure, and if not found will block the IP. If the IP is either found directly in the Radix Tree, or it is determined to be contained within a netblock defined in the Radix Tree, then it is not blocked. The Radix Tree is used for very fast lookups of IP addresses.
This is a multithreaded operation as Suricata has multiple threads reading packets and processing alerts. These multiple threads are in turn looking up IP addresses in the two Radix Trees continually (one Radix Tree for IPv4 addresses and another for IPv6 addresses). If the firewall interface monitoring thread that is looking at the kernel routing messages sees that your WAN IP has changed (or has been removed for some reason), then that thread will remove the WAN IP (the old one) from the Radix Tree. If later the WAN IP returns either with the same value or a new value, the Radix Tree is appropriately updated. But if, during the interval between the WAN IP being removed and then added back at some later point, some other thread processed an alert containing the WAN IP (that was just deleted), then the IP is going to get blocked because it's not currently in the Radix Tree. If you read the Pass List debugging log carefully, you can see which threads blocked the WAN IP and which did not. I'm thinking this is all related to the rapid number of changes going on with your interface IPs (and all those OpenVPN addresses, too). Your WAN IP is being periodically cycled for some reason, and that results in it being removed from the Radix Tree (same as Pass List) for a bit. Then later it gets added back, but that short time it's missing is long enough for one of the running Suricata packet processing threads to process an alert with the IP address in it and thus trigger a block.
So, the first order of business in your case is to figure out why your interface IP addresses are changing so often. As I mentioned seeing in the log snippet you sent me, there are dozens and dozens of lines of interface IP address changes happening. That is not normal, and will definitely confuse the Suricata custom blocking module threads. They expect only very infrequent interface IP changes such as might happen if a PPPoE WAN interface cycled or your ISP renewed your WAN DHCP address with a new and different one. The threading logic is not expecting nearly as many changes per minute as I am seeing in your logs. I don't know if this is an issue with your CARP configuration, something with all the OpenVPN instances I see, something with the WireGuard Tunnel, or something related to LAGG interfaces that has cropped up in the recent pfSense release. But all those interface IP address ups and downs are the cause of your Pass List issue I suspect.
You say this worked in the past, but did you check back then to see if you had nearly as many interface IP changes as you do now? They would always have been logged in the
suricata.logas that logic has been around for quite a while. -
@bmeeks So that was confusing me too, but to add even more confusion, my WAN IP very rarely changes. I keep the same WAN IP for months in most cases before it updates to a new IP that I keep for some more months at a time. On top of that, ALL of my internal interfaces never change IP addresses for the interfaces, ever. They have been the same for years. I've never had any issues with Suricata previously (since I started running on this platform in 2019), so I've never had a need to look at the suricata logs to see if there were these IPs being added/deleted so often.
I'm not sure why it is happening or how, but I can assure you the IP does not change at all on the internal interfaces, and only changes a few times a year on the WAN.
I don't see any correlation that I can recognize between the suricata logs and the added/deleted logs on the debug file.
Side note, Merry Christmas! I didn't figure I would get any response until after.
-
One thought just came to mind. I believe this specific alert/block happened while I was loading suricata on the other interfaces, which were not completely up yet.
It seems that each time I load/reload a single suricata interface, it causes all of the other suricata instances to reload some portions. Could that be causing Suricata to think the IPs are being updated when they aren't?
Each time I reload a single interface, I see these logs in the main system log, and there is a set of these for ALL interfaces by a single interface restart/start. I never understood why restarting one interface did this for all interfaces.
2023-12-24 19:31:04.068717-05:00 php 86831 [Suricata] Building new sid-msg.map file for 00_WAN... 2023-12-24 19:31:03.828809-05:00 php 86831 [Suricata] Enabling any flowbit-required rules for: 00_WAN... 2023-12-24 19:30:57.681301-05:00 php 86831 [Suricata] Updating rules configuration for: 00_WAN ...I just checked my other interface, the lag0.33. Where it was blocking the internal 10.10.33.2 address before I turned on the pass list debugging, now that pass list debugging is on, the 10.10.33.2 address is no longer being blocked when an alert hits. I've had two alerts hit since I restarted the interface with the pass list debug enabled and it only blocked the external address, not the internal address. So something different is happening for me with the pass list debugging that is not triggering the internal IP address to be blocked, where it does before the debug was enabled. I'm baffled.
-
@sgnoc:
When you make changes to interfaces inside pfSense, then pfSense itself will send all packages a "restart all packages" command. This is done in the event IP addresses or subnets change on configured interfaces and packages might need to know they are changing. Even when nothing "really" changes, the message is still sent. Suricata will see this command from pfSense and dutifully restart itself. It uses the shell script in/usr/local/etc/rc.d/suricata.shto do this. That shell script restarts all configured Suricata interfaces.Also, a few versions back Suricata upstream made some changes in how the PCAP function works. The changes were to address a bug that cropped up occasionally when the underlying interface PCAP was running on restarted. I believe a side-effecct of this bug fix is that now when Suricata starts PCAP on some interface types it will cause a brief "reset" of the interface. That might cause it to appear to have gone down and come righ back up.
When I said your WAN IP was changing, I don't literally mean a different one. I mean the kernel is cycling the interface for some reason and sending the interface monitoring thread in Suricata's custom blocking module a notification of removing and adding an IP address to an interface. The kernel sends two messages: one deleting the IP address from the interface; and then another adding it back. 9 times out of 10 it adds back the same IP. But the monitoring thread can't assume that. So, when it gets the "DELETE ADDRESS" message it removes the IP from the internal Radix Tree. Later, when it gets the "ADD ADDRESS" message, it will put it back in the tree. But in the interval between those two messages on a very busy network it's entirely possible for one of the other Suricata packet processing threads to have received a packet with the WAN IP and go ahead and block it since for a brief second it is removed from the Radix Tree and would not be found on the "pass list" when looked up.
These are the messages I'm talking about where the automatic Interface Monitoring Thread (thread ID IM#01) is removing your WAN IP from the internal pass list then later adding it back. It appears to do this multiple times. During the interval when the WAN IP has been automatically removed, then it most certainly can be blocked (and it is, according to the Pass List debugging log).
[117623 - Suricata-IM#01] 2023-12-24 19:11:23 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface ix0. [117623 - Suricata-IM#01] 2023-12-24 19:11:23 Info: alert-pf: Deleted address <WAN IP> from automatic firewall interface IP Pass List. [117623 - Suricata-IM#01] 2023-12-24 19:11:24 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface ix0. [117623 - Suricata-IM#01] 2023-12-24 19:11:24 Info: alert-pf: Added address <WAN IP> to automatic firewall interface IP Pass List. [117623 - Suricata-IM#01] 2023-12-24 19:11:25 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface ix0. [117623 - Suricata-IM#01] 2023-12-24 19:11:25 Info: alert-pf: Deleted address <WAN IP> from automatic firewall interface IP Pass List. [117623 - Suricata-IM#01] 2023-12-24 19:11:26 Info: alert-pf: Monitor Thread IM#01 received notification of IP address change on interface ix0. [117623 - Suricata-IM#01] 2023-12-24 19:11:26 Info: alert-pf: Added address <WAN IP> to automatic firewall interface IP Pass List.It's getting removed because the pfSense kernel is sending the monitoring thread a notification telling it that. The first line of log text is from the kernel routing message subscription. It says an IP change happened on the
ix0interface. As I said earlier, here "change" does not necessarily mean a new IP. It might simply mean the interface was flapping. Shortly thereafter the kernel sent another message aboutix0and resulted in putting the IP address back. At that point the IM#01 monitoring thread would have put the WAN IP back in the Radix Tree (same as Pass List). But then shortly thereafter, the process repeats and the address is removed again. You need to find out what's going on that is causing all the interface flapping. That's why things are getting blocked, I think.Because Suricata is multithreaded, this can be a bit hard to visualize in your head. But consider that in your case Suricata has 4 threads processing packets, comparing them to the rule signatures, and generating alerts when something matches. Those threads are labeled in the Pass List Debug Log as W#01 through W#04. You can follow them in the Pass List Debug Log. A totally separate thread is running at the same time listening for Kernel Routing messages from pfSense. That thread is automatically adding or removing firewall interface IPs from the pass list depending on whether the kernel says "I'm deleting an IP" or "I'm adding an IP". So that thread (called #IM01, for "interface monitoring thread #01) is constantly deleting and then adding back firewall interface IP addresses in your case because something appears to be flapping (interfaces going up and down). So, now it's a random race with the four packet processing worker threads. Does one of them see a packet with your WAN IP in it during the brief interval where the interface monitoring thread has deleted the WAN IP from the internal pass list? If so, then the WAN IP gets blocked.
Looking in your Pass List Debug log I see instances where a worker thread found the WAN IP in the pass list and did NOT block it, and then later a different thread checked and the WAN IP was NOT in the pass list at that instant and thus got blocked.
-
@bmeeks That makes sense with the multiple threads and one catching an alert between deleting and readding the IP to the list. I'm not really sure how to go about finding what would cause the interface to flap. I haven't seen any other logs on the system that would indicate that to be the case, so this is the first I have found that to be happening. The original issue I had with Suricata blocking IPs was not occurring on my WAN, but only the internal interfaces.
I only had the issue happen once on the WAN, when I reinstalled Suricata. You mentioned at the time to remove and then install rather than reinstall Suricata. I decided to start with a fresh install on Suricata this go around and the WAN IP being blocked popped back up. I can't restart again right now, but I'm planning to try and see if I can restart again with it completely set back up to see if the issue on the WAN goes away again or if it is still persistent.
I'll start going through different logs and see if anything jumps out to show what could be causing the flapping.
-
@sgnoc We have the same interface flapping in the suricata logs sent to @bmeeks since going to suricata 7.
But CARP/HA does not set off any warnings on the same interfaces when not running suricata and no issues are observed otherwise. Switch does not log any interface flapping either.
Suricata blocks WAN VIP within seconds of being started on WAN but it also blocks internal IPs randomly after running for hours without issue on an internal interface.We are using CARP/HA on these interfaces and were not having any issues before suricata 7.
@bmeeks Switching to suricata inline should overcome these issues shouldn't it ? -
@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:
The original issue I had with Suricata blocking IPs was not occurring on my WAN, but only the internal interfaces.
Don't get hung up focusing on just the WAN. I used it in my explanations simply as an example of what's happening in the big picture. If you look in the
suricata.logfor your instances you will see that apparently all of the firewall interfaces are exhibiting that behavior. I see OpenVPN IPs, a WG Tunnel, etc., in the log messages. Anytime you see a message logged from the "IM#01" interface monitoring thread, that's one from the thread receiving the kernel routing messages. That thread will receive notifications for any IP address assigned on the firewall. That includes interfaces such as WAN and LAN, but also things like Virtual IPs, OpenVPN interfaces, etc.I see those kinds of messages for a ton of firewall interfaces, and they show the IPs being added, then deleted, then added back again. I don't know why that is happening, but I can certainly see how that behavior can lead to both the WAN and internal interface IPs getting blocked because for brief instances of time those addresses are indeed not present in the in-memory Radix Tree that comprises what we call the "pass list".
Suricata uses a large common packet queue for processing. Packets are obtained off the wire and placed into this queue by packet acquisition threads. Then running worker threads service that queue by pulling out a queued up packet and processing it against the signatures. The worker thread then either generates an alert- or not- for that packet, then it goes back and grabs another packet from the common queue to process. You can see how in this process a packet with one of your interface IPs could get placed into the queue, then shortly thereafter the kernel for some reason deletes the address from that interface (and then later adds it back). If one of the worker threads happens to pull out that interface's packet to process during the brief interval where the kernel had "deleted the IP", then the alert would cause a block because at that instant the interface's IP is not in the internal pass list. Later it gets put back and some other worker thread analyzing a packet would find the interface IP in the internal pass list and not block it. But then at some point, the process of deletion recurs and you are back to being blocked. This is what I see happening when I correlate the data from both the
suricata.logand the Pass List Debug log.Here is an example:
12/24/2023-19:11:38.942701 Thread: W#01 DST IP: <WAN IP> covered by Pass List entry <WAN IP>/32 - not blocking.....
12/24/2023-19:11:53.817090 Thread: W#03 DST IP: <WAN IP> did not match any Pass List entry, so adding to block list.Looking at the above log snippets, you will see that at 19:11:38 Worker thread W#01 tested the WAN IP address and found it covered by a Pass List entry and thus did not block it. But then at 19:11:53 Worker thread W#03 processed a packet containing your WAN IP as the destination IP address, and when that thread tested your WAN IP against the Pass List it found it not there so it blocked it. If you look through the
suricata.logfile, I suspect you will find an entry in that time window between 19:11:38 and 19:11:53 where the interface monitoring thread (#IM01) received a kernel notification to delete the WAN IP address from the pass list because it was being removed from the interface. I'm not saying the interface flapping stuff is the only problem, but it certainly is a major contributor to your firewall interface IPs getting blocked.What does your pfSense system log show during this time interval (say between Suricata starting and your WAN getting blocked)? I am curious about what's getting logged about the LAGG interface and all those OpenVPN interfaces. It might simply be a CARP sensitivity thing that could be mitigated by relaxing some timing. I'm not a CARP expert, but I do recall reading something about a configuration parameter that essentially increased the length of "down time" allowed from the Primary before the Secondary decided the Primary must be dead and tries to assume the Master role. The Primary needs some time to get all those interfaces up and running and Suricata stable on them. Maybe the Secondary is not waiting long enough at startup before trying to assume the Master role?
To be honest, you seem to have a crap ton of things defined on that firewall between the GUI pass list and the kernel routing notifications. I think that's the most IP subnets I've ever seen in configurations shared by users in the past.
You are also using two configurations I have never tested with Suricata: LAGG interfaces and CARP. There comes a point where your needs may outweigh the capabilities of free software and it's time to move up to the big boys club and pay the piper for enterprise-level security
. -
@btspce said in Suricata blocking IPs on passlist, legacy mode blocking both:
Switching to suricata inline should overcome these issues shouldn't it ?
Inline IPS Mode does not use a pass list as one is not necessary. So, yes it would prevent permanent blocking of an interface IP. Individual packets might still get dropped for some particular rule.
Note that with Inline IPS Mode nothing will be blocked unless you explicitly change the rule action for the signatures that you want to block traffic to DROP. The default action for rule signatures is ALERT, and that will log an alert but will not block (or drop) traffic using Inline IPS Mode.
You can change rule actions either one-by-one on the RULES tab, or as groups using the features on the SID MGMT tab. There is Sticky Post or two at the top of this sub-forum explaining how to do that.
-
@bmeeks We are using traffic shaping (limiters) to combat bufferbloat and limit other traffic. Are there still issues running suricata in inline mode with limiters ?
-
@btspce said in Suricata blocking IPs on passlist, legacy mode blocking both:
@bmeeks We are using traffic shaping (limiters) to combat bufferbloat and limit other traffic. Are there still issues running suricata in inline mode with limiters ?
Yes, limiters and the netmap device used by Inline IPS Mode in Suricata and Snort do not like each other and will not play well together.
Do you have an asymmetrical ISP connection? Typically buffer bloat is not nearly as big an issue on symmetrical connections (same speed for both upload and download).
-
@bmeeks Better up.. dual asymmetrical ISP connections :)
These issues did not seem like an easy fix so I will try to migrate the suricata wan instance to an internal interface instead (were no limiters are placed at the moment) and switch to inline to overcome this. -
@btspce said in Suricata blocking IPs on passlist, legacy mode blocking both:
@bmeeks Better up.. dual asymmetrical ISP connections :)
These issues did not seem like an easy fix so I will try to migrate the suricata wan instance to an internal interface instead (were no limiters are placed at the moment) and switch to inline to overcome this.Like those of @sgnoc, your
suricata.logfile you sent me is full of what appears to be some type of interface flapping. I see messages repeatedly from the interface monitoring thread (IM#01) saying it received a kernel routing message to delete an IP address from the WAN (and VPN interfaces), and then a second or so later another kernel routing message saying it is adding it back.That sort of activity is going to result in the same thing I described in a post up above whereby interface IPs on the firewall could get blocked during the interval between when the kernel deleted them from the interface and then later added them back. The interface monitoring thread (IM#01) is listening for those kernel routing messages and dutifully removes (or adds back) that interface IP from the internal Radix Tree (which is what the pass list logic uses to see if an IP address is covered by a pass list entry before blocking it).
It also seems this issue is rather limited in scope. There are users reporting in the thread that are NOT having the problem. I think at least one of those not having a problem is also using CARP (but I've read so many posts from so many different threads the past two weeks that I may not be remembering the correct info for the correct thread).
Two things changed in the recent update. First was the underlying pfSense kernel, and then of course Suricata went from version 6.x from upstream to version 7.x from upstream. Have you investigated what I suggested to @sgnoc about seeing if the CARP failover interval can be lengthened? Interfaces may be flapping because during startup Suricata and CARP are interfering with each other. Just a guess as CARP is not something I've ever tested with Suricata- and I don't have the equipment here at home to do that anyway.
-
@bmeeks FWIW on our HA setup, which has Suricata running on LAN not WAN, the only "IM#01" messages logged are the "Info: alert-pf: Firewall Interface IP Address Change monitoring thread IM#01 has successfully started" notice, one each on Dec. 22.
@sgnoc, @btspce There's not a log entry in System Logs about the interface actually flapping?
-
@SteveITS said in Suricata blocking IPs on passlist, legacy mode blocking both:
FWIW on our HA setup, which has Suricata running on LAN not WAN, the only "IM#01" messages logged are the "Info: alert-pf: Firewall Interface IP Address Change monitoring thread IM#01 has successfully started" notice, one each on Dec. 22.
This is what I would expect as normal. Perhaps also a late entry about an OpenVPN interface coming online. But the amount of deletions and additions I'm seeing in the logs provided me by @btspce and @sgnoc is extreme.
During Suricata startup, a dedicated function within the custom Legacy Blocking Module scans all the firewall interfaces, grabs the assigned IP addresses for each one, then puts those in the internal Radix Tree structure that holds the pass list info.
Then, a monitoring thread is started. That thread is responsible for keeping tabs on firewall interface IPs and adjusting membership in the pass list if a firewall interface IP changes later (after the initial startup scan). The monitoring thread does this by subscribing to kernel routing messages via a socket and then looping waiting on the kernel to send it a message. This monitoring thread logic was added several years ago in response to complaints from users whose PPPoE WAN interfaces would get blocked. This happened if the interface was cycled by the ISP and the user got a new WAN IP. Suricata would be still using the old original IP it scanned at package startup and thus might block the new WAN IP. The monitoring thread logic fixed that issue.
-
@bmeeks @SteveITS I don't see any logs or records of interfaces flapping anywhere outside of Suricata, which leads me to think it is something inside Suricata going on. No other services, or logs indicate any interface flapping. Each time I restart my Suricata instance, or install Suricata, it starts each of the interfaces, which is currently 7. It seems that every time one interface on Suricata is started or changed, it causes every interface to reload, which seems to be causing the flapping. At least that's the best that I can tell. So 7 interfaces causing all 7 to start would generate a whole lot of noise.
I did a complete fresh install of pfSense on my router. From what I'm best able to tell, every fresh install of Suricata now causes the WAN interface to block the WAN IP for all alerts. This continues, but is somehow fixed when I restart the pfSense router, then it begins to operate normally. There is still no WAN IP listed in the default pass list IPs, but must be on a hidden list as explained by @bmeeks in a previous post.
The internal interfaces continue to block internal IPs. I am having to manually unblock my web server, which blocks every time an alert is generated. However, when I start pass list debugging, the interface starts to block properly. It's done this 3 times now. That is my more noisy interface. My other interfaces are experiencing this same issue, but just far less frequent.
The only logs I'm finding generated are of Suricata.log which shows nothing around the time of the alert causing the block, and the block.log showing the block, as well as the alert.log showing the alert.
I'm now trying to do a manually added pass list with all of the default generated rules. My interfaces are all on a 10.10.x.0/29 or 10.10.x.0/24, so I added 10.10.0.0/16 manually to the custom pass list table. So far that has seemed to stop blocking the internal IPs, which leads me to believe there is something going on with the default pass list that is generated not functioning properly.
*** Correction, I haven't had an alert generate yet to see if adding the custom pass list subnet works. The alerts I had are all older than me making that change, so I don't know if it has made a difference or the same result.
-
@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:
There is still no WAN IP listed in the default pass list IPs
I've stated this several times -- you will NEVER see any firewall interface IP (including the WAN IP) listed in the Pass List you see in the GUI (default or otherwise unless you manually add it as a specific extra entry). That's not how the logic works since several years ago. The custom blocking module queries pfSense itself at startup to get a list of all the firewall interface IPs (including the WAN IP). It then adds those to the Radix Tree. A Radix Tree is a fast-search software structure that is used to store IP addresses and netblocks for quick lookup and testing. Here is some additional information about them: https://medium.com/@rakesht2499/ip-model-radix-tree-implementation-cc4485e68a9e. All IPs from the GUI pass list along with those queried directly from pfSense go into the Radix Tree. The Radix Tree becomes the pass list internally.
Your problem is the frequent deletions and additions of firewall interface IPs. Suricata does not do that by itself in a vacuum. It subscribes to pfSense kernel routing messages for that info as described here: https://man.freebsd.org/cgi/man.cgi?query=route&apropos=0&sektion=4&manpath=FreeBSD+14.0-RELEASE&arch=default&format=html. It then acts upon the routing messages sent to it by the kernel. I don't know why your configuration and that of @btspce is doing the repeated deletions and additions. I have never seen that before.
-
@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:
This continues, but is somehow fixed when I restart the pfSense router, then it begins to operate normally.
When you do this, are you still seeing the same of number of "deleted IP address xxxx" and "added IP address xxxx" in the
suricata.logfile for the interface?I see basically about 1 per second in the previous logs you supplied. There should be zero normally - or 1 or 2 at most if a VPN or something comes online.
You comments lead me to believe you are missing the point here. Quit worrying about the pass list and fiddling around with changes there. Your problem is the frequent interface IP changes showing in the
suricata.log. You need to find out why that is happening in your configuration. @SteveITS says he does not see that, and he has SG-6100s in HA mode if I understand him correctly. You and @btspce are reporting the same issue, but I see the same interface flapping stuff in both of your logs. Again, not seeing that behavior in reports from others. -
@bmeeks I'm not missing the point, quite the contrary I have been trying to figure out how to find the root cause, since this has only started being an issue with the most recent versions of Suricata. Since I have owned my XG-7100 from 2019 I have not had any issues until now. This is the rest of the sentence you quoted, but left out:
but must be on a hidden list as explained by @bmeeks in a previous post.
The flapping interface ONLY happens when Suricata is initializing. At no other time do I see these deletions in the Suricata.log. I had 4 of these instances with the WAN IP when I restarted from scratch on my system. I don't see how your claim of the deletions being the issue, because when the WAN interface has the issues (only after a fresh Suricata install BEFORE rebooting the system after completing setup), the flapping ceases but the interface continues to block EVERY alert to include the SRC and DST including the WAN IP. If this was just the monitor threads catching the IP between deletion and addition, then it would not occur after the flapping stops once initialization is completed. Again, the flapping only occurs during the Suricata initialization. So if that were the case, the WAN IP would not longer be blocked because the deletions have halted. Also, after I have everything setup and complete a full system reboot, the WAN interface blocking the WAN IP issue goes away and it is just the other interfaces blocking internal IPs after that point. There are two issues I see here. The primary issue is internal IPs being blocked on my internal Suricata interfaces, despite the subnets being in the default pass list, and there are no logged flapping of interface IPs on the internal interface Suricata instances. The second issue is that currently with the latest Suricata package, after a fresh install of Suricata, the WAN interfaces is not behaving as expected, where ther are a larger number of the flapping WAN IP, and the WAN IP being blocked (even after the flapping stops). This is only resolved on a full system reboot AFTER the Suricata has finished setting up from being installed and pulling new updates, etc, After the reboot, the WAN interface acts normally again and there are a minimal (4 in this case) number of deleted WAN IPs logged during the initialization of the WAN interface.
Suricata.log:
Further, my internal interface consisting of my public facing web server has zero deletions of any related subnets from that interface, which are 10.10.33.0/29. It also catches the 4 WAN interface deletions/additions, but again after the initialization of Suricata, it clears up. I get it that you say Suricata doesn't control that, but it has to be related to Suricata's initialization, because that is the only time it occurs. Only during interface initialization of Suricata. The fact that there are exactly 4, and 4 monitors for that interface, doesn't seem to be a coincidence to me.
Here is the Suricata.log file for the 10.10.33.0/29 interface on my XG-7100:
Suricata_Log_lag0.33.txtThe only reason I've been "fiddling" with my Suricata settings, is because this is only a Suricata issue. No other logs on my system indicate any flapping, and the flapping ONLY occurs on initialization of the Suricata interfaces. That also cannot be a coincidence.
Here is the block log for my lag0.33 interface:
12/25/2023-01:23:53.793033 [Block Src] [**] [1:2032979:2] ET SCAN Yandex Webcrawler User-Agent (YandexBot) [**] [Classification: Not Suspicious Traffic] [Priority: 3] {TCP} 162.158.238.72:45214 12/25/2023-01:23:53.793033 [Block Dst] [**] [1:2032979:2] ET SCAN Yandex Webcrawler User-Agent (YandexBot) [**] [Classification: Not Suspicious Traffic] [Priority: 3] {TCP} 10.10.33.2:80 12/25/2023-12:24:43.090673 [Block Src] [**] [1:2853548:1] ETPRO HUNTING Suspicious Empty Referer Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 141.101.105.12:28970 12/25/2023-12:24:43.090673 [Block Dst] [**] [1:2853548:1] ETPRO HUNTING Suspicious Empty Referer Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 10.10.33.2:80 12/25/2023-20:21:18.406299 [Block Src] [**] [1:2853568:1] ETPRO HUNTING Suspicious Empty Sec-CH-UA Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 162.158.159.59:64511 12/25/2023-20:21:18.406299 [Block Dst] [**] [1:2853568:1] ETPRO HUNTING Suspicious Empty Sec-CH-UA Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 10.10.33.2:80 12/25/2023-20:21:18.542709 [Block Src] [**] [1:2853568:1] ETPRO HUNTING Suspicious Empty Sec-CH-UA Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 162.158.158.220:45036 12/25/2023-20:21:18.400555 [Block Src] [**] [1:2853568:1] ETPRO HUNTING Suspicious Empty Sec-CH-UA Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 162.158.159.202:65230 12/25/2023-20:21:18.415277 [Block Src] [**] [1:2853568:1] ETPRO HUNTING Suspicious Empty Sec-CH-UA Header [**] [Classification: Misc activity] [Priority: 3] {TCP} 162.158.159.197:31129 12/26/2023-03:45:26.912634 [Block Src] [**] [1:2049255:1] ET SCAN LeakIX Inbound User-Agent [**] [Classification: Misc activity] [Priority: 3] {TCP} 172.71.242.59:65252 12/26/2023-03:45:26.912634 [Block Dst] [**] [1:2049255:1] ET SCAN LeakIX Inbound User-Agent [**] [Classification: Misc activity] [Priority: 3] {TCP} 10.10.33.2:80Here is the Default Pass List:
10.10.5.0/24 10.10.5.101/32 10.10.6.0/24 10.10.7.0/24 10.10.8.0/24 10.10.9.0/24 10.10.10.0/24 10.10.11.0/24 10.10.15.0/24 10.10.25.0/24 10.10.31.0/29 10.10.32.0/29 10.10.33.0/29 10.10.34.0/29 10.10.35.0/29 10.10.36.0/29 10.10.37.0/29 10.10.45.0/24 10.10.55.0/24 10.10.60.0/29 <WAN Gateway>/32 fe80:6::/64 fe80:7::/64 fe80:8::/64 fe80:9::/64 fe80:10::/64With no other logs generated from pfSense that show any interfaces going down/up, I don't see where else I may be able to look outside of this being a specific Suricata phenomenon. If there is any guidance where else I may be able to look, I would certainly give it a try. This is netgate hardware that has been running for years without these issues until Suricata 7, and it is only affecting Suricata. No hardware changes have been made, and only regular maintenance of updating packages and software have occurred to keep things on the most recent stable releases.
-
@sgnoc said in Suricata blocking IPs on passlist, legacy mode blocking both:
This is netgate hardware that has been running for years without these issues until Suricata 7, and it is only affecting Suricata.
The pfSense kernel also changed with the Suricata update. The new Suricata version came as part of the 23.09 pfSense Plus and 2.7.1 CE rollouts. So, more than just Suricata has changed.
My advice at this point is to remove Suricata from your environment. That will definitely solve your unwanted blocking issues.
