Suricata custom rule alerts but won't block
-
@kiokoman said in Suricata custom rule alerts but won't block:
do you think that if one ip is on the passlist and the "Which IP to Block" is set to both it will ignore both ip instead of only the one in the passlist?
No, but it is possible (but unlikely) that if the Source IP is on a Pass List the code could erroneously assume the Destination IP is also on the list.
-
@RedDelPaPa said in Suricata custom rule alerts but won't block:
@bmeeks Oh I see. Thank you.
Is there a way I can set surricata to not log certain alerts, but to continue blocking them? I really need to clear out all the SIP stuff so I can watch for more important things.
No. It is not possible to suppress alert logging but still block. The blocking plugin actually works off receiving the alerts, so without getting the alerts the blocking plugin will see nothing to block (as it won't see alerts from that rule or host).
-
@bmeeks Ack. Thank you.
I will get back to you later with the results of that test you requested.
-
@bmeeks Ok guess what?
I ran your test with my original rule:
alert tcp $SMTP_SERVERS 25 -> $EXTERNAL_NET any (msg:"Custom SMTP AUTH LOGON brute force attempt"; flow:from_server,established; content:"Authentication failed"; offset:54; nocase; threshold:type threshold, track by_dst, count 3, seconds 60; classtype:suspicious-login; sid:1000001; rev:3;)
Changing the "which IP to block" to DST from BOTH, and restarting suricata, now alerts and blocks as it should.
Changing "which IP to block" back to BOTH from DST, restarting suricata and running the same test again with the same rule alerts but will not block.
What does this mean?
-
@RedDelPaPa said in Suricata custom rule alerts but won't block:
@bmeeks Ok guess what?
I ran your test with my original rule:
alert tcp $SMTP_SERVERS 25 -> $EXTERNAL_NET any (msg:"Custom SMTP AUTH LOGON brute force attempt"; flow:from_server,established; content:"Authentication failed"; offset:54; nocase; threshold:type threshold, track by_dst, count 3, seconds 60; classtype:suspicious-login; sid:1000001; rev:3;)
Changing the "which IP to block" to DST from BOTH, and restarting suricata, now alerts and blocks as it should.
Changing "which IP to block" back to BOTH from DST, restarting suricata and running the same test again with the same rule alerts but will not block.
What does this mean?
Thank you for the test results feedback. That gives me a valuable clue about processing flow in the custom plugin I wrote. I think I know how to fix the issue in the source code, but I'm not sure why the fix is needed. It's a sort of long explanation, but the short version is a variable is passed to another built-in Suricata function to search a list of IP addresses in order to determine if a given IP is present in the list. So essentially this is searching the Pass List for one of the IP addresses from the current alerting packet. When BOTH is selected for the "Which IP to Block" parameter, two consecutive calls are made to this function. One with the SRC IP and then another following with the DST IP. The variable passed that indicates if the IP was found or not is supposedly reset by the built-in Suricata function each time. But your results indicate that perhaps it is not always getting reset. Very weird.
Nonetheless, I will make the change in the C source code of my custom blocking plugin. It will take at least a couple of days to get it posted and for the pfSense developer team to merge the change into the package repositories.
TL;DR; section for programming geeks --
When the "Which IP to Block" parameter is set to BOTH, two consecutive calls are made to the built-in Suricata function SCRadixFindKeyBestMatchIPv4() for IPv4 addresses or SCRadixFindKeyBestMatchIPv6() for IPv6 addresses. With each call to this function, a pointer variable is passed that will be set to NULL if the IP passed is not present in the Radix Tree. If the IP is present in the tree, the variable will be filled with a pointer to the user data associated with that IP entry. In the custom blocking module used by Suricata, when the BOTH mode is enabled, the first call to the Radix Tree search function is made using the SRC IP from the alerting packet. If the SRC IP is not found in the Radix Tree (which is actually the pass list), then the SRC IP is blocked. Next, a call is made to the same function to search the Radix Tree for the DST IP. What is happening, and I can't really explain why yet, is that variable that is supposed to be set to NULL when an IP is not in the list is actually retaining the old value from the first SRC IP search. Thus the code mistakenly thinks the DST IP is on the pass list whenever the SRC IP is on the list.
So applying the described logic to your rule above, it's working this way. When set to BOTH, the first call to the Radix Tree search is done with the SRC IP from the offending packet. In your rule, the SRC IP will be your mail server. Your mail server is on your local network and thus its IP address will automatically be in the default pass list. So the Radix Tree search finds that the IP address of your mail server (the SRC IP in your rule) is covered by the pass list so it does not block that IP. Next, the code makes another call to the Radix Tree search function but this time with the DST IP from the alert (in your rule, that would be $EXTERNAL_NET). But now, due to the bug described above, the search function returns a value indicating that the $EXTERNAL_NET IP is also in the pass list when in reality it is not. Thus the $EXTERNAL_NET IP is not blocked.
By switching the "Which IP to Block" mode over to DST only, that results in only a single call being made to the SCRadixFindKeyBestMatchIPv()* function. The function then correctly determines the IP is not in the pass list and thus it is blocked.
-
@bmeeks Glad I could be of some help, Bill! How will I know when the fix is ready for download?
-
@RedDelPaPa said in Suricata custom rule alerts but won't block:
@bmeeks Glad I could be of some help, Bill! How will I know when the fix is ready for download?
A new package version will appear in the Package Manager section under SYSTEM > PACKAGE MANAGER.
-
@bmeeks Ack, thank you.
-
@RedDelPaPa said in Suricata custom rule alerts but won't block:
@bmeeks Ack, thank you.
Suricata upstream just released version 4.1.5 of the Suricata binary today (September 24, 2019), so give me an extra day or two so I can also include the updated binary with my custom blocking fix.
-
@bmeeks Ok thanks! I will watch for it.
-
@kiokoman Something along those lines is exactly what's happening. See my rule test results above. Bill has submitted a code fix to solve this issue.
-
The pull request to hopefully fix this issue has been submitted first to the pfSense-2.5-DEVEL branch. It should show up as an available update under SYSTEM > PACKAGE MANAGER once it is merged by the pfSense developer team. Here is a link to the request: https://github.com/pfsense/FreeBSD-ports/pull/676.
I will let this test for a few days in the pfSense-2.5-DEVEL snapshot world. If no issues are identifed, I will ask the pfSense team to move the change to the 2.4.4 RELEASE branch.
-
@bmeeks does this mean we’re not going to see this fix for quite some time?
-
@RedDelPaPa said in Suricata custom rule alerts but won't block:
@bmeeks does this mean we’re not going to see this fix for quite some time?
No, it should be just a few days at most. I will test it some more, and hopefully once it shows up in the snapshot branch it will get tested there. If no showstopper issue pops up from the snapshot crowd, it should be ready for RELEASE.
-
@bmeeks I did some more testing of those rules today using telnet and then trying to send auth login. Even with both rules, I noticed that Suricata is very very slow to alert and block. It had as much as a two minute delay between the time I sent the auth command that the rules are scanning for and the time that Suricata picked it up and blocked my telnet session. Of course, I have it running in Legacy mode, but is it normal for it to take that long to recognize and block?
-
@RedDelPaPa said in Suricata custom rule alerts but won't block:
@bmeeks I did some more testing of those rules today using telnet and then trying to send auth login. Even with both rules, I noticed that Suricata is very very slow to alert and block. It had as much as a two minute delay between the time I sent the auth command that the rules are scanning for and the time that Suricata picked it up and blocked my telnet session. Of course, I have it running in Legacy mode, but is it normal for it to take that long to recognize and block?
Two minutes does not sound realistic at all. How are you measuring the blocking time? If you are watching the GUI's ALERTS tab or the BLOCKS tab, that will be very misleading because those pages only update their displayed results every 60 seconds by default. The actual blocking happens within a fraction of a second (typically a few milliseconds at most between the packet and the block, and that would be if a number of packets had to be analyzed before the rule had enough info to trigger).
-
@bmeeks so the way I’m doing it is, I will login to my Mail server using telnet from a remote location, and then I will immediately send the auth login command. Then I’m counting how long I can continue to mess around in the Telnet session before the connection gets dropped and blocked. And it was taking an average of a minute to kill the connection and in some cases two minutes.
-
@bmeeks Ok, I just ran wire shark capture and discovered why it takes so long to block when "auth login" is sent to my mailserver via telnet. Telnet sends 1 packet at a time with one character at a time in each packet. 1 A 2 U 3 T 4 H, etc. A mail client sends the entire "auth login" in one packet. Making it much easier and faster for suricata to alert on.
-
@RedDelPaPa said in Suricata custom rule alerts but won't block:
@bmeeks Ok, I just ran wire shark capture and discovered why it takes so long to block when "auth login" is sent to my mailserver via telnet. Telnet sends 1 packet at a time with one character at a time in each packet. 1 A 2 U 3 T 4 H, etc. A mail client sends the entire "auth login" in one packet. Making it much easier and faster for suricata to alert on.
That is unwittingly a beautiful demonstration of how an IDS actually works at the packet level. It must receive enough of the flow to see all of the data specified by the signature. It blocks as soon as it has all the data to trigger the rule, but it may need to wait for a ton of packets before the rule has enough data to match and trigger on.
This also illustrates, somewhat, the difference between Legacy Mode and Inline IPS Mode blocking. With Legacy Mode, because PCAP is used to capture data, the original packets are continuing on to the network stack of pfSense while Suricata is analyzing a copy of the original packets. Once Suricata sees enough data to trigger the rule, it inserts a block of the offender's IP in the firewall and then clears any open states established by the original traffic that was allowed in while Suricata was analyzing the copy. Inline IPS Mode works differently. It literally sits between the network card hardware driver and the pfSense network stack. The only way data can get from the NIC to pfSense is by passing through the
netmappipe created by Suricata. In that mode, Suricata will buffer traffic for the flow until it has seen the entire exchange, then it will analyze the data and either drop the traffic or pass it on. -
@bmeeks Yes that makes perfect sense. 60 seconds to block though still seams and 50 seconds too long though when trying to fire the block via telnet.
I can't use inline mode though because I'm using traffic shaping.
