[Solved] Suricata disablesid.conf
-
@Raffi.:
I just tried the same on my WAN. The grep search still found the sid in the file even after creating the new single entry in disablesid2.conf file. The odd thing is that the sid log show 1 found but no rules that match. Maybe that's a clue? Why no matches when the grep is showing the rule is there?
/usr/local/etc/suricata/rules: grep -r "2018959" /usr/local/etc/suricata/rules
/usr/local/etc/suricata/rules/emerging-policy.rules:alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET POLICY PE EXE or DLL Windows file download HTTP"; flow:established,to_client; flowbits:isnotset,ET.http.binary; flowbits:isnotset,ET.INFO.WindowsUpdate; file_data; content:"MZ"; within:2; byte_jump:4,58,relative,little; content:"PE|00 00|"; distance:-64; within:4; flowbits:set,ET.http.binary; reference:url,doc.emergingthreats.net/bin/view/Main/2000419; classtype:policy-violation; sid:2018959; rev:3; metadata:created_at 2014_08_19, updated_at 2017_02_01;)Raffi
You are looking in the wrong place for the active rules. Your path of /usr/etc/local/suricata/rules is the main repository location for all the downloaded rules. It is the big bucket where rules are picked from to include in the suricata.rules file for an interface. That is not the actual list of "in use" rules for an interface. You will find those in a file in a sub-directory underneath such as /usr/local/etc/suricata/suricata_xxxxx/rules. The "xxxxx" part will be a random GUID and the physical interface name.
For example, on my test machine the path to the active rules for the LAN is:
/usr/local/etc/suricata/suricata_59991_em1/rules/suricata.rules
Bill
-
Thanks, I wasn't sure if I had the right path. Here is the new result.
[2.4.3-RELEASE][admin@pfsense.telebyte.localdomain]/usr/local/etc/suricata/suricata_37173_em0/rules: grep -r "2018959" /usr/local/etc/suricata/suricata_37173_em0/rules/
/usr/local/etc/suricata/suricata_37173_em0/rules/flowbit-required.rules:# Category: emerging-policy GID:1 SID:2018959
/usr/local/etc/suricata/suricata_37173_em0/rules/flowbit-required.rules:alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET POLICY PE EXE or DLL Windows file download HTTP"; flow:established,to_client; flowbits:isnotset,ET.http.binary; flowbits:isnotset,ET.INFO.WindowsUpdate; file_data; content:"MZ"; within:2; byte_jump:4,58,relative,little; content:"PE|00 00|"; distance:-64; within:4; flowbits:set,ET.http.binary; reference:url,doc.emergingthreats.net/bin/view/Main/2000419; classtype:policy-violation; sid:2018959; rev:3; metadata:created_at 2014_08_19, updated_at 2017_02_01;)
[2.4.3-RELEASE][admin@pfsense.telebyte.localdomain]/usr/local/etc/suricata/suricata_37173_em0/rules:Raffi
-
It would be pleasant to be able to search for rules…some rule is blocking contact.ebay.com and it's not in emerging-policy.rules
-
@Raffi.:
Thanks, I wasn't sure if I had the right path. Here is the new result.
[2.4.3-RELEASE][admin@pfsense.telebyte.localdomain]/usr/local/etc/suricata/suricata_37173_em0/rules: grep -r "2018959" /usr/local/etc/suricata/suricata_37173_em0/rules/
/usr/local/etc/suricata/suricata_37173_em0/rules/flowbit-required.rules:# Category: emerging-policy GID:1 SID:2018959
/usr/local/etc/suricata/suricata_37173_em0/rules/flowbit-required.rules:alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET POLICY PE EXE or DLL Windows file download HTTP"; flow:established,to_client; flowbits:isnotset,ET.http.binary; flowbits:isnotset,ET.INFO.WindowsUpdate; file_data; content:"MZ"; within:2; byte_jump:4,58,relative,little; content:"PE|00 00|"; distance:-64; within:4; flowbits:set,ET.http.binary; reference:url,doc.emergingthreats.net/bin/view/Main/2000419; classtype:policy-violation; sid:2018959; rev:3; metadata:created_at 2014_08_19, updated_at 2017_02_01;)
[2.4.3-RELEASE][admin@pfsense.telebyte.localdomain]/usr/local/etc/suricata/suricata_37173_em0/rules:Raffi
The rule (2018959) has a flowbit parameter (that flowbits:set,ET.http.binary string in the rule). Since that rule "sets" a flowbit that is later checked by a different rule that has a matching "isset" flowbit keyword, then the automatic flowbit rule resolution process is enabling that rule for you automatically. You have two options. First, you can simply suppress that 2018959 rule to prevent the alert (and block). It would still set the flowbit when needed that some other rules depend on. Option 2 is to disable that rule. But if you do that, then it will not set that dependent flowbit and the other rules with the "isset" check on that flowbit will fail to trigger because the dependent flowbit is not set.
Bill
-
Thanks for the explanation Bill. You learn something new everyday. I have that rule suppressed and seems to be fine.
Is this normal behavior? If so, how come it didn't come up as a flowbit rule when you did the grep? Does it depend on rules you have active?
It looks like the disablesid.conf can't be 100% relied on for disabling rules. This behavior was very confusing to me and may be to others as well. Is there anyway that the sid management could automatically suppress a rule that it can't disable due to flowbits? If that's not possible maybe having the sid mgmt log display a message stating that it was not able to disable the rule due to flowbits. This would give the user some insight as to what's going on. I thought it was a bug, but it does not seem to be.Raffi
-
@Raffi.:
Thanks for the explanation Bill. You learn something new everyday. I have that rule suppressed and seems to be fine.
Is this normal behavior? If so, how come it didn't come up as a flowbit rule when you did the grep? Does it depend on rules you have active?
It looks like the disablesid.conf can't be 100% relied on for disabling rules. This behavior was very confusing to me and may be to others as well. Is there anyway that the sid management could automatically suppress a rule that it can't disable due to flowbits? If that's not possible maybe having the sid mgmt log display a message stating that it was not able to disable the rule due to flowbits. This would give the user some insight as to what's going on. I thought it was a bug, but it does not seem to be.Raffi
Yes, the total of your enabled rules influences what flowbit rules may be required. You can always see the auto-enabled flowbit rules on the RULES tab by selecting that category to view in the drop-down selector.
Every IDS/IPS user needs to go search Google for flowbits to learn what they are and how they work. That will help you understand why the flowbits process works the way it does in the GUI.
Anytime you have a rule triggering that you think should be disabled, look first in the auto-flowbits set to see if the rule is in there. If it is and you don't want to see alerts from it, then suppress that SID.
I've used this crude example in the past to describe what flowbits are all about. Flowbits are mechanisms for rules to store "states" in the processing of packet information. Flowbits are TCP session-based. That means they stay set for the duration of a TCP session. Suppose a certain pattern of bytes in a PDF file indicates that file contains a malicious payload. However, that pattern of bytes is only malicious when it is part of a PDF. The same pattern of bytes in a GIF, for instance, may be totally harmless. So if I just made a rule that said "anytime I see this pattern of bytes I'm going to alert", then I would experience false positives from my rule any time some random sequence of bytes happens to match my pattern in something like an MP3 music file, a GIF graphic, etc. I would also match on that pattern in a PDF file (which is malicious in my example).
So it would be really nice if the detection engine could know when a PDF file was being downloaded and only then would it enable my rule to look for that malicious byte pattern. For any other type of traffic, my rule would not trigger even if the same malicious byte pattern were present. Flowbits let this happen. When a PDF file download session starts there are some indicators a rule can match on that signal a PDF file is downloading. So when those indicators are seen in a stream a flowbit for "PDF file download" is set. My rule that is looking for the malicious PDF payload pattern will have an option set to check that "PDF file download" flowbit by using the "isset" keyword. Only when the flowbit is set (i.e., isset = TRUE) will my rule fire. If the flowbit is not set, the rule does not fire and thus won't false positive needlessly.
In the actual rule giving you trouble, the flowbit name was "ET.http.binary". The name is totally arbitrary and is selected by the rule author. The only important thing about the name is that later, when checking if the flowbit is set, you must use the exact same name. So another rule would have the action keyword "flowbits:isset,ET.http.binary". Only when that condition is TRUE (meaning a flowbit with that name is "set") will the rule trigger an alert.
Bill
-
Awesome. Thanks for the education. I have a feeling I won't find that explanation on google. It makes prefect sense without getting too technical. You should be writing books on this stuff if you aren't already. I will do some more reading up on flowbits.
Raffi
-
@Raffi.:
Awesome. Thanks for the education. I have a feeling I won't find that explanation on google. It makes prefect sense without getting too technical. You should be writing books on this stuff if you aren't already. I will do some more reading up on flowbits.
Raffi
Thanks! Writing a book is not my thing, though… :).
So back to the rule that was giving you trouble, that flowbit flag literally means it is from an Emerging Threats rule (the "ET" prefix) and then the rest of the tag means "an HTTP binary download is in progress". You can infer all of that from the name the rule author used. There are no hard and fast rules for flowbit names, though. As I mentioned, the only requirement is that every test for "isset" obviously can only be TRUE when you are checking for an exact name. The rule author could have just as easily named his flowbit "Joes.binary.download" or whatever tickled his fancy. Luckily he chose a name that helps us discern what the rule really is for. So rule 2018959 has a critical job of setting a flowbit to signal a binary download over HTTP is happening in the current session. Subsequent rules can now test that flowbit to see if a binary download over HTTP is happening, and if it is, their content may match and "fire" an alert.
Let's look at that 2018959 rule in a bit more detail to see what it is really up to:
alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET POLICY PE EXE or DLL Windows file download HTTP"; flow:established,to_client; flowbits:isnotset,ET.http.binary; flowbits:isnotset,ET.INFO.WindowsUpdate; file_data; content:"MZ"; within:2; byte_jump:4,58,relative,little; content:"PE|00 00|"; distance:-64; within:4; flowbits:set,ET.http.binary; reference:url,doc.emergingthreats.net/bin/view/Main/2000419; classtype:policy-violation; sid:2018959; rev:3; metadata:created_at 2014_08_19, updated_at 2017_02_01;)Look at the "flowbits" tags in the rule. Basically this rule is setting a flag for other rules to look at, but it is checking some things first. The first flowbits check is to test that the "ET.http.binary" flag is not already set. If it is, then we don't need to set it again so the 2018959 rule won't "fire" if that flowbit is already set. Next a check is made to see if the flowbits for "ET.INFO.WindowsUpdate" is set. If it is, then rule 2018959 won't fire (see, it is testing for "isnotset" on that flowbit, so if that flowbit is actually set it inhibits this rule from firing). The idea is that a DLL or binary coming from Windows Update is OK. Other binaries or DLLs over HTTP may not be OK, so if the first two conditions are true (the ET.http.binary flag is not set and the ET.INFO.WindowsUpdate flag is not set) and the content of the packet indicates a binary transfer is happening, the rule fires by setting the flowbit for "ET.http.binary".
One thing that Snort Subscriber rules do that Emerging Threats does not (at least not as often) is also use the "flowbits:noalert" tag in their rules that set flowbits. When the "flowbits:noalert" tag is present, the rule will not generate an alert. So the text of rule 2018959 would be better in my opinion if it contained this set of flowbit actions:
flowbits:set,ET.http.binary;flowbits:noalert;This would still set the "ET.http.binary" flowbit, but it would also automatically suppress the alert from the rule firing. Pretty much every Snort Subscriber rule that sets a flowbit includes the "flowbits:noalert" tag as well. I don't see that as often in the Emerging Threats rules.
Bill
-
Cool, I think I actually understand it. The rule only fires if all the flowbit checks are true? isnoset is basically an inverse check. If the flag is not set, then it's a true statement and it goes onto the next flowbit check if there is one.
Raffi
-
@Raffi.:
Cool, I think I actually understand it. The rule only fires if all the flowbit checks are true? isnoset is basically an inverse check. If the flag is not set, then it's a true statement and it goes onto the next flowbit check if there is one.
Raffi
Yes, and there are many different combinations of flowbits checks. There are keywords for "set", "isset", "isnotset" and "noalert". So flowbits are a mechanism that allow a number of rules to cooperate with each other over time as a data transfer occurs during a given session by setting flags other rules can check.
One key takeaway, though, is that any rule that has a "flowbits:isset" or "flowbits:isnotset" check in it can never fire if the rule that sets that flowbit is disabled or not loaded. That's why the auto-flowbits feature I put in the GUI is important. It scans all of your enabled rules to find any and all rules that contain "flowbits:isset" or "flowbits:isnotset" checks in them, and then it makes sure that at least one "flowbits:set" rule is enabled for each flowbit name it finds in your enabled rules. So that is how SID 2018959 got "enabled behind your back" so to speak. It was the auto-flowbits logic in the GUI code. It found at least one enabled rule in your setup that was testing the "ET.http.binary" flowbit, but it did not find any enabled rule that was setting that same flowbit. So it went looking through your rules until it found one that set that flowbit and then enabled that rule. It was helping you not shoot yourself in the foot by disabling a rule that other enabled rules depended on. If rule 2018959 had remained disabled, then that "ET.http.binary" flowbit would never get set and thus any of your other enabled rules that needed to see that flowbit set in order to fire would never see it and thus never fire. You would have disabled other rules by killing their flowbit dependency and would not have even known it.
P.S. – oh, and for others who may have followed this thread; all of this description of flowbits is true for Snort as well. In fact, the whole idea of flowbits began in Snort and was later incorporated into Suricata by the Suricata developers.
Bill
-
Nice tutorial so to speak, maybe you could do a sticky post, in order for others to find it more easily in the future?
Where to read more about rules tips & tricks ? "How to create Snort rules documentation", on their site is ok?
-
Nice tutorial so to speak, maybe you could do a sticky post, in order for others to find it more easily in the future?
Where to read more about rules tips & tricks ? "How to create Snort rules documentation", on their site is ok?
There is some useful documentation on the Snort.org site. However, to be honest, I've never found a great all-in-one location for this kind of information. Bits and pieces are scattered all over. As with lots of software, especially open-source and other "free" software, the developers spend more time on coding and adding features than creating documentation. I am guilty of that as well with the Snort and Suricata packages.
Bill
