[Solved] Suricata disablesid.conf

Raffi_

2.4.3-RELEASE
Suricata 4.0.4_1
Legacy Mode

I have been noticing some more false positives then usual in Suricata lately. I looked at the specific sid and it's usually 1:2018959 # "ET POLICY PE EXE or DLL Windows file download HTTP".
This seems to be blocking sites like Microsoft Office 365 apps and pages. No big deal, I added it to my suppress list. I then thought, maybe I should instead add it to my disablesid.conf. So I added it there, and as I was looking through the file, I saw it was already on there! Interesting. So why is it not being disabled on startup? Is my file wrong or a setting not right? My file is below. I also attached a screen shot of the sid settings and sid log. According to the log it seems to be loading the disablesid.conf and applying the changes.


# disabled SIDs, https://forum.pfsense.org/index.php?topic=56267.75
1:536        #  "DELETED NETBIOS SMB D$ share access"
1:648        #  "INDICATOR-SHELLCODE x86 NOOP"
1:653        #  "DELETED SHELLCODE x86 0x90 unicode NOOP"
1:1390       #  "INDICATOR-SHELLCODE x86 inc ebx NOOP"
1:2452       #  "POLICY-SOCIAL Yahoo IM ping"
1:8375       #  "BROWSER-PLUGINS QuickTime Object ActiveX clsid access"
1:11192      #  "FILE-EXECUTABLE download of executable content"
1:12286      #  "FILE-OTHER PCRE character class heap buffer overflow attempt"
1:15147      #  "BROWSER-IE Microsoft Internet Explorer malformed iframe buffer overflow attempt"
1:15306      #  "FILE-EXECUTABLE Portable Executable binary file magic detected"
1:15362      #  "INDICATOR-OBFUSCATION obfuscated javascript excessive fromCharCode - potential attack"
1:16313      #  "FILE-EXECUTABLE download of executable content"
1:16482      #  "BROWSER-IE Microsoft Internet Explorer userdata behavior memory corruption attempt"
1:17458      #  "FILE-OTHER BitDefender Internet Security script code execution attempt"
1:20583      #  "BROWSER-FIREFOX Mozilla multiple location headers malicious redirect attempt"
1:23098      #  "FILE-MULTIMEDIA Adobe Flash Player MP4 sequence parameter set parsing overflow attempt"
1:23256      #  "FILE-EXECUTABLE Armadillo v1.71 packer file magic detected"
1:24889      #  "FILE-FLASH Adobe Flash Player Action InitArray stack overflow attempt"
1:2000334    #  "ET P2P BitTorrent peer sync"
1:2000418    #  "ET POLICY Executable and linking format (ELF) file download"
1:2000419    #  "ET POLICY PE EXE or DLL Windows file download"
1:2002878    #  "ET POLICY iTunes User Agent"
1:2003195    #  "ET POLICY Unusual number of DNS No Such Name Responses"
1:2007727    #  "ET P2P possible torrent download"
1:2008120    #  "ET TFTP Outbound TFTP Read Request"
1:2008578    #  "ET SCAN Sipvicious Scan"
1:2008989    #  "ET POLICY Internal Host Retrieving External IP via showmyip.com - Possible Infection"
1:2010516    #  "ET WEB_CLIENT Possible HTTP 403 XSS Attempt (External Source)"
1:2010525    #  "ET WEB_CLIENT Possible HTTP 500 XSS Attempt (External Source)"
1:2010935    #  "ET POLICY Suspicious inbound to MSSQL port 1433"
1:2010937    #  "ET POLICY Suspicious inbound to mySQL port 3306"
1:2011716    #  "ET SCAN Sipvicious User-Agent Detected (friendly-scanner)"
1:2012078    #  "ET POLICY Windows-Based OpenSSL Tunnel Outbound"
1:2012086    #  "ET DELETED Possible Call with No Offset TCP Shellcode"
1:2012087    #  "ET SHELLCODE Possible Call with No Offset UDP Shellcode"
1:2012088    #  "ET SHELLCODE Possible Call with No Offset TCP Shellcode"
1:2012089    #  "ET SHELLCODE Possible Call with No Offset UDP Shellcode"
1:2012141    #  "ET POLICY Protocol 41 IPv6 encapsulation potential 6in4 IPv6 tunnel active"
1:2012252    #  "ET SHELLCODE Common 0a0a0a0a Heap Spray String"
1:2012758    #  "ET INFO DYNAMIC_DNS Query to *.dyndns. Domain"
1:2013028    #  "ET POLICY curl User-Agent Outbound"
1:2013031    #  "ET POLICY Python-urllib/ Suspicious User Agent"
1:2013222    #  "ET DELETED Excessive Use of HeapLib Objects Likely Malicious Heap Spray Attempt"
1:2013414    #  "ET POLICY Executable served from Amazon S3"
1:2013504    #  "ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management"
1:2014472    #  "ET INFO JAVA - Java Archive Download"
1:2014518    #  "ET INFO EXE - OSX Disk Image Download"
1:2014520    #  "ET INFO EXE - Served Attached HTTP"
1:2014726    #  "ET POLICY Outdated Flash Version M1"
1:2014734    #  "ET P2P BitTorrent - Torrent File Downloaded"
1:2014819    #  "ET INFO Packed Executable Download"
1:2015561    #  "ET INFO PDF Using CCITTFax Filter"
1:2015744    #  "ET INFO EXE IsDebuggerPresent (Used in Malware Anti-Debugging)"
1:2015820    #  "ET INFO Suspicious Windows NT version 7 User-Agent"
1:2016360    #  "ET INFO JAVA - ClassID"
1:2016877    #  "ET POLICY Unsupported/Fake FireFox Version 2."
1:2017364    #  "ET INFO SUSPCIOUS Non-standard base64 charset used for encoding"
1:2018959    #  "ET POLICY PE EXE or DLL Windows file download HTTP"
1:2019416    #  "ET POLICY SSLv3 outbound connection from client vulnerable to POODLE attack"
1:2022913    #  "ET INFO WinHttp AutoProxy Request wpad.dat Possible BadTunnel"
1:2100366    #
1:2100368    #
1:2100651    #
1:2101390    #
1:2101424    #
1:2102314    #
1:2103134    #
1:2103192    #
1:2200075    #
1:2210000    #
1:2210007    #
1:2210020    #
1:2210036    #
1:2210042    #
1:2210044    #
1:2210045    #
1:2210050    #
1:2210054    #
1:2221028    #
1:2230010    #
1:2402000    #  "ET DROP Dshield Block Listed Source group 1"
1:2403323    #  "ET CINS Active Threat Intelligence Poor Reputation IP UDP group 12"
1:2403344    #  "ET CINS Active Threat Intelligence Poor Reputation IP TCP group 23"
1:2406003    #
1:2406067    #
1:2406069    #
1:2406424    #
1:2500050    #  "ET COMPROMISED Known Compromised or Hostile Host Traffic TCP group 26"
1:2500056    #  "ET COMPROMISED Known Compromised or Hostile Host Traffic TCP group 29"
1:2520199    #
1:2520205    #
1:2522614    #  "ET TOR Known Tor Relay/Router (Not Exit) Node TCP Traffic group 308"
1:2522618    #  "ET TOR Known Tor Relay/Router (Not Exit) Node TCP Traffic group 310"
1:100000230  #
3:14772      #  "FILE-IMAGE libpng malformed chunk denial of service attempt"
3:19187      #  "PROTOCOL-DNS TMG Firewall Client long host entry exploit attempt"
3:21355      #  "PROTOCOL-DNS potential dns cache poisoning attempt - mismatched txid"
119:2        #  "HI_CLIENT_DOUBLE_DECODE"
119:4        #  "HI_CLIENT_BARE_BYTE"
119:7        #  "HI_CLIENT_IIS_UNICODE"
119:14       #  "HI_CLIENT_NON_RFC_CHAR"
119:31       #  "HI_CLIENT_UNKNOWN_METHOD"
119:32       #  "HI_CLIENT_SIMPLE_REQUEST"
119:33       #  "HI_CLIENT_UNESCAPED_SPACE_IN_URI"
120:2        #  "HI_SERVER_INVALID_STATCODE"
120:3        #  "HI_SERVER_NO_CONTLEN"
120:4        #  "HI_SERVER_UTF_NORM_FAIL"
120:6        #  "HI_SERVER_DECOMPR_FAILED"
120:8        #  "HI_CLISRV_MSG_SIZE_EXCEPTION"
120:9        #  "HI_SERVER_JS_OBFUSCATION_EXCD"
120:10       #  "HI_SERVER_JS_EXCESS_WS"
122:19       #  "PSNG_UDP_PORTSWEEP"
122:21       #  "PSNG_UDP_FILTERED_PORTSCAN"
122:22       #  "PSNG_UDP_FILTERED_DECOY_PORTSCAN"
122:23       #  "PSNG_UDP_PORTSWEEP_FILTERED"
122:26       #  "PSNG_ICMP_PORTSWEEP_FILTERED"
123:10       #  "FRAG3_IPV6_BAD_FRAG_PKT"
124:3        #  "SMTP_RESPONSE_OVERFLOW"
125:2        #  "FTPP_FTP_INVALID_CMD"
137:1        #  "SSL_INVALID_CLIENT_HELLO"
138:2        #  "SENSITIVE-DATA Credit Card Numbers"
138:3        #  "SENSITIVE-DATA U.S. Social Security Numbers (with dashes)"
138:4        #  "SENSITIVE-DATA U.S. Social Security Numbers (w/out dashes)"
138:5        #  "SENSITIVE-DATA Email Addresses"
138:6        #  "SENSITIVE-DATA U.S. Phone Numbers"
141:1        #  "IMAP_UNKNOWN_CMD"
#END disabled SIDs from PFSense Forum

#Personal additions
1:2016538 #akamaitechnologies.com ET INFO Executable Retrieved With Minimal HTTP Headers - Potential Second Stage Download
1:2020565 #Dropbox AWSDNS
1:2014380 #Allow speedtest.net
#ET SCAN Sipvicious User-Agent Detected (friendly-scanner)
suppress gen_id 1, sig_id 2011716
#ET SCAN Sipvicious Scan
suppress gen_id 1, sig_id 2008578
#ET DROP Dshield Block Listed Source group 1
suppress gen_id 1, sig_id 2402000
#ET CINS Active Threat Intelligence Poor Reputation IP group 65
suppress gen_id 1, sig_id 2403364
#ET CINS Active Threat Intelligence Poor Reputation IP group 3
suppress gen_id 1, sig_id 2403302
#ET CINS Active Threat Intelligence Poor Reputation IP group 82
suppress gen_id 1, sig_id 2403381
1:2013505    #  "ET POLICY GNU/Linux YUM User-Agent Outbound likely related to package management"

Any ideas? Thanks.
Raffi
![SID log.JPG](/public/imported_attachments/1/SID log.JPG)
![SID log.JPG_thumb](/public/imported_attachments/1/SID log.JPG_thumb)
![SID Mgmt.JPG](/public/imported_attachments/1/SID Mgmt.JPG)
![SID Mgmt.JPG_thumb](/public/imported_attachments/1/SID Mgmt.JPG_thumb)

bmeeks

What is the state shown for that rule on the RULES tab? Is it perhaps "forced" there to an enabled state? The user overrides on the RULES tab are the last things applied to the rule set, so if a rule is altered there, that will override any other settings including those on the SID MGMT tab.

You have a lot more than 12 rules listed in your disablesid.conf file, but the system is only finding 12 matches between your enabled rules categories and the values in the disablesid.conf file. So only 12 of the rules you have listed in the file are actually being disabled according to the log. That would mean all the others listed are not matching.

You can always view your actual "in use" rules by using the new category "Active Rules" on the RULES tab. You can also find the actual file containing the active rules in the path /usr/local/etc/suricata/suricata_xxxxx/rules/suricata.rules. The "xxxxx" will be a GUID and the physical interface name. You can search through that file to see what is actually being used and compare it to what you have enabled and then disabled via SID MGMT. If you find what you believe is a logic error in the PHP code, then post back or open a bug report and I will take a look.

Bill

Raffi_

Hi Bill,

Thanks for the advice. The active rules shows that rule as being auto disabled by the SID mgmt tab (see attached). That doesn't seem to be the case though based on my alerts and blocks tabs. I also found it odd that I only had 12 matches, but it's possible that most don't apply since I copied it over from when I was using snort. I may end up saving this current list as a snort backup disablesid.conf in case I want to switch back to snort. I could then slim down the actual list for Suricata so that I don't have as many entries which are doing nothing. The cleaner, the better.

I made sure I checked rebuild on the WAN interface before hitting save on the SID mgmt tab and I even restarted Suricata, but I was still getting that rule triggering blocks. I haven't rebooted since updating to the latest pfSense build which was almost 5 days ago. I could also try uninstalling and reinstalling.

Thanks,
Raffi

![Active Rules.JPG](/public/imported_attachments/1/Active Rules.JPG)
![Active Rules.JPG_thumb](/public/imported_attachments/1/Active Rules.JPG_thumb)

Raffi_

It looks like uninstalling and reinstalling the package fixed it. I will let you know otherwise. Maybe after the pfSense update, something didn't go so smoothly with Suricata?

Raffi_

It was triggered again so it's not solved. Adding it to the suppress list I think is working in the meantime. How do I report a bug?

Thanks,
Raffi

bmeeks

@Raffi.:

It was triggered again so it's not solved. Adding it to the suppress list I think is working in the meantime. How do I report a bug?

Thanks,
Raffi

You can report bugs here: https://redmine.pfsense.org/projects/pfsense-packages.

Bill

bmeeks

I just tested this in a virtual machine setup and I could not duplicate your problem. I configured an instance of Suricata on a LAN interface and enabled the ET-Policy rules. I then did a grep search of the actual rules file (suricata.rules in the appropriate interface sub-directory under /usr/local/etc/suricata) and the rule SID 2018959 was present in the file as expected.

I then created a disablesid.conf configuration under SID management and added this single line to it:

1:2018959

I selected that list in the Disable SID List drop-down for the LAN, checked the box to rebuild the interface rules and clicked Save.

I went back and repeated the grep search for "2018959" and the rule was not present in the file. So the rule has been disabled by virtue of the fact it is not present in the enforcing rules file for the interface.

Are you 100% positive that you don't accidentally have that rule SID force "on" by maybe clicking on it in the past on the RULES tab?

Bill

Raffi_

I just tried the same on my WAN. The grep search still found the sid in the file even after creating the new single entry in disablesid2.conf file. The odd thing is that the sid log show 1 found but no rules that match. Maybe that's a clue? Why no matches when the grep is showing the rule is there?

/usr/local/etc/suricata/rules: grep -r "2018959" /usr/local/etc/suricata/rules
/usr/local/etc/suricata/rules/emerging-policy.rules:alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET POLICY PE EXE or DLL Windows file download HTTP"; flow:established,to_client; flowbits:isnotset,ET.http.binary; flowbits:isnotset,ET.INFO.WindowsUpdate; file_data; content:"MZ"; within:2; byte_jump:4,58,relative,little; content:"PE|00 00|"; distance:-64; within:4; flowbits:set,ET.http.binary; reference:url,doc.emergingthreats.net/bin/view/Main/2000419; classtype:policy-violation; sid:2018959; rev:3; metadata:created_at 2014_08_19, updated_at 2017_02_01;)

Raffi

![no matches.JPG](/public/imported_attachments/1/no matches.JPG)
![no matches.JPG_thumb](/public/imported_attachments/1/no matches.JPG_thumb)

bmeeks

@Raffi.:

I just tried the same on my WAN. The grep search still found the sid in the file even after creating the new single entry in disablesid2.conf file. The odd thing is that the sid log show 1 found but no rules that match. Maybe that's a clue? Why no matches when the grep is showing the rule is there?

/usr/local/etc/suricata/rules: grep -r "2018959" /usr/local/etc/suricata/rules
/usr/local/etc/suricata/rules/emerging-policy.rules:alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET POLICY PE EXE or DLL Windows file download HTTP"; flow:established,to_client; flowbits:isnotset,ET.http.binary; flowbits:isnotset,ET.INFO.WindowsUpdate; file_data; content:"MZ"; within:2; byte_jump:4,58,relative,little; content:"PE|00 00|"; distance:-64; within:4; flowbits:set,ET.http.binary; reference:url,doc.emergingthreats.net/bin/view/Main/2000419; classtype:policy-violation; sid:2018959; rev:3; metadata:created_at 2014_08_19, updated_at 2017_02_01;)

Raffi

You are looking in the wrong place for the active rules. Your path of /usr/etc/local/suricata/rules is the main repository location for all the downloaded rules. It is the big bucket where rules are picked from to include in the suricata.rules file for an interface. That is not the actual list of "in use" rules for an interface. You will find those in a file in a sub-directory underneath such as /usr/local/etc/suricata/suricata_xxxxx/rules. The "xxxxx" part will be a random GUID and the physical interface name.

For example, on my test machine the path to the active rules for the LAN is:

/usr/local/etc/suricata/suricata_59991_em1/rules/suricata.rules

Bill

Raffi_

Thanks, I wasn't sure if I had the right path. Here is the new result.

[2.4.3-RELEASE][admin@pfsense.telebyte.localdomain]/usr/local/etc/suricata/suricata_37173_em0/rules: grep -r "2018959" /usr/local/etc/suricata/suricata_37173_em0/rules/
/usr/local/etc/suricata/suricata_37173_em0/rules/flowbit-required.rules:# Category: emerging-policy GID:1 SID:2018959
/usr/local/etc/suricata/suricata_37173_em0/rules/flowbit-required.rules:alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET POLICY PE EXE or DLL Windows file download HTTP"; flow:established,to_client; flowbits:isnotset,ET.http.binary; flowbits:isnotset,ET.INFO.WindowsUpdate; file_data; content:"MZ"; within:2; byte_jump:4,58,relative,little; content:"PE|00 00|"; distance:-64; within:4; flowbits:set,ET.http.binary; reference:url,doc.emergingthreats.net/bin/view/Main/2000419; classtype:policy-violation; sid:2018959; rev:3; metadata:created_at 2014_08_19, updated_at 2017_02_01;)
[2.4.3-RELEASE][admin@pfsense.telebyte.localdomain]/usr/local/etc/suricata/suricata_37173_em0/rules:

Raffi

NollipfSense

It would be pleasant to be able to search for rules…some rule is blocking contact.ebay.com and it's not in emerging-policy.rules

bmeeks

@Raffi.:

Thanks, I wasn't sure if I had the right path. Here is the new result.

[2.4.3-RELEASE][admin@pfsense.telebyte.localdomain]/usr/local/etc/suricata/suricata_37173_em0/rules: grep -r "2018959" /usr/local/etc/suricata/suricata_37173_em0/rules/
/usr/local/etc/suricata/suricata_37173_em0/rules/flowbit-required.rules:# Category: emerging-policy GID:1 SID:2018959
/usr/local/etc/suricata/suricata_37173_em0/rules/flowbit-required.rules:alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET POLICY PE EXE or DLL Windows file download HTTP"; flow:established,to_client; flowbits:isnotset,ET.http.binary; flowbits:isnotset,ET.INFO.WindowsUpdate; file_data; content:"MZ"; within:2; byte_jump:4,58,relative,little; content:"PE|00 00|"; distance:-64; within:4; flowbits:set,ET.http.binary; reference:url,doc.emergingthreats.net/bin/view/Main/2000419; classtype:policy-violation; sid:2018959; rev:3; metadata:created_at 2014_08_19, updated_at 2017_02_01;)
[2.4.3-RELEASE][admin@pfsense.telebyte.localdomain]/usr/local/etc/suricata/suricata_37173_em0/rules:

Raffi

The rule (2018959) has a flowbit parameter (that flowbits:set,ET.http.binary string in the rule). Since that rule "sets" a flowbit that is later checked by a different rule that has a matching "isset" flowbit keyword, then the automatic flowbit rule resolution process is enabling that rule for you automatically. You have two options. First, you can simply suppress that 2018959 rule to prevent the alert (and block). It would still set the flowbit when needed that some other rules depend on. Option 2 is to disable that rule. But if you do that, then it will not set that dependent flowbit and the other rules with the "isset" check on that flowbit will fail to trigger because the dependent flowbit is not set.

Bill

Raffi_

Thanks for the explanation Bill. You learn something new everyday. I have that rule suppressed and seems to be fine.

Is this normal behavior? If so, how come it didn't come up as a flowbit rule when you did the grep? Does it depend on rules you have active?
It looks like the disablesid.conf can't be 100% relied on for disabling rules. This behavior was very confusing to me and may be to others as well. Is there anyway that the sid management could automatically suppress a rule that it can't disable due to flowbits? If that's not possible maybe having the sid mgmt log display a message stating that it was not able to disable the rule due to flowbits. This would give the user some insight as to what's going on. I thought it was a bug, but it does not seem to be.

Raffi

bmeeks

@Raffi.:

Thanks for the explanation Bill. You learn something new everyday. I have that rule suppressed and seems to be fine.

Is this normal behavior? If so, how come it didn't come up as a flowbit rule when you did the grep? Does it depend on rules you have active?
It looks like the disablesid.conf can't be 100% relied on for disabling rules. This behavior was very confusing to me and may be to others as well. Is there anyway that the sid management could automatically suppress a rule that it can't disable due to flowbits? If that's not possible maybe having the sid mgmt log display a message stating that it was not able to disable the rule due to flowbits. This would give the user some insight as to what's going on. I thought it was a bug, but it does not seem to be.

Raffi

Yes, the total of your enabled rules influences what flowbit rules may be required. You can always see the auto-enabled flowbit rules on the RULES tab by selecting that category to view in the drop-down selector.

Every IDS/IPS user needs to go search Google for flowbits to learn what they are and how they work. That will help you understand why the flowbits process works the way it does in the GUI.

Anytime you have a rule triggering that you think should be disabled, look first in the auto-flowbits set to see if the rule is in there. If it is and you don't want to see alerts from it, then suppress that SID.

I've used this crude example in the past to describe what flowbits are all about. Flowbits are mechanisms for rules to store "states" in the processing of packet information. Flowbits are TCP session-based. That means they stay set for the duration of a TCP session. Suppose a certain pattern of bytes in a PDF file indicates that file contains a malicious payload. However, that pattern of bytes is only malicious when it is part of a PDF. The same pattern of bytes in a GIF, for instance, may be totally harmless. So if I just made a rule that said "anytime I see this pattern of bytes I'm going to alert", then I would experience false positives from my rule any time some random sequence of bytes happens to match my pattern in something like an MP3 music file, a GIF graphic, etc. I would also match on that pattern in a PDF file (which is malicious in my example).

So it would be really nice if the detection engine could know when a PDF file was being downloaded and only then would it enable my rule to look for that malicious byte pattern. For any other type of traffic, my rule would not trigger even if the same malicious byte pattern were present. Flowbits let this happen. When a PDF file download session starts there are some indicators a rule can match on that signal a PDF file is downloading. So when those indicators are seen in a stream a flowbit for "PDF file download" is set. My rule that is looking for the malicious PDF payload pattern will have an option set to check that "PDF file download" flowbit by using the "isset" keyword. Only when the flowbit is set (i.e., isset = TRUE) will my rule fire. If the flowbit is not set, the rule does not fire and thus won't false positive needlessly.

In the actual rule giving you trouble, the flowbit name was "ET.http.binary". The name is totally arbitrary and is selected by the rule author. The only important thing about the name is that later, when checking if the flowbit is set, you must use the exact same name. So another rule would have the action keyword "flowbits:isset,ET.http.binary". Only when that condition is TRUE (meaning a flowbit with that name is "set") will the rule trigger an alert.

Bill

Raffi_

Awesome. Thanks for the education. I have a feeling I won't find that explanation on google. It makes prefect sense without getting too technical. You should be writing books on this stuff if you aren't already. I will do some more reading up on flowbits.

Raffi

bmeeks

@Raffi.:

Awesome. Thanks for the education. I have a feeling I won't find that explanation on google. It makes prefect sense without getting too technical. You should be writing books on this stuff if you aren't already. I will do some more reading up on flowbits.

Raffi

Thanks! Writing a book is not my thing, though… :).

So back to the rule that was giving you trouble, that flowbit flag literally means it is from an Emerging Threats rule (the "ET" prefix) and then the rest of the tag means "an HTTP binary download is in progress". You can infer all of that from the name the rule author used. There are no hard and fast rules for flowbit names, though. As I mentioned, the only requirement is that every test for "isset" obviously can only be TRUE when you are checking for an exact name. The rule author could have just as easily named his flowbit "Joes.binary.download" or whatever tickled his fancy. Luckily he chose a name that helps us discern what the rule really is for. So rule 2018959 has a critical job of setting a flowbit to signal a binary download over HTTP is happening in the current session. Subsequent rules can now test that flowbit to see if a binary download over HTTP is happening, and if it is, their content may match and "fire" an alert.

Let's look at that 2018959 rule in a bit more detail to see what it is really up to:


alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET POLICY PE EXE or DLL Windows file download HTTP"; flow:established,to_client; flowbits:isnotset,ET.http.binary; flowbits:isnotset,ET.INFO.WindowsUpdate; file_data; content:"MZ"; within:2; byte_jump:4,58,relative,little; content:"PE|00 00|"; distance:-64; within:4; flowbits:set,ET.http.binary; reference:url,doc.emergingthreats.net/bin/view/Main/2000419; classtype:policy-violation; sid:2018959; rev:3; metadata:created_at 2014_08_19, updated_at 2017_02_01;)

Look at the "flowbits" tags in the rule. Basically this rule is setting a flag for other rules to look at, but it is checking some things first. The first flowbits check is to test that the "ET.http.binary" flag is not already set. If it is, then we don't need to set it again so the 2018959 rule won't "fire" if that flowbit is already set. Next a check is made to see if the flowbits for "ET.INFO.WindowsUpdate" is set. If it is, then rule 2018959 won't fire (see, it is testing for "isnotset" on that flowbit, so if that flowbit is actually set it inhibits this rule from firing). The idea is that a DLL or binary coming from Windows Update is OK. Other binaries or DLLs over HTTP may not be OK, so if the first two conditions are true (the ET.http.binary flag is not set and the ET.INFO.WindowsUpdate flag is not set) and the content of the packet indicates a binary transfer is happening, the rule fires by setting the flowbit for "ET.http.binary".

One thing that Snort Subscriber rules do that Emerging Threats does not (at least not as often) is also use the "flowbits:noalert" tag in their rules that set flowbits. When the "flowbits:noalert" tag is present, the rule will not generate an alert. So the text of rule 2018959 would be better in my opinion if it contained this set of flowbit actions:

flowbits:set,ET.http.binary;flowbits:noalert;

This would still set the "ET.http.binary" flowbit, but it would also automatically suppress the alert from the rule firing. Pretty much every Snort Subscriber rule that sets a flowbit includes the "flowbits:noalert" tag as well. I don't see that as often in the Emerging Threats rules.

Bill

Raffi_

Cool, I think I actually understand it. The rule only fires if all the flowbit checks are true? isnoset is basically an inverse check. If the flag is not set, then it's a true statement and it goes onto the next flowbit check if there is one.

Raffi

bmeeks

@Raffi.:

Cool, I think I actually understand it. The rule only fires if all the flowbit checks are true? isnoset is basically an inverse check. If the flag is not set, then it's a true statement and it goes onto the next flowbit check if there is one.

Raffi

Yes, and there are many different combinations of flowbits checks. There are keywords for "set", "isset", "isnotset" and "noalert". So flowbits are a mechanism that allow a number of rules to cooperate with each other over time as a data transfer occurs during a given session by setting flags other rules can check.

One key takeaway, though, is that any rule that has a "flowbits:isset" or "flowbits:isnotset" check in it can never fire if the rule that sets that flowbit is disabled or not loaded. That's why the auto-flowbits feature I put in the GUI is important. It scans all of your enabled rules to find any and all rules that contain "flowbits:isset" or "flowbits:isnotset" checks in them, and then it makes sure that at least one "flowbits:set" rule is enabled for each flowbit name it finds in your enabled rules. So that is how SID 2018959 got "enabled behind your back" so to speak. It was the auto-flowbits logic in the GUI code. It found at least one enabled rule in your setup that was testing the "ET.http.binary" flowbit, but it did not find any enabled rule that was setting that same flowbit. So it went looking through your rules until it found one that set that flowbit and then enabled that rule. It was helping you not shoot yourself in the foot by disabling a rule that other enabled rules depended on. If rule 2018959 had remained disabled, then that "ET.http.binary" flowbit would never get set and thus any of your other enabled rules that needed to see that flowbit set in order to fire would never see it and thus never fire. You would have disabled other rules by killing their flowbit dependency and would not have even known it.

P.S. – oh, and for others who may have followed this thread; all of this description of flowbits is true for Snort as well. In fact, the whole idea of flowbits began in Snort and was later incorporated into Suricata by the Suricata developers.

Bill

NRgia

Nice tutorial so to speak, maybe you could do a sticky post, in order for others to find it more easily in the future?

Where to read more about rules tips & tricks ? "How to create Snort rules documentation", on their site is ok?

bmeeks

@NRgia:

Nice tutorial so to speak, maybe you could do a sticky post, in order for others to find it more easily in the future?

Where to read more about rules tips & tricks ? "How to create Snort rules documentation", on their site is ok?

There is some useful documentation on the Snort.org site. However, to be honest, I've never found a great all-in-one location for this kind of information. Bits and pieces are scattered all over. As with lots of software, especially open-source and other "free" software, the developers spend more time on coding and adding features than creating documentation. I am guilty of that as well with the Snort and Suricata packages.

Bill