Suricata true inline IPS mode coming with pfSense 2.3 – here is a preview
-
Because I don't know relationship between categories and SID Mgmt, so I made a test. If I enable emerging-scan.rules from categories, use "Disable SID File" for A sid, then use "Drop SID File" for A sid, I found alerts dropped include A sid, so it means "Drop SID File" override categories and other sid file setting, right?
If I add emerging-scan to "Drop SID File", all emerging-scan traffic (include default disable from categories) dropped. That's right?
Thanks!
The behavior you document is the result of errors in the way I implemented the logic. A fix is coming. See the full explanation below.
When I looked into the new "Drop SID" stuff I realized that in the rush to get it completed I had not thoroughly thought through all the logic. I found several issues. I have them fixed in the patch I'm working on now. Should be posting it in a day or two at the latest. I have one more thing with XMLRPC Sync I may try and sneak in. If that fix is going to take too long, then I will post what I've fixed so far.
Here is how the "Drop SID" logic will work in the corrected version coming out soon. Understanding the sequence of events for the whole SID management piece will help. I also hope describing the logic here will help others check my thought process.
1. First the two enablesid and disablesid files are processed. The order in which these are processed (in other words, which is run first) is set on the SID MGMT tab in a drop-down for the interface. These two files choose the rules that will be used. There are a number of ways to include rules in the enforcing set: (a) by giving category names, (b) by using SID numbers or ranges and
using PCRE to select matching rules.2. Next the modifysid and dropsid files are processed. They are fed the collection of rules selected in the first step by the enablesid and disablesid files. The modifysid file will alter the actual contents of the rule using PCRE. The dropsid file will change the rule action from whatever it is to DROP for rules that the terms in the dropsid file match in the enabled rules list. Neither the modifysid nor the dropsid files will change the state (enabled or disabled) for any rules! This is key to remember. All dropsid selectors will do is change the rule action keyword. It won't "turn on" nor "turn off" the rule.
All of the activity above happens on an in-memory array that is constructed as the configuration for an interface is being generated. At the end of the process, all enabled rules with any modifysid and dropsid modifications are written to a file called suricata.rules in the application sub-directory for the specific interface. That suricata.rules file, along with the custom.rules and flowbit-required.rules files, contains the active rule set to be used for the interface. Only enabled rules will be in those files. It makes no sense to include disabled rules there. They would just take up space for nothing (they are commented out, or disabled, anyway).
The new patch adds some additional functionality on the RULES tab to help you see what actually is in the enforcing rules. When a rule has been selected and is managed by a SID MGMT file (meaning the SID matched one of the selectors in the SID MGMT conf files), up to two icons will be shown beside the rule. If the rule was forced ON or forced OFF by SID MGMT, then an icon is shown for that state modification. If the rule's content or action keyword has been changed by SID MGMT (via modifysid or dropsid), then a second icon is shown to indicate this modification. Clicking on the SID number displayed or double-clicking on the row displaying the rule results in a Bootstrap Modal dialog opening where the modified contents of the rule are displayed. This lets you see the actual impact, in the rule, of your SID MGMT modifications.
On the CATEGORIES tab, things are pretty much like they have always been. There, one of two different icons is displayed by a Category Name when that category was either "turned on" or "turned off" by selectors in the enablesid or disablesid files on the SID MGMT tab.
Bill
-
@Giuanìn:
awesome work bmeeks. i think this is a very important enhancement for the overall pfsense project.
i think that it would be nice if it were possible to configure the action alert/drop directly in the "Rules" section.I agree that would be a nice enhancement. Let me think about how best to implement it. This change probably won't make it into the next bug fix update because I want to get that one out quickly. Thank you for the suggestion!
Bill
-
Bill,
I appreciate your detail explanation. Thanks for your help :)
-
@Giuanìn:
awesome work bmeeks. i think this is a very important enhancement for the overall pfsense project.
i think that it would be nice if it were possible to configure the action alert/drop directly in the "Rules" section.+10000
-
I have posted the pull request containing the latest bug fixes. After review by the pfSense developers, it will be merged and then show up as an update on the System > Packages > Installed Packages tab for pfSense 2.3-BETA users.
Here is a link to the pull request with a list of the fixes: https://github.com/pfsense/FreeBSD-ports/pull/84
Bill
-
Hi Bill,
I test 3.0_1, found below.
1. Enable emerging-scan.rules from categories, and disable sid A from SID Mgmt, It not show "Auto-disabled by SID Mgmt" from rules.
2. Add emerging-scan to enablesid.conf, Categories show emerging-scada.rules and emerging-scan.rules. But Add emerging-scan.rules, It correct.
3.Add emerging-scan.rules to disable.conf and enablesid.conf, Whether I use "disable,enable" or "enable,disable" to State Order, It show "Category is auto-disabled by SID Mgmt conf files".
4.IP Lists, It missing -> http://emergingthreats.net/products/iqrisk-rep-list/
5.Auto-enable rules required for checked flowbits, Default is Checked, but not checked.
Could you check it please?
-
Hi Bill,
I test 3.0_1, found below.
1. Enable emerging-scan.rules from categories, and disable sid A from SID Mgmt, It not show "Auto-disabled by SID Mgmt" from rules.
2. Add emerging-scan to enablesid.conf, Categories show emerging-scada.rules and emerging-scan.rules. But Add emerging-scan.rules, It correct.
3.Add emerging-scan.rules to disable.conf and enablesid.conf, Whether I use "disable,enable" or "enable,disable" to State Order, It show "Category is auto-disabled by SID Mgmt conf files".
4.IP Lists, It missing -> http://emergingthreats.net/products/iqrisk-rep-list/
5.Auto-enable rules required for checked flowbits, Default is Checked, but not checked.
Could you check it please?
I am looking into this. I did some testing using another set of conditions over the weekend and it seemed to work. This morning, however, I was able to reproduce your issue.
UPDATE: I found the problem with auto-management icons. It is a cosmetic display issue only, so the rules are actually being managed as defined in the SID MGMT files. I will post the fix for the display problem soon. Working on a couple of other user-reported issues as well. The IQRisk download problem is also identified and will be fixed. That one was a failure to declare a variable as global when used in a function call.
Bill
-
Look for an update soon that addresses the user-reported bugs above. The pull request is here: https://github.com/pfsense/FreeBSD-ports/pull/85.
Bill
-
I use SID State Order(disable,enable), add emerging-scan.rules to disablesid.conf and add sid A to enablesid.conf, I didn't see any enable rules. Sid A is also not in XXX/rules/suricata.rules :)
I use pcre:xxxxx, It can be selected, but use pcre:"xxxxx", It can't.
By the way, Is it possiable for drop all match rules traffic? Do you have any suggestions?
Thanks!
-
Latest Suricata bug fix is now available for download. Here is what the pfSense-pkg-suricata 3.0_2 package update addresses.
Bug Fixes
-
Rules auto-managed by SID MGMT tab files are not tagged correctly on the RULES tab.
-
IQRisk IP Reputation files not downloading for users with subscription code.
-
Icons indicating rule GID:SID added to Supress Lists get duplicated when IPs are the same on ALERTS tab.
-
Snort VRT rules checkboxes not auto-disabled when IPS-Policy is selected on CATEGORIES tab.
-
PCRE selection of SIDs not working correctly for auto-SID management.
Known Limitations:
At the moment, if you try select a Snort VRT IPS Policy and try to view all the rules selected by the policy on the RULES tab, you will crash the PHP process for the Suricata GUI and get a blank browser screen. This happens because the large IPS Policy rule set for the "Balanced" or "Security" policies exhausts the maximum PHP memory pool allowed by current pfSense settings. A solution for that is being looked into by the pfSense developers. In the interim, do not attempt to view an IPS Policy rule set on the RULES tab. Note that although doing so will crash the PHP process showing you the RULES tab page, it won't impact the firewall operation and will not crash the Suricata binary. It just crashes the individual process that was attempting to display the rules.Bill
-
-
I use SID State Order(disable,enable), add emerging-scan.rules to disablesid.conf and add sid A to enablesid.conf, I didn't see any enable rules. Sid A is also not in XXX/rules/suricata.rules :)
This is by design. The logic excludes based on the largest net. So if you disable a category by putting it in the disablesid.conf file, then none of that category's rules can be used. Enabling individual SIDs from an excluded category in the enablesid.conf file is not possible (because the whole category has been excluded). It would be better to use SID ranges from the category in the two files and work out a solution that way. The biggest "excluding" statement wins between the two files (enablesid and disablesid). What the "order" drop-down controls is which conf file runs first and which last. So if you have the order set to "enable/disable" and you enable a category in enableside.conf and you disable the same category in disablesid.conf, then the category's rules are not present in the enforcing rules array at all because the entire category has been disabled. On the other hand, if you leave the conf files the same but switch the "order" drop-down to "disable/enable", then the category will be enabled and its rules present in the enforcing rules array because the enablesid.conf file ran last and enabled the category even though the disablesid.conf had previously disabled it.
I use pcre:xxxxx, It can be selected, but use pcre:"xxxxx", It can't.
Yeah, that was a typo by me in the example. You do not need the quote marks. Just use the format pcre:xxxx. There was also an unneeded called in that part of the code to the preg_quote() function. That caused the PCRE compare to fail as well. This is fixed in the update just posted this morning.
By the way, Is it possiable for drop all match rules traffic? Do you have any suggestions?
Not sure I understand what you are asking me here. If you just want all triggered rules to cause traffic to be blocked, you will need to switch back to legacy mode on the interface. That will let a few packets leak through, though, as I explained in previous posts. There is no way without rewriting the Suricata binary to have it "drop" all alerts the way legacy mode works.
Bill
-
How to evaluate performance and detect packet loss for pfSense 2.3 in inline IPS mode? stats.log or?
Thanks again!
-
How to evaluate performance and detect packet loss for pfSense 2.3 in inline IPS mode? stats.log or?
Thanks again!
I think that log primarily shows how much time was spent analyzing particular rules. But I don't remember for sure. It's been a long time since I activated the stats.log option and looked over the log.
Bill
-
hmm…
I can't see capture.kernel_drops in stats.log. :P
-
Hi Bill,
I can't start Barnyard2, I use Syslog Output.
It's seem none barnyard2 installed.
Can you check?
Thx!
-
Hi Bill,
I can't start Barnyard2, I use Syslog Output.
It's seem none barnyard2 installed.
Can you check?
Thx!
Yep, it looks like the Barnyard2 binary is not there. I will need to investigate how it got missed in the build/run dependencies. It should have been installed along with Suricata. In the meantime, you should be able to just install it using pkg from the command line.
Bill
-
A quick fix for the missing Barnyard2 binary was posted today by the pfSense team. If you remove and then reinstall Suricata, it should bring along Barnyard2 now like it did in the past.
Bill
-
There has been nothing new on this project in many months. Is it dead? I certainly hope not, but I assume we are waiting on improvements to netmap which hasn't been updated in quite a while. Suricata 3.1.1 is out now and seems to be in current development.
I could really use this package as I am sure many others are chomping at the bit for it.
Progress preview maybe?
Dan
-
I guess this really is a dead project after all. It's a real shame.
-
This desperately needs to happen… I need Inline mode so bad, I can't describe how badly I need it. We have so much junk traffic tossed at valid IPs that perfectly good sites get blocked and many web/cloud based tools that my faculty and staff depend on become useless.
I've tuned Suricata rules until I can't see straight, and still, valid sites get blocked.
Come on devs, roll this stuff out! We are all rooting for you (and whining a bit).
Suricata 3.1.1 has been out for a while now in production, I wonder if there are still underlying netmap/driver issues causing problems with Inline mode?
