Suricata true inline IPS mode coming with pfSense 2.3 – here is a preview

bmeeks

Suricata Users:

Rejoice! True high-speed inline mode IPS is coming with pfSense 2.3 and its new Netmap support. I have a working Suricata package and in the next few posts will show some screen shots of the new IPS mode in action and how to set it up.

First some important disclaimers!
#1 – No, this is not and will not be available for Suricata 2.0.x on pfSense 2.2.x. The inline IPS mode will only be available on pfSense 2.3 and above with the Suricata 3.0 and above binary. Sorry, but the needed Netmap support just isn't there in the older versions. So plan on upgrading if you want true inline IPS mode with Suricata.

#2 – No, this is not currently available for Snort. It might be in the future, but that is currently uncertain. Snort's DAQ module is not as Netmap friendly as Suricata is. If this is ever available for Snort, it would also only be on the pfSense 2.3 and higher platform.

#3 – Not all network cards support Netmap. Many of the popular ones do, but not all. Someone posted a list of the FreeBSD drivers that currently support Netmap over in the pfSense 2.3-BETA forum. I suspect support will get more common, but for now just realize not all cards will work. I have been using the em driver in my virtual machines and it works fine. If your card does not support the new inline mode, just use it in the Legacy Mode (described below) until such time as your card is either supported or you decide to switch to one that is.

What is Inline IPS?
This basically refers to a mode of operation whereby the rule inspection engine (Suricata in our case) sits between the network interface card and the kernel's network stack. All traffic coming from the network card to the kernel, and all traffic going to the network card from the kernel, passes through the rule inspection engine. There is no bypass and no packet copying. If a Suricata rule configured with the DROP keyword fires on a packet, that packet is not passed on and is in effect "dropped". Now here is another key piece of information. When running in inline IPS mode, Suricata sits between the network interface card and the pf packet filter (the firewall). That means Suricata will see, inspect, and either pass or drop inbound traffic from the interface to the firewall BEFORE any firewall rules are applied. However, in the opposite direction (from the firewall packet filter outbound to the NIC) Suricata will see the traffic BEFORE the NIC does and AFTER the firewall rules have been applied. Suricata can still pass or drop the outbound traffic, though.

How is This Different Than Today and Why is It Better?
Today, Suricata works in conjunction with the packet filter (pf) engine in the firewall to block offending traffic. However, Suricata does not sit inline between the NIC and the firewall. Instead, the PCAP library is used to make copies of the packets as they move from the network card to the kernel and packet filter. Suricata examines those copies while the original packets continue on to the packet filter. The firewall may drop them for not matching rules, but otherwise those packets are allowed through. Suricata generally needs to examine several packets in order to determine if traffic is malicious or not. While it is doing that with those copies of the packets, remember that the original packets are still getting through. They have even established a "state" in the firewall's state table and will continue flow even if later Suricata wants to block them. We call these early packets that get through "packet leakage". If Suricata determines the traffic is malicious (matches one or more rules), then it inserts the offending IP address into the FreeBSD packet filter table called snort2c. This is a custom table created by pfSense at boot up and is located very early in the firewall rule chain. If you have enabled the "kill states" option for Suricata, then it will also clear the state table entries for the IPs it blocks in order to terminate any flows (sessions) that may have been established by those early packets that got through while Suricata was analyzing the copies. With inline mode, there is no packet leakage. This greatly enhances security.

Are There Any Key Changes in How I Configure Suricata for IPS Mode?
Yes, there a few key differences when operating in inline IPS mode versus what we are calling now the old Legacy Mode. Legacy Mode is the process described above where PCAP is used to copy packets and offending IP addresses are inserted into the packet filter's snort2c table. In the new Suricata package, each interface is still treated independently. You can have one interface in the new inline IPS mode while another uses the Legacy Mode.

The first big change is that inline IPS mode will not initially make use of any Pass List. Support for that does not exist in the same way that Legacy Mode has it. For inline IPS mode you will need to make bigger use of Suppress Lists and IP Reputation Lists (IP rules with the PASS action can take the place of the old Legacy Mode Pass Lists).

Another difference is where you go look for blocks. They are not shown on the BLOCKS tab. This is because in Netmap IPS mode Suricata does not log things the same way. Also, what the BLOCKS tab in Legacy Mode was really showing you, were the IP addresses Suricata had inserted into the snort2c table. The code on the page literally dumped the IP addresses from that table and displayed them on the tab. Since inline IPS mode dispenses with use of the snort2c table, there are no IP addresses to dump out and display. With inline IPS mode, you see dropped traffic on the ALERTS tab. There it will be highlighted to indicate it was dropped. I will include some screenshots in a subsequent post.

Perhaps the biggest sea change with inline IPS mode is how you must configure your rules. Everyone has probably grown accustomed to the Legacy Mode process where any rule generating an alert produced a block when blocking was enabled. With IPS mode, that changes significantly! Only rules that have the action word DROP will actually cause traffic to be dropped (or blocked if you still prefer that term). Rules with ALERT as the action word will do only that (that is, produce an alert without a subsequent block or drop). Most all of the popular rule vendors (Snort VRT and Emerging Threats, as two examples) ship their rule sets with all the rules configured for ALERT. You must manually edit the rules to change them to DROP. To help make this easier for those who want to use the new IPS mode, I modified the SID MGMT tab to also take a dropsid.conf file. You can put the same kinds of automatic rule selection criteria in there as you can in the other SID management files. Any rules "selected" by the criteria in dropsid.conf will automatically get changed to DROP. I will include an example in subsequent posts.

If you made it this far and have not fallen asleep, then read on to the next few posts for some screenshots of the new inline IPS mode for Suricata.

Bill

bmeeks

Here is a screenshot showing how simple it is to enable inline IPS mode. This is from the INTERFACE SETTINGS tab.

Simply enable the Block Offenders option and then choose Inline Mode. Note that Kill States and Which IP to Block are not shown when Inline Mode is selected. This is because they are irrelevant in that mode.

Bill

bmeeks

After you enable Inline Mode, go to the SID MGMT tab and create a dropsid.conf file to select and change key rules to DROP. You could change all rules, but I would not recommend that. Instead be selective. There are several options, and hopefully more experienced IPS users will post their suggestions and examples once the new package is released.

Here is an example dropsid.conf file I used to change all the Emerging-Scan category rules from ALERT to DROP. (Hint: putting the rules selector in the dropsid.conf file is what causes the change from ALERT to DROP).


# Category DROPS
emerging-scan

#  PCRE IPS Policy DROPS  |
# -----------------
pcre:pcre:security-ips\s*drop

I'm not currently using that second PCRE match because I don't have the Snort VRT rules running in my test virtual machine. If I did, that PCRE would cause all the IPS POLICY - SECURITY rules tagged with the metadata property "drop" to be selected and automatically changed from ALERT to DROP.

Here is a screen shot of the SID MGMT tab.

bmeeks

Here are some alerts (drops) I generated by using the dropsid.conf file described above on a test machine and then scanning the machine with nmap. The dropsid file altered all of the Emerging-Scan rules to be DROP instead of ALERT. The nmap scan I used caused hits on both the ET-Scan and ET-Policy rules. Only the ET-Scan rules were configured to drop. Those are shown in red by the new IPS mode. So the red rules resulted in blocked traffic while the other rules just produced alerts without blocking the traffic.

In the following screenshots the 192.168.10.12 address was my Kali Linux malicious scanner (running nmap scans) and 192.168.10.118 was the pfSense 2.3-BETA virtual machine I was scanning.

Bill

bmeeks

You can also filter the ALERTS tab and show only the alerts that resulted in packet drops. Here is a screenshot of the ALERTS tab filtering for just the DROPs.

Bill

bmeeks

The new Suricata package should be out soon. Remember, though, that it will only be available for pfSense 2.3 and newer. I welcome any feedback or suggestions for improvement.

Bill

fsansfil

Very good news. Cant wait !

+1

F.

bmeeks

The pull request has been posted for review and approval. Here is the link: https://github.com/pfsense/FreeBSD-ports/pull/76

Bill

ntct

Hi bmeeks,

About Logs View, I can view alerts.log,http.log and suricata.log, but others disappear. I have wrong setting?

bmeeks

@ntct:

Hi bmeeks,

About Logs View, I can view alerts.log,http.log and suricata.log, but others disappear. I have wrong setting?

You can only view logs which have been specifically enabled on the INTERFACE SETTINGS tab for the interface. The alerts.log and http.log are enabled by default unless the user unchecks them on that page. The suricata.log is the "system log" for the Suricata binary, so it is always populated.

To see data in the other choices on the LOGS VIEW tab you must enable the log on the INTERFACE SETTINGS tab. Most are disabled by default because they generate a lot of text and can really chew up disk space on busy networks.

Bill

ntct

Hi,

I enable SID File, add emerging-scan, But interface categories show emerging-shellcode.rules (auto-enabled by SID Mgmt conf files).

bmeeks

@ntct:

Hi,

I enable SID File, add emerging-scan, But interface categories show emerging-shellcode.rules (auto-enabled by SID Mgmt conf files).

I will check on this. I am working on a bug fix release now. There are still some GUI issues left over from the Bootstrap conversion.

Update: I checked and found that I got this part of the code wrong with the new dropsid functionality. Working on the fix and will post an updated package soon.

Bill

ntct

Because I don't know relationship between categories and SID Mgmt, so I made a test. If I enable emerging-scan.rules from categories, use "Disable SID File" for A sid, then use "Drop SID File" for A sid, I found alerts dropped include A sid, so it means "Drop SID File" override categories and other sid file setting, right?

If I add emerging-scan to "Drop SID File", all emerging-scan traffic (include default disable from categories) dropped. That's right?

Thanks!

Ryu

awesome work bmeeks. i think this is a very important enhancement for the overall pfsense project.
i think that it would be nice if it were possible to configure the action alert/drop directly in the "Rules" section.

bmeeks

@ntct:

Because I don't know relationship between categories and SID Mgmt, so I made a test. If I enable emerging-scan.rules from categories, use "Disable SID File" for A sid, then use "Drop SID File" for A sid, I found alerts dropped include A sid, so it means "Drop SID File" override categories and other sid file setting, right?

If I add emerging-scan to "Drop SID File", all emerging-scan traffic (include default disable from categories) dropped. That's right?

Thanks!

The behavior you document is the result of errors in the way I implemented the logic. A fix is coming. See the full explanation below.

When I looked into the new "Drop SID" stuff I realized that in the rush to get it completed I had not thoroughly thought through all the logic. I found several issues. I have them fixed in the patch I'm working on now. Should be posting it in a day or two at the latest. I have one more thing with XMLRPC Sync I may try and sneak in. If that fix is going to take too long, then I will post what I've fixed so far.

Here is how the "Drop SID" logic will work in the corrected version coming out soon. Understanding the sequence of events for the whole SID management piece will help. I also hope describing the logic here will help others check my thought process.

1. First the two enablesid and disablesid files are processed. The order in which these are processed (in other words, which is run first) is set on the SID MGMT tab in a drop-down for the interface. These two files choose the rules that will be used. There are a number of ways to include rules in the enforcing set: (a) by giving category names, (b) by using SID numbers or ranges and using PCRE to select matching rules.

2. Next the modifysid and dropsid files are processed. They are fed the collection of rules selected in the first step by the enablesid and disablesid files. The modifysid file will alter the actual contents of the rule using PCRE. The dropsid file will change the rule action from whatever it is to DROP for rules that the terms in the dropsid file match in the enabled rules list. Neither the modifysid nor the dropsid files will change the state (enabled or disabled) for any rules! This is key to remember. All dropsid selectors will do is change the rule action keyword. It won't "turn on" nor "turn off" the rule.

All of the activity above happens on an in-memory array that is constructed as the configuration for an interface is being generated. At the end of the process, all enabled rules with any modifysid and dropsid modifications are written to a file called suricata.rules in the application sub-directory for the specific interface. That suricata.rules file, along with the custom.rules and flowbit-required.rules files, contains the active rule set to be used for the interface. Only enabled rules will be in those files. It makes no sense to include disabled rules there. They would just take up space for nothing (they are commented out, or disabled, anyway).

The new patch adds some additional functionality on the RULES tab to help you see what actually is in the enforcing rules. When a rule has been selected and is managed by a SID MGMT file (meaning the SID matched one of the selectors in the SID MGMT conf files), up to two icons will be shown beside the rule. If the rule was forced ON or forced OFF by SID MGMT, then an icon is shown for that state modification. If the rule's content or action keyword has been changed by SID MGMT (via modifysid or dropsid), then a second icon is shown to indicate this modification. Clicking on the SID number displayed or double-clicking on the row displaying the rule results in a Bootstrap Modal dialog opening where the modified contents of the rule are displayed. This lets you see the actual impact, in the rule, of your SID MGMT modifications.

On the CATEGORIES tab, things are pretty much like they have always been. There, one of two different icons is displayed by a Category Name when that category was either "turned on" or "turned off" by selectors in the enablesid or disablesid files on the SID MGMT tab.

Bill

bmeeks

@Giuanìn:

awesome work bmeeks. i think this is a very important enhancement for the overall pfsense project.
i think that it would be nice if it were possible to configure the action alert/drop directly in the "Rules" section.

I agree that would be a nice enhancement. Let me think about how best to implement it. This change probably won't make it into the next bug fix update because I want to get that one out quickly. Thank you for the suggestion!

Bill

ntct

Bill,

I appreciate your detail explanation. Thanks for your help :)

pfcode

@Giuanìn:

awesome work bmeeks. i think this is a very important enhancement for the overall pfsense project.
i think that it would be nice if it were possible to configure the action alert/drop directly in the "Rules" section.

+10000

bmeeks

I have posted the pull request containing the latest bug fixes. After review by the pfSense developers, it will be merged and then show up as an update on the System > Packages > Installed Packages tab for pfSense 2.3-BETA users.

Here is a link to the pull request with a list of the fixes: https://github.com/pfsense/FreeBSD-ports/pull/84

Bill

ntct

Hi Bill,

I test 3.0_1, found below.

1. Enable emerging-scan.rules from categories, and disable sid A from SID Mgmt, It not show "Auto-disabled by SID Mgmt" from rules.

2. Add emerging-scan to enablesid.conf, Categories show emerging-scada.rules and emerging-scan.rules. But Add emerging-scan.rules, It correct.

3.Add emerging-scan.rules to disable.conf and enablesid.conf, Whether I use "disable,enable" or "enable,disable" to State Order, It show "Category is auto-disabled by SID Mgmt conf files".

4.IP Lists, It missing -> http://emergingthreats.net/products/iqrisk-rep-list/

5.Auto-enable rules required for checked flowbits, Default is Checked, but not checked.

Could you check it please?

bmeeks

@ntct:

Hi Bill,

I test 3.0_1, found below.

1. Enable emerging-scan.rules from categories, and disable sid A from SID Mgmt, It not show "Auto-disabled by SID Mgmt" from rules.

2. Add emerging-scan to enablesid.conf, Categories show emerging-scada.rules and emerging-scan.rules. But Add emerging-scan.rules, It correct.

3.Add emerging-scan.rules to disable.conf and enablesid.conf, Whether I use "disable,enable" or "enable,disable" to State Order, It show "Category is auto-disabled by SID Mgmt conf files".

4.IP Lists, It missing -> http://emergingthreats.net/products/iqrisk-rep-list/

5.Auto-enable rules required for checked flowbits, Default is Checked, but not checked.

Could you check it please?

I am looking into this. I did some testing using another set of conditions over the weekend and it seemed to work. This morning, however, I was able to reproduce your issue.

UPDATE: I found the problem with auto-management icons. It is a cosmetic display issue only, so the rules are actually being managed as defined in the SID MGMT files. I will post the fix for the display problem soon. Working on a couple of other user-reported issues as well. The IQRisk download problem is also identified and will be fixed. That one was a failure to declare a variable as global when used in a function call.

Bill

bmeeks

Look for an update soon that addresses the user-reported bugs above. The pull request is here: https://github.com/pfsense/FreeBSD-ports/pull/85.

Bill

ntct

I use SID State Order(disable,enable), add emerging-scan.rules to disablesid.conf and add sid A to enablesid.conf, I didn't see any enable rules. Sid A is also not in XXX/rules/suricata.rules :)

I use pcre:xxxxx, It can be selected, but use pcre:"xxxxx", It can't.

By the way, Is it possiable for drop all match rules traffic? Do you have any suggestions?

Thanks!

bmeeks

Latest Suricata bug fix is now available for download. Here is what the pfSense-pkg-suricata 3.0_2 package update addresses.

Bug Fixes

Rules auto-managed by SID MGMT tab files are not tagged correctly on the RULES tab.
IQRisk IP Reputation files not downloading for users with subscription code.
Icons indicating rule GID:SID added to Supress Lists get duplicated when IPs are the same on ALERTS tab.
Snort VRT rules checkboxes not auto-disabled when IPS-Policy is selected on CATEGORIES tab.
PCRE selection of SIDs not working correctly for auto-SID management.

Known Limitations:
At the moment, if you try select a Snort VRT IPS Policy and try to view all the rules selected by the policy on the RULES tab, you will crash the PHP process for the Suricata GUI and get a blank browser screen. This happens because the large IPS Policy rule set for the "Balanced" or "Security" policies exhausts the maximum PHP memory pool allowed by current pfSense settings. A solution for that is being looked into by the pfSense developers. In the interim, do not attempt to view an IPS Policy rule set on the RULES tab. Note that although doing so will crash the PHP process showing you the RULES tab page, it won't impact the firewall operation and will not crash the Suricata binary. It just crashes the individual process that was attempting to display the rules.

Bill

bmeeks

@ntct:

I use SID State Order(disable,enable), add emerging-scan.rules to disablesid.conf and add sid A to enablesid.conf, I didn't see any enable rules. Sid A is also not in XXX/rules/suricata.rules :)

This is by design. The logic excludes based on the largest net. So if you disable a category by putting it in the disablesid.conf file, then none of that category's rules can be used. Enabling individual SIDs from an excluded category in the enablesid.conf file is not possible (because the whole category has been excluded). It would be better to use SID ranges from the category in the two files and work out a solution that way. The biggest "excluding" statement wins between the two files (enablesid and disablesid). What the "order" drop-down controls is which conf file runs first and which last. So if you have the order set to "enable/disable" and you enable a category in enableside.conf and you disable the same category in disablesid.conf, then the category's rules are not present in the enforcing rules array at all because the entire category has been disabled. On the other hand, if you leave the conf files the same but switch the "order" drop-down to "disable/enable", then the category will be enabled and its rules present in the enforcing rules array because the enablesid.conf file ran last and enabled the category even though the disablesid.conf had previously disabled it.

@ntct:

I use pcre:xxxxx, It can be selected, but use pcre:"xxxxx", It can't.

Yeah, that was a typo by me in the example. You do not need the quote marks. Just use the format pcre:xxxx. There was also an unneeded called in that part of the code to the preg_quote() function. That caused the PCRE compare to fail as well. This is fixed in the update just posted this morning.

@ntct:

By the way, Is it possiable for drop all match rules traffic? Do you have any suggestions?

Not sure I understand what you are asking me here. If you just want all triggered rules to cause traffic to be blocked, you will need to switch back to legacy mode on the interface. That will let a few packets leak through, though, as I explained in previous posts. There is no way without rewriting the Suricata binary to have it "drop" all alerts the way legacy mode works.

Bill

ntct

How to evaluate performance and detect packet loss for pfSense 2.3 in inline IPS mode? stats.log or?

Thanks again!

bmeeks

@ntct:

How to evaluate performance and detect packet loss for pfSense 2.3 in inline IPS mode? stats.log or?

Thanks again!

I think that log primarily shows how much time was spent analyzing particular rules. But I don't remember for sure. It's been a long time since I activated the stats.log option and looked over the log.

Bill

ntct

hmm…

I can't see capture.kernel_drops in stats.log. :P

ntct

Hi Bill,

I can't start Barnyard2, I use Syslog Output.

It's seem none barnyard2 installed.

Can you check?

Thx!

bmeeks

@ntct:

Hi Bill,

I can't start Barnyard2, I use Syslog Output.

It's seem none barnyard2 installed.

Can you check?

Thx!

Yep, it looks like the Barnyard2 binary is not there. I will need to investigate how it got missed in the build/run dependencies. It should have been installed along with Suricata. In the meantime, you should be able to just install it using pkg from the command line.

Bill

bmeeks

A quick fix for the missing Barnyard2 binary was posted today by the pfSense team. If you remove and then reinstall Suricata, it should bring along Barnyard2 now like it did in the past.

Bill

dcol

There has been nothing new on this project in many months. Is it dead? I certainly hope not, but I assume we are waiting on improvements to netmap which hasn't been updated in quite a while. Suricata 3.1.1 is out now and seems to be in current development.

I could really use this package as I am sure many others are chomping at the bit for it.

Progress preview maybe?

Dan

dcol

I guess this really is a dead project after all. It's a real shame.

dhboyd26

This desperately needs to happen… I need Inline mode so bad, I can't describe how badly I need it. We have so much junk traffic tossed at valid IPs that perfectly good sites get blocked and many web/cloud based tools that my faculty and staff depend on become useless.

I've tuned Suricata rules until I can't see straight, and still, valid sites get blocked.

Come on devs, roll this stuff out! We are all rooting for you (and whining a bit).

Suricata 3.1.1 has been out for a while now in production, I wonder if there are still underlying netmap/driver issues causing problems with Inline mode?

repne

I installed suricata, and the installer complained about some mysql client vulnerability that will not be patched. Something to be worried about?

bmeeks

@<deleted>:</deleted>

Suricata package has been updated today from 3.0_7 to 3.0_8.

From the changelogs I see only a fix for "Suricata, a broken download should not wait forever." ,and some changes in licenses.

@bmeeks I don't understand, why not jumping to the latest version, with latest fixes, because they are alot ?

10x

I have been very busy with other work outside of my volunteer package maintainer duties for Suricata and Snort. The other work pays me, the volunteer maintainer duties do not … ;).

I am testing the latest 3.1.1 binary this weekend and hope to have a pull request posted very soon.

Bill

dcol

Will inline IDS be working with the latest Suricata update?

bmeeks

@dcol:

Will inline IDS be working with the latest Suricata update?

Hopefully better than it currently does. The issues are pretty much all netmap related as netmap is a relatively new technology. Suricata has had some upstream bugs reported around the netmap interface used for inline mode. A lot of those reported issues are fixed in the 3.1.1 release.

Bill

Wisiwyg

Thank you @BMeeks!

Looks like there's an updated Suricata in Package Manager with the latest 3.1.1_1 version. Trying it out now!

ntct

Does latest suricata 3.1.1_1 support hyperscan pattern match ?