Suricata 2.0.3 Package Preview
-
Been doing a lot of work the last few months readying the Suricata package for a major update to the latest version of the binary (2.0.3) and incorporating the new features available in the 2.0.x code tree into the Suricata GUI package for pfSense. Testing is almost completed, and I hope to post the Pull Request for review by the pfSense developers in the next couple of days.
In addition to the numerous bug fixes, here is a small preview of the new features coming to the Suricata package on pfSense. Because of the size of some of the screenshots, I will spread this across a number of sequential posts.
The first major upgrade is bumping the underlying Suricata binary to version 2.0.3. The GUI package will also get a new version number as it's bumped up to 2.0. The new INTERFACES tab is shown below.
-
A handy new feature is the ability to filter the alert log output on the ALERTS tab. You can filter on any or all of the available columns. In the example below, the source port (SPort) is filtered to only display alerts where the source port is 143.
Clicking the HIDE button will shrink the filter dialog up to a single line with a SHOW FILTER button displayed. Clicking that button will show the filter dialog.
-
Support has been added in the GUI for the new application layer parsers available in Suricata 2.0.x. In the screenshot below you see these new parsers and the settings available for them.
-
And what is likely to be the most popular new feature – full support for automatic SID management using the same configuration file syntax as PulledPork or Oinkmaster. You can now create text files that will be parsed on each rule update and when saving changes and rebuilding the rules in the GUI. The commands in the text configuration files will allow automatic enabling or disabling of individual rule SIDs, ranges of SIDs, matching of SIDs using PCRE (Perl-compatible regular expression) syntax, and several other specific options.
Here is a screenshot of the new SID MGMT tab showing how you can upload configuration files to the firewall and then assign them to your configured interfaces. Icons are available for uploading files from your local machine, creating and editing files directly in the GUI, or deleting files. There are also download icons that allow you to download any individual SID management file to your local machine. The icon in the upper right of the file management table allows you to download a gzip file containing all the SID management files currently stored on the firewall.
For each interface you can set a processing order (the SID State Order drop-down) for the interface configuration files. To enable automated SID management for an interface, simply select a desired file in the ENABLE, DISABLE or MODIFY drop-downs. If you want to disable the automated management feature for a given interface, simply set all three drop-downs to "none".
-
When the new automatic SID management feature is active for an interface, you can see the results of the applied SID configuration files on both the CATEGORIES tab and the RULES tab. New icons show which rule categories and which rules are matched by, and thus auto-managed by, tokens in the assigned configuration files.
The lighter icon indicates the category or rule is auto-disabled by a match from the "disablesid.conf" file assigned to the interface. The darker icon indicates the category or rule is auto-enabled by a match from the "enablesid.conf" file assigned to the interface. Rules denoted by the new auto-managed icons cannot be changed within the GUI. You must edit the appropriate SID management configuration file to make changes. During each rule rebuild, including the scheduled download updates of the ruleset files, a new sid_changes.log file is created summarizing the changes made automatically.
Example of enabling categories or SIDs
Here is a sample section from the "enablesid.conf" file used to produce the images below. The last line of the file will automatically enable the snort_server-apache rules category and thus auto-enable all of the "default enabled" rules from that file. They will be added to any manual files/rules selections made by the user in the GUI.# Example of modifying state for specific categories entirely (see README.CATEGORIES) # VRT-web-iis,ET-shellcode,ET-emergingthreats-smtp,Custom-shellcode,Custom-emergingthreats-smtp # Any of the above values can be on a single line or multiple lines, when # on a single line they simply need to be separated by a , # 1:9837,1:220-1:3264,3:13010-3:13013,pcre:MS(0[0-7])-\d+,MS09-008,cve:2009-0233 # The modifications in this file are for sample/example purposes only and # should not actively be used, you need to modify this file to fit your # environment. # ENABLED CATEGORIES snort_server-apacheHere is an example from the "disablesid.conf" file showing individual SIDs and a number of rule categories being auto-disabled.
Example of disabling categories and/or SIDs
#GPL SNMP public access UDP 1:2101411 # RULES WITH UNSUPPORTED KEYWORDS 1:16479, 1:21923, 1:29592, 1:30944 # # DISABLED SNORT VRT CATEGORIES browser-firefox.rules,browser-other.rules,chat.rules,content-replace.rules,os-other.rules,os-solaris.rules,policy-multimedia.rules,policy-other.rules,policy-social.rules,policy-spam.rules,server-mail.rules,server-oracle.rules,protocol-scada.rules # DISABLED ETPRO CATEGORIES chat.rules web_specific_apps.rules,web_server.rules,tor.rules,rbn.rules,rbn-malvertisers.rules,botcc.rules,ciarmy.rulesHere is a sample output of the sid_changes.log –
******************************************************** Starting auto RULE CATEGORY management for WAN Start Time: 2014-08-24 22:07:01 Processing enable_sid file: enablesid.conf Enabled rule category: snort_server-apache.rules Parsed 1 potential Rule Categories to match from the list of tokens. Enabled 1 matching Rule Categories. Processing disable_sid file: disablesid.conf Disabled rule category: snort_browser-firefox.rules Disabled rule category: snort_browser-other.rules Disabled rule category: snort_chat.rules Disabled rule category: emerging-chat.rules Disabled rule category: snort_content-replace.rules Disabled rule category: snort_os-other.rules Disabled rule category: snort_os-solaris.rules Disabled rule category: snort_policy-multimedia.rules Disabled rule category: snort_policy-other.rules Disabled rule category: snort_policy-social.rules Disabled rule category: snort_policy-spam.rules Disabled rule category: snort_server-mail.rules Disabled rule category: snort_server-oracle.rules Disabled rule category: snort_protocol-scada.rules Disabled rule category: emerging-web_specific_apps.rules Disabled rule category: emerging-web_server.rules Disabled rule category: snort_indicator-compromise.rules Disabled rule category: snort_indicator-obfuscation.rules Disabled rule category: snort_indicator-scan.rules Disabled rule category: snort_indicator-shellcode.rules Disabled rule category: emerging-tor.rules Disabled rule category: emerging-rbn-malvertisers.rules Disabled rule category: emerging-rbn.rules Disabled rule category: emerging-botcc.portgrouped.rules Disabled rule category: emerging-botcc.rules Disabled rule category: emerging-ciarmy.rules Parsed 21 potential Rule Categories to match from the list of tokens. Disabled 26 matching Rule Categories. End Time: 2014-08-24 22:07:01 ******************************************************** ******************************************************** Starting auto SID management for WAN Start Time: 2014-08-24 22:07:03 Processing enable_sid file: enablesid.conf Parsed 91 potential SIDs to match from the provided list of tokens. Found 69 matching SIDs in the active rules. Changed state for 69 SIDs to 'enabled'. Processing disable_sid file: disablesid.conf Parsed 431 potential SIDs to match from the provided list of tokens. Found 307 matching SIDs in the active rules. Changed state for 307 SIDs to 'disabled'. End Time: 2014-08-24 22:07:06 ********************************************************
-
One word Bill: FAAAAAAAAAAANTASTIC!!
-
One word Bill: FAAAAAAAAAAANTASTIC!!
Thanks! Also note that, except for the Suricata-specific application layer parsers, these same features will be coming to the Snort package. That means the ALERTS tab display filter and the ability to auto-manage SIDs with text configuration files.
Bill
-
Spent a good 10 minutes writing, deleting, re-writing and re-deleting this reply.
I'm speechless. Excellent work!
If only the pfsense devs dropped everything they do and approve the pull request immediately ;)
-
@jflsakfja:
If only the pfsense devs dropped everything they do and approve the pull request immediately ;)
Seriously? That's the behavior you desire?
While I think this is an interesting update, and I try to privately help Bill (ask him) get his changes I to pfSense quickly (as I do with any volunteer who does quality work), "drop everything" isn't the right path here.
-
Don't want to start a fight, but waiting a month for a pull request to be approved to a very important package (snort?) doesn't exactly put the fun in the package dev community. This particular update to this particular package contains a lot of necessary bugfixes, a LOT of requested functionality, and should be IMHO on the top of the pfsense's dev's todo list. Nobody will cry if pfsense is delayed a couple of weeks, but this package would make a lot of people cry (I personally can't see 4K hosts on the blocked tabs because I'm affected by a particular bug this fixes + I've spent a lot of time syncing configuration changes back and forth between systems (CARP sync is in this package).
And I haven't even touched on the conf files auto conf yet. Literally (and I'm not exaggerating a bit here) thousands of man hours will be saved by pfsense's users from this single feature.
My honest and humble opinion. Not an attack in any way, shape and/or form. Not trying to put down the amount of work the pfsense devs put into pfsense, I'm just saying that pfsense is actually extremely stable for the large majority of users out there. Since the majority wouldn't benefit from an upgrade to pfsense (other than bug fixes), that can hold off for a while.
EDIT:share?shape
-
Couldnt agree more.
@jflsakfja:
Don't want to start a fight, but waiting a month for a pull request to be approved to a very important package (snort?) doesn't exactly put the fun in the package dev community. This particular update to this particular package contains a lot of necessary bugfixes, a LOT of requested functionality, and should be IMHO on the top of the pfsense's dev's todo list. Nobody will cry if pfsense is delayed a couple of weeks, but this package would make a lot of people cry (I personally can't see 4K hosts on the blocked tabs because I'm affected by a particular bug this fixes + I've spent a lot of time syncing configuration changes back and forth between systems (CARP sync is in this package).
And I haven't even touched on the conf files auto conf yet. Literally (and I'm not exaggerating a bit here) thousands of man hours will be saved by pfsense's users from this single feature.
My honest and humble opinion. Not an attack in any way, share and/or form. Not trying to put down the amount of work the pfsense devs put into pfsense, I'm just saying that pfsense is actually extremely stable for the large majority of users out there. Since the majority wouldn't benefit from an upgrade to pfsense (other than bug fixes), that can hold off for a while.
-
Just for that, Brian, it's gone to the bottom of the pile.
-
Like your sense of humor :D
@gonzopancho:
Just for that, Brian, it's gone to the bottom of the pile.
-
Things are progressing well with the new package. I found a longstanding bug with IPv6 addresses in a PASS LIST this afternoon and fixed it. The problem was actually upstream in the Suricata binary and not within the pfSense GUI package. The bug is also present in 1.4.6. I've submitted the appropriate patch to the pfSense guys already so the binary packages for pfSense will be fixed, and will work on getting the patch into upstream as well.
I am getting support from the pfSense team even though they are terribly busy with their work on 2.2 and an upcoming security fix for 2.1.4, so I applaud them for that. Just please continue to be patient as final testing is completed and the pfSense developers jostle their work schedules to fit in the package review. There are a LOT of code changes in this update! I promise it's not vaporware… ;D
Bill
-
The new Suricata package update has been posted for the pfSense developers to review. You can follow the request here:
https://github.com/pfsense/pfsense-packages/pull/696
The notes at the top of the Pull Request show the changes.
One thing I suggest you do to get ready for the auto-SID management feature is to thoroughly review the documentation associated with the enablesid.conf, disablesid.conf and modifysid.conf files you can use with PulledPork or Oinkmaster. A little Google research should turn up several links and examples. Some kind users here on the forum may also be willing to share their files as examples. With the ability to upload these files to the firewall, it will also be possible for experienced users to share their SID management conf files with novices to help reduce the number of false positives and other issues less experienced users sometimes face with a powerful tool such as Suricata. Note that the new auto-SID management feature is disabled by default. You can enable the feature and upload or create the necessary conf files on the new SID MGMT tab.
Bill
-
I bow deeply for you, Lord Bill ;D
-
The new Suricata package update has been posted for the pfSense developers to review. You can follow the request here:
So everyone is aware, Bill thinks he's found a bug, and has asked us to hold off.
-
I can confirm the bug. My recommendation (if Bill agrees) is to go ahead and release the package, since the bug is there in the existing package anyway (and by the looks of it, it's upstream).
Just to be clear, the bug affect IPv6 deployments. There are workarounds, but no need to go through that if the fix is just round the corner. If the fix isn't around the corner though, having the new package is better than waiting for the fix, since the fix is unlikely to be backported to the old (current) package.
As i said, my recommendation is to go ahead with the package and just issue the warning for a known bug and the workarounds. One of the workarounds can be implemented using modifysid for example. My $0.02.
-
@jflsakfja:
I can confirm the bug. My recommendation (if Bill agrees) is to go ahead and release the package, since the bug is there in the existing package anyway (and by the looks of it, it's upstream).
Just to be clear, the bug affect IPv6 deployments. There are workarounds, but no need to go through that if the fix is just round the corner. If the fix isn't around the corner though, having the new package is better than waiting for the fix, since the fix is unlikely to be backported to the old (current) package.
As i said, my recommendation is to go ahead with the package and just issue the warning for a known bug and the workarounds. One of the workarounds can be implemented using modifysid for example. My $0.02.
Yes, the bug is in the IPv6 address parsing logic. It is present upstream and comes into play if you set $EXTERNAL_NET to the standard value of !$HOME_NET. It only impacts IPv6 traffic, but the impact is you don't get IPv6 alerts from rules where source or destination is $EXTERNAL_NET. A good many rules use this qualifier.
The Suricata source code around this functionality is quite complicated and IMHO not well commented. I have thus far been unable to locate the source of the problem. I think the best I can do is report this upstream. I want to conduct a little more testing, and then I will release the Pull Request for further review by the pfSense developers.
Bill
-
Oh c'mon people. We are crying where and you said hold off?? you are kidding right? Bill :P
-
@mais_um:
Oh c'mon people. We are crying where and you said hold off?? you are kidding right? Bill :P
Not kidding about the bug in IPv6 addresses when $EXTERNAL_NET is set to !$HOME_NET. If you set $EXTERNAL_NET to "any", then no problem except that causes a ton more alerts that are mostly false positives. This is because many rules are written to discriminate traffic based on $HOME_NET and $EXTERNAL_NET reflecting your true setup (where $HOME_NET represents only the networks your protecting, and $EXTERNAL_NET is everything else). If $EXTERNAL_NET is set to "any", then this rule paradigm is not true.
The Suricata and Snort packages have always used "$EXTERNAL_NET = !$HOME_NET" on pfSense as the defaults. So I have decided for now to not change those defaults and instead post a warning that IPv6 traffic will not always be correctly alerted on in Suricata until either I or the upstream Suricata guys can find out what's wrong in the binary.
Bill
-
You'll have to get Supermule to acknowledge that the bug isn't in pfSense, but is, rather, upstream. I don't want to have to hear his complaints.
-
I can also confirm this bug.. Hopefully it can be found but at least there is a workaround for now…
-
I can also confirm this bug.. Hopefully it can be found but at least there is a workaround for now…
I've sent a message to the Suricata team about it, but received no response yet. Also tagged onto a similar (if not possibly the same) issue posted on the Suricata Bug Tracker Redmine site.
I've released the package for review by the pfSense guys, but will continue looking for the bug in the binary. It's a complicated source code package, and it's a little tough to reverse engineer something in the first place, and as I mentioned previously, IMHO there is not a lot of commenting in the code explaining the functions or logic flow. So finding this bug is a challenge…but I do love a challenge... ;)
Bill
-
Thanks Bill! Looking forward to the new release
-
Thanks
I will test PPPoE support in my WAN interface, and other things.
-
Update on IPv6 bug progress –
I have received an acknowledgement from the Suricata project guys on the IPv6 bug. At least one of them was able to reproduce the issue with IPv6 addresses within rule variables not generating alerts even on a match where they should. Hopefully a fix from them will be forthcoming soon. In the meantime, I'm continuing to look for the bug on my own.
As mentioned in a post above, I released the package for review and merge by the pfSense team. If the binary fix for Suricata comes through before they complete the GUI package code review, I will try to get it included with the initial 2.0.3 binary and GUI v2.0 release. If not, then as soon as the binary fix becomes available, I will post an update to the update (that is, a v2.0.1 of the GUI package versus what will be v2.0 of the GUI package).
The bug only impacts IPv6 alerts, and only when rules contain RULE VARS. But since a ton of the rules do contain the RULE VARS $EXTERNAL_NET and $HOME_NET, then a lot of IPv6 alerts don't happen. However, IPv4 works just fine. So if you have an IPv4-only installation, you will have no issue. Only IPv6 setups will have some potential "no alerts" on IPv6 traffic until the bug in the binary is found and fixed.
Bill
-
How 'bout just merge it, call it "beta" and let us play with it already… I know, i know... I need a life... LOL! ;D
-
We need this package update as soon as possible. Suricata just doesn't stay running for me right now.
-
Is that REALLY the level that you are dragging everything down to??
Disgusted…
@gonzopancho:
You'll have to get Supermule to acknowledge that the bug isn't in pfSense, but is, rather, upstream. I don't want to have to hear his complaints.
-
Bump…
-
The IPv6 bug has been found :D :D :D
I have submitted the Pull Request to the Suricata Github site containing the fix. I will also soon be sending it to the pfSense team. Although it worked in all my testing, the pfSense team and I would still like to get confirmation of the fix from the Suricata developers. So give us another day or so.
Edit – updated URL to point to most recent request
If you are interested, here is the link to the Suricata Github pull request: https://github.com/inliniac/suricata/pull/1120Bill
-
The IPv6 bug has been found :D :D :D
I have submitted the Pull Request to the Suricata Github site containing the fix.
'
This line should have been written:
I have found The IPv6 bug :D :D :D
I have submitted the Pull Request to the Suricata Github site containing MY fix.
I love having a package maintainer who is an active contributor of the software he maintains. 8)
Great job as always Bill.
-
@AhnHEL:
The IPv6 bug has been found :D :D :D
I have submitted the Pull Request to the Suricata Github site containing the fix.
'
This line should have been written:
I have found The IPv6 bug :D :D :D
I have submitted the Pull Request to the Suricata Github site containing MY fix.
I love having a package maintainer who is an active contributor of the software he maintains. 8)
Great job as always Bill.
Thanks… ;)
I spent many, many hours poring over the Suricata source code trying to find that bug. I first had to figure out how Suricata works internally, and after that start tracking down where and how some IPv6 address comparisons were failing. Finally found the problem last night and started working on a fix. My eyes are crossed and I tend to see everything as C source code now... ;D
Bill
-
Awesome work!!!
"If you are interested, here is the link to the Suricata Github pull request: https://github.com/inliniac/suricata/pull/1119"
Awesome description, I actually understood this… LOL. I really need to get out.
It's IPv6, though it's important and it should be adapted quicker, Much quicker. Many of us are still stuck at IPv4 and could really use the update. This update will accept more 'Modern' rules', allows updates without having to manually edit files, and adds many abilities many people are looking for. Looking at the rate that Suricata merges important updates, and the time it takes the pfSense team to push them our way. It would be nice to have the update merged, and when the Suricata team makes this available have this as another update.
Last Suricata merge was on Aug. 15th.
Updates, especially security related updates can not wait 30 days… :)
-
@AhnHEL: Others helped identify that there was indeed a bug, you know.
@wcrowder: Security fixes faster than 30 days? Are you F***ING CRAZY? We must get the gold button in the code before security fixes. Getting paid is more important than actually providing something to get paid for.
I'm not sure [sarcasm] tags are appropriate, given the recent(ish) Ciscofying of pfSense.
Queue mod deletion in 3…2...1...
-
Ciscofying is a cool word :D
@jflsakfja:
@AhnHEL: Others helped identify that there was indeed a bug, you know.
@wcrowder: Security fixes faster than 30 days? Are you F***ING CRAZY? We must get the gold button in the code before security fixes. Getting paid is more important than actually providing something to get paid for.
I'm not sure [sarcasm] tags are appropriate, given the recent(ish) Ciscofying of pfSense.
Queue mod deletion in 3…2...1...
-
Ciscofying is a cool word :D
It is also unfortunately the truth. I'm expecting the announcement that you have to pay a subscription if you want your packages updated any day now.
-
Thats what I have been yelling about for quite some time…
Apparently the karma button doesnt like that :D
But the comfort of that is I know it can be manually changed so no matter what, it will never become positive in my lifetime in here :D
Buts its freaking annoying that you have to deal with kiddoes like that and not grown ups.
Currently trying to find a sponsor to paid development of a fork of pfsense. Just like the one that happened to M0n0wall to PFsense.
-
A fork isn't needed if the devs (not only of this project, all open source projects) learn what the correct procedure to having a project of their own is: Stop trying to re-invent the wheel, stick with upstream, and slap on your customizations as a separate package, available upstream.
Imagine installing the latest freebsd version, and just installing the pfsense package, which transforms that freebsd into a full blown pfsense.
It already takes care of:
- keeping up to date with security fixes
- keeping up to date with outdated drivers
- actually implementing a proper syslog for pfsense (anyone that thinks the current way to log is the absolute best, please do humanity a favor, here's a gun, here's a bullet)
- keeping up to date with packages. Yeap, no more waiting for devs to approve a newer suricata version
Will the projects realize that sticking with upstream is the proper way to do it? Don't be silly, of course not. Pride, greed, power, all have something to do with the dev's denial to accept that what I say is in fact the truth.
Eventually pfsense will be forked. My only hope is that the fork follows the upstream example given above and not waste developers time running around in circles. Install a freebsd base, then install the firewall-webgui package which takes care of installing the necessary dependencies and providing a way for you to configure the system. If that happens, I'll be the first to say "so long and thanks for all the fish, so sad that it should come to this".
