Suricata 2.0.3 pkg v2.0.1 Update – Release Notes
-
The long awaited update for the Suricata package on pfSense has been posted. Here are the release notes for the new version.
Suricata 2.0.3 package version 2.0.1
EDIT: the GUI package version was bumped to 2.0.1 due to a function name typo bug fix Renato applied for me (thanks Renato :D).
This updates the Suricata binary to version 2.0.3 and the GUI package to version 2.0.1. A number of bug fixes and new features are included with this update.
IMPORTANT INFORMATION: The alert log format in this new version has been changed back to the native Suricata format. Previously there was a pfSense-specific patch to output alerts in a CSV format. That became problematic because some signature message fields apparently also have some commas in them. Those extra commas would confuse the ALERTS tab code when parsing the alerts log. For this version (and future versions), the CSV patch was dropped. The impact on previous 1.4.6 version users is that the alerts.log file will not parse correctly. So prior to upgrading, clear your alerts log by clicking the CLEAR button on the ALERTS tab. Failure to do this may lead to some errors on the ALERTS tab or on the Dashboard Widget. If you have already upgraded and are seeing ALERTS tab or Dashboard Widget errors, clear the alerts log and that should fix things up.
Snort VRT Rule Set Users: you must now go to the GLOBAL SETTINGS tab and enter the Snort VRT rules snapshot filename you want to download. Formerly this was defaulted to snortrules-snapshot-edge.tar.gz. However, beginning in July of 2014 the VRT discontinued posting of this file (which always pointed to the latest version of the rule set). Instead, you must now specify a specific rule version. So a text box field has been added on the GLOBAL SETTINGS tab. It will appear when you enable Snort VRT rules in Suricata. Type the current VRT rules filename in the box. Provide the filename only! Not the URL. For example, as of early September 2014, the current filename is snortrules-snapshot-2962.tar.gz. The filename you provide is saved in the Suricata configuration. Remember to periodically check the Snort VRT site and update the filename when a new version number of rules is released. The version number changes approximately once per quarter or so. You can check the versions available here: https://www.snort.org/downloads/#rule-downloads
Bug Fixes
1. The phrase "sid-msg.map" is misspelled as "sig-msg.map" in some log messages.
2. On the LOGS MGMT tab, when choosing NO LIMIT for max log sizes or KEEP ALL for retention periods, the change is not saved and the setting reverts back to the default values.
3. The cron task for rotating log files reports an error about a missing file when a log file is not actually present because it has not been enabled. The cron task should check first to see if the log file is present before attempting to check for rotation.
4. On the BLOCKED and ALERTS tabs, certain events may trigger an error similar to: "Warning: inet_pton(): Unrecognized address…". This also results in garbled formatting for the tab display.
5. IPv6 Link-Local addresses are not correctly parsed and added to the HOME_NET variable and to the default pass list. The interface is left appended to the end of the address, and this causes Suricata to throw an error and not parse the HOME_NET variable.
6. Pass Lists created on the PASS LIST tab are not available in the drop-down for HOME NET selection on the INTERFACE tab for a Suricata instance.
7. Suricata decoder events are not logged in the ALERTS tab of the GUI.
8. The cron task for clearing blocked hosts sends an error e-mail when there are no hosts currently being blocked. Need to add the "-q" option to the pfctl command line.
9. After recent changes at the snort.org web site, Snort VRT rules no longer download in Suricata because the VRT eliminated the snortrules-snapshot-edge.tar.gz file the Suricata package uses.
10. The interface name in the widget is not always correct. The name is ok on the alert-tab. e.g. WAN2 is displayed as OPT2 even when OPT2 doesn't exist.
11. IPv6 addresses do not have an unblock button on the alerts page, although they are correctly added to the blocked table (snort2c) and the blocked page.
12. IPv6 networks specified with a CIDR mask in a Pass List are not properly parsed and thus subject to being blocked by Suricata.
13. sksort() function causes redefinition error and crash report on Dashboard when Snort and Suricata dashboard widgets are both displayed at the same time.
14. Sorting of some clickable column headers in tables is broken in newer pfSense versions.
15. When cloning an existing interface to a new one, a new UUID is not created for the cloned interface.
16. A space is allowed as a valid character in the Sensor Name field on the BARNYARD tab, but the barnyard2 binary does not correctly interpret sensor name values containing spaces.
New or Enhanced Features
1. Add option to set specific syslog facility for Suricata logging, and facility and priority for Barnyard2 local syslog output
2. Add sync to CARP members (also includes new SID mgmt conf files as part of sync data)
3. Add definition for variables FTP_SERVERS and SSH_SERVERS to Variables tab
4. Revert rule forced enable/disable icon behavior back to original two-states (forced on or forced off)
5. Add filtering dialog to ALERTS tab
6. Add ability to utilize enablesid.conf, modifysid.conf and disablesid.conf files such as those used by PulledPork or Oinkmaster for automatically managing enabled/disabled rules and rule categories.
7. Add support for new DNS logging feature available in Suricata 2.0.x
8. Add support for new Application-Layer parsers and detection available in Suricata 2.0.x
9. Remove deprecated "stream_max_sessions" parameter from suricata.yaml config
10. Add support for new EVE JSON logging feature available in Suricata 2.0.x
11. Alter reverse DNS lookup icons on ALERTS and BLOCKED tabs to conform to recent changes in pfSense 2.2
12. Add a REFRESH button to the LOGS BROWSER tab to reload the currently displayed file
13. The "Save Settings when Removing Package" parameter on the GLOBAL SETTINGS tab now defaults to "on" for new greenfield installs of Suricata. Upgrades of existing installs will continue to use the old saved setting. Reported via Issue #3838.
Bill
-
There is a cool new feature available in this version. It's on the new SID MGMT tab. You can now use PulledPork and Oinkmaster compatible enablesid.conf, disablesid.conf and modifysid.conf files to automatically manage your rules during updates. This is a somewhat advanced feature, though, and a bit of experience with managing rules manually is helpful before diving in and using the automated feature. The idea is for some of the more advanced Suricata users here on the Forum to help novices by posting some good example configuration files. You can upload, download, and edit-in-place the configuration files. This will make it easy to share files with others if desired. There are also three sample files included with the package installation. There are comments in those files to help guide you in using the new feature.
This feature also makes it very easy to duplicate a complicated rule set across many Suricata installations if desired. Simply upload the files to the additional instances and activate the new automatic SID management feature by checking the box on the SID MGMT tab.
Bill
-
Another often asked for feature was the ability to filter the displayed alerts on the ALERTS tab. That feature is now in the Suricata package. It works pretty much just like the filter dialog available for firewall rules. Below is an example screenshot.
Bill
-
Hi Bill!
Can you implement that filtering in Snort as well?
-
Hi Bill!
Can you implement that filtering in Snort as well?
Yep. It's coming in the next update. I expect to start coding on it in the next day or two. The filter and SID MGMT are coming to Snort.
Bill
-
You know I said I would by you beer and give you a hug if we meet!
Consider that a case and some fine Scottish single malt!
Hi Bill!
Can you implement that filtering in Snort as well?
Yep. It's coming in the next update. I expect to start coding on it in the next day or two. The filter and SID MGMT are coming to Snort.
Bill
-
You know I said I would by you beer and give you a hug if we meet!
Consider that a case and some fine Scottish single malt!
Hi Bill!
Can you implement that filtering in Snort as well?
Yep. It's coming in the next update. I expect to start coding on it in the next day or two. The filter and SID MGMT are coming to Snort.
Bill
But you won't be able to get it without a pfSense Gold subscription! Break out your credit card, Supermule!
Bwahahahahaha!
(I kid. LOL.)
-
Many thanks Bill (and pfsense devs) for making it happen. Time to put the new package to the production test.
As far as the files go, I'm already working on 2 sets of files. One brings a newly installed suricata to the blueprint specs (yes no more enable all disable one by one), and the other is a bare minimum disabled rules.
@gonzopancho: If someone gives me a reason to break out my card, I'd be happy to. I'm not ungrateful as some would like to think. Since this package incorporated features that would save me a lot of time, some of which I have personally requested. Based on that, do you think I'll be donating soon, or not?
-
@jflsakfja:
@gonzopancho: If someone gives me a reason to break out my card, I'd be happy to. I'm not ungrateful as some would like to think. Since this package incorporated features that would save me a lot of time, some of which I have personally requested. Based on that, do you think I'll be donating soon, or not?
since there are no "donations" anymore, the answer is "no".
You could buy Gold though. We'll assume it was because of the menu. :P
-
Thanks for all the hard work Bill! ;D
I found another bug in the Blocks tab (pfSense 2.1 64bit):
Warning: inet_pton(): Unrecognized address TCP in /usr/local/www/suricata/suricata_blocked.php on line 265
Came back even after clearing the blocked hosts.The last "bug" I found is the widget doesn't keep it's old position after updating the package. (other widgets, like pfBlocker do).
-
Thanks for all the hard work Bill! ;D
I found another bug in the Blocks tab (pfSense 2.1 64bit):
Warning: inet_pton(): Unrecognized address TCP in /usr/local/www/suricata/suricata_blocked.php on line 265
Came back even after clearing the blocked hosts.The last "bug" I found is the widget doesn't keep it's old position after updating the package. (other widgets, like pfBlocker do).
Did you clear your logs as suggested by Bill? (Alerts tab > Save or remove logs > Clear button)
EDIT: Yeap, run into this as well.
To get it fixed:
- Stop suricata (so that it doesn't add any new hosts while you are messing with logs)
- Blocked tab > clear all hosts
- Alerts tab > clear all logs
- Logs view > select the alert log. MAKE A NOTE OF THE PATH, IT SHOULD BE DISPLAYED RIGHT ABOVE THE WHITE BOX CONTAINING THE LOG
WARNING!!!
MAKE SURE THAT NO SPACE IS CONTAINED IN THE PATH OR THE FILENAMES BELOW. DON'T SAY I DIDN'T WARN YOU. - Diagnostics> command prompt> enter "rm (path as displayed above)/alerts.log." (an example: "rm /var/log/suricata/suricata_em150851/alerts.log.". Press enter. No output will be given, don't worry
- While still on the command prompt > enter "rm (path as displayed above)/block.log." (an example: "rm /var/log/suricata/suricata_em150851/block.log.". Press enter. No output will be given, don't worry
- restart suricata
8 ) profit
-
Minor typo: Blocked tabs > Save or remove hosts > Clear button, pop up says "blah blah Click OK to continue or CANCLE to quit."
EDIT:
Shouldn't existing disabled rules get replicated over to the CARP slave? Had a rule fire up on the slave that was disabled on the master.
When selecting sync to configured system backup server the replication breaks. You need to explicitly select sync to hosts defined below, and enter the SYNC interface's IP of the slave, along with the correct port (if you followed my guide, it should already be a different port than the default) and the admin password. After that, already disabled rules on the master will be replicated correctly to the slave and everything is fine. -
jflsakfja, thanks. I already removed the logs using the clear button, but it seams the block.log wasn't cleared at all.
alerts.log was 0 bytes after clearing, the block.log was in all interfaces still serveral kb.I also noticed a (very) big html.log file. This was probably because I didn't activate the Automatic log management in "Log Mgnt".
After removing all log and barnyard files the interfaces now start much quicker.Maybe this will also fix the issue I had with some interfaces randomly not starting.
Edit: Just tested the new SID Mgmt with a converted "jflsakfja blueprint" and it works perfectly!
-
Thanks for all the hard work Bill! ;D
I found another bug in the Blocks tab (pfSense 2.1 64bit):
Warning: inet_pton(): Unrecognized address TCP in /usr/local/www/suricata/suricata_blocked.php on line 265
Came back even after clearing the blocked hosts.The last "bug" I found is the widget doesn't keep it's old position after updating the package. (other widgets, like pfBlocker do).
I see a workaround for the BLOCK tab issue has been posted by @jflsakfja, so I won't repeat it. It will work until I get a fix into the next update. I forgot that the block.log file is not cleared. Only the <snort2c>pf table is cleared by the CLEAR button on the BLOCK tab.
Sorry for the problems the change in log file format caused, but going forward I think this is better. It is one less customization of the Suricata binary (the patch to output CSV was on pfSense only). It should also take care of those occasional formatting issues on the ALERTS and BLOCK tabs caused by rule messages in the DESC field that contained embedded commas.
I will look into the issue with the Dashboard Widget remembering its position and see about getting that fixed.
I will give you guys another day or so to report anything else you find, then I will post a fix for everything reported instead of posting little individual fixes for each. That's assuming no show-stopper problems crop up.
Bill</snort2c>
-
I've been testing the new package all day. Threw everything I could at it. I can't find anything else that needs fixing. I've got one last thing I want to test out, after that I can't think of anything more I can do to break it :o
-
@jflsakfja:
I've been testing the new package all day. Threw everything I could at it. I can't find anything else that needs fixing. I've got one last thing I want to test out, after that I can't think of anything more I can do to break it :o
Good news! Thank you for trying to break it… ;)
Just post back here or send me a PM if you find something else. I have collected three issues from the posts thus far here and in the older Preview thread. Those are:
1. Widget doesn't keep it's old position
2. Misspelled "CANCEL" in tooltip for CLEAR button on BLOCKED tab
3. When updating and saving the BARNYARD tab settings, the error "Fatal error: Can't use function return value in write context..." is displayed.
NOTE -- I think #3 may only impact 2.1.x versions of pfSense, but I have not tested to validate. I did specifically test that change on a 2.2 VM before I submitted it and did not get any errors. However, the function it is complaining about is really not necessary in that context, so I will remove it.
Bill
-
I have 3 interfaces running Suricata. Sometimes one or more interfaces won't start. If I start the not started interface manually it works.
Could it be that Suricata should wait longer before starting the next interface (CPU)?.
b.t.w. pfSense v2.1.5 64bit has 6Gb memory (4Gb left) and the CPU (when all interfaces are up) is 6%Besides that Barnyard stops working on all interfaces. Sometimes Barnyard works on one interface, sometimes on two and sometimes on all three. I disabled the pushing of the sigmap (if I remember correctly) on all but one interface, but still the random startups are the same.
I can't access the barnyard tab at all anymore. ("Fatal error: Can't use function return value in write context in /usr/local/www/suricata/suricata_barnyard.php on line 99 ")One feature request: Could you add "User disabled" in the "–- Category Rules Summary ---"
All of this isn't a showstopper. The randomly not starting of interfaces was already in the previous version and the logging is also done in the firewall.
-
I have 3 interfaces running Suricata. Sometimes one or more interfaces won't start. If I start the not started interface manually it works.
Could it be that Suricata should wait longer before starting the next interface (CPU)?.
b.t.w. pfSense v2.1.5 64bit has 6Gb memory (4Gb left) and the CPU (when all interfaces are up) is 6%Besides that Barnyard stops working on all interfaces. Sometimes Barnyard works on one interface, sometimes on two and sometimes on all three. I disabled the pushing of the sigmap (if I remember correctly) on all but one interface, but still the random startups are the same.
I can't access the barnyard tab at all anymore. ("Fatal error: Can't use function return value in write context in /usr/local/www/suricata/suricata_barnyard.php on line 99 ")One feature request: Could you add "User disabled" in the "–- Category Rules Summary ---"
All of this isn't a showstopper. The randomly not starting of interfaces was already in the previous version and the logging is also done in the firewall.
Here is a quick fix for the BARNYARD problem –
Go to DIAGNOSTICS…EDIT FILE and open /usr/local/www/suricata/suricata_barnyard.php
Scroll down in the fie and find this section of code:
// Validate Sensor Name contains no spaces if ($_POST['barnyard_enable'] == 'on') { if (!empty(trim($_POST['barnyard_sensor_name'])) && strpos(trim($_POST['barnyard_sensor_name']), " ") !== FALSE) $input_errors[] = gettext("The value for 'Sensor Name' cannot contain spaces."); }
Edit it to read like this and then save the changes:
// Validate Sensor Name contains no spaces if ($_POST['barnyard_enable'] == 'on') { if (!empty($_POST['barnyard_sensor_name']) && strpos($_POST['barnyard_sensor_name'], " ") !== FALSE) $input_errors[] = gettext("The value for 'Sensor Name' cannot contain spaces."); }
You can copy and paste from this post if you want. The key is removing the calls to the trim() function, but pay attention to the nested parentheses.
As for your random no start problems, how did you originally create the extra interfaces for Suricata? Did you use the + icons to the immediate right of an existing interface on the INTERFACES tab (the icon whose tooltip says "…create a new interface based on this one...")? If so, there was a bug that caused duplicated UUID numbers. If you created your additional interfaces using the + icon at the upper right of the INTERFACES tab, then the duplicate UUID should not be a problem. If you want, PM me and we can work together offline to get this fixed. I have several reports of this problem, but I have been unable to duplicate in my testing environment. I would love to find out what is going on.
Yes, I can add the additional User Disabled count to the rules summary on the RULES tab.
Barnyard2 has proven to be extra troublesome since the update of the underlying binary to the 2.1.3 version. The barnyard2 developers made some changes in the way it writes to the MySQL database, and those have seemed to cause a lot of trouble. I've investigated this a bit, but not in great detail. The core of the problems seem to be, IMHO, caused by a failure to anticipate multiple barnyard2 instances writing to the same MySQL database. That is just my opinion based on a cursory review.
Bill
-
I have a feeling that hitting the apply button does not trigger the sync command to replicate the changes and signal the slave to reload its config.
In /usr/local/www/suricata/suricata_rules.php, under elseif ($_POST['apply']) shouldn't there be a command to trigger the replication? Rule changes only get replicated when you change the replication target, ie trigger the sync manually.
Any insights as to what I can change to test it? I've spent too many hours looking at text scrolling on the screen and I may be missing something, please do feel free to throw rotten tomatoes at me ;D
-
@jflsakfja:
I have a feeling that hitting the apply button does not trigger the sync command to replicate the changes and signal the slave to reload its config.
In /usr/local/www/suricata/suricata_rules.php, under elseif ($_POST['apply']) shouldn't there be a command to trigger the replication? Rule changes only get replicated when you change the replication target, ie trigger the sync manually.
Any insights as to what I can change to test it? I've spent too many hours looking at text scrolling on the screen and I may be missing something, please do feel free to throw rotten tomatoes at me ;D
Just looked. You are correct that APPLY is not triggering an immediate re-sync. I can fix that. In the interim, you can do the following –
IMPORTANT CAVEAT – for experienced users only! You can break things very badly with a mistake in the steps below. You were warned... ;)
Edit the file /usr/local/www/suricata/suricata_rules.php
In the code section handing the APPLY button that begins with this statement –
elseif ($_POST['apply']) { /* Save new configuration */ write_config("Suricata pkg: new rules configuration for {$a_rule[$id]['interface']}."); /*************************************************/ /* Update the suricata.yaml file and rebuild the */ /* rules for this interface. */ /*************************************************/ $rebuild_rules = true; conf_mount_rw(); suricata_generate_yaml($a_rule[$id]); conf_mount_ro(); $rebuild_rules = false; /* Signal Suricata to "live reload" the rules */ suricata_reload_config($a_rule[$id]); // We have saved changes and done a soft restart, so clear "dirty" flag clear_subsystem_dirty('suricata_rules'); }
Add the line "suricata_sync_on_changes();" to the end just before the final closing brace '}" character. The new section should look like this –
elseif ($_POST['apply']) { /* Save new configuration */ write_config("Suricata pkg: new rules configuration for {$a_rule[$id]['interface']}."); /*************************************************/ /* Update the suricata.yaml file and rebuild the */ /* rules for this interface. */ /*************************************************/ $rebuild_rules = true; conf_mount_rw(); suricata_generate_yaml($a_rule[$id]); conf_mount_ro(); $rebuild_rules = false; /* Signal Suricata to "live reload" the rules */ suricata_reload_config($a_rule[$id]); // We have saved changes and done a soft restart, so clear "dirty" flag clear_subsystem_dirty('suricata_rules'); suricata_sync_on_changes(); }
Save the changes and that should trigger a re-sync to slaves when changing rules (if SYNC is enabled and targets configured).
@jflsakfj: let me know if you try this and works correctly. I will then incorporate the fix into the next update.
Bill