Suricata 3.2.1 Package Update – Release Notes
-
Suricata 3.2.1 Package Update
This updates the Suricata package on pfSense to version 3.2.1. The underlying Suricata binary is also versioned up to 3.2.1. This update is initially available for pfSense 2.4-BETA snapshots, but will become available for the 2.3.x Release versions of pfSense shortly. Some Suricata GUI configuration parameters were changed as a result of the update. See the Release Notes below for details.
Important Upgrade Information
The recommended way to upgrade the Suricata package is to first remove it and then reinstall it. This bypasses any caching that may occur with the PHP code files. This is particularly important for this update as some suricata.yaml configuration parameters have changed and must be migrated to new values during the installation of the package. Removing and then reinstalling the package ensures the latest PHP code files are used to perform the migration of impacted configuration parameters.Release Notes:
1. Suricata 3.2.1 now supports hyperscan for the pattern matcher algorithm. Hyperscan is a high-performance regex pattern matching library. Several older pattern matching algorithms were deprecated. If your existing Suricata configuration is using any pattern matcher algorithm not shown in the list of acceptable values below, the setting will be migrated to "Auto". If your existing configuration is "AC", then it will be left at that value and you will need to manually change the Pattern Matcher setting on the INTERFACE SETTINGS tab. The new valid options for Pattern Matching are:
Auto - will use hyperscan when available, else defaults to AC AC - Aho-Corasick (default implementation) AC-BS - Aho-Corasick (reduced memory implementation) AC-KS - Aho-Corasick (Ken Steele variant) HS - Hyperscan (available when built with hyperscan support)Please note that hyperscan is only available with 64-bit builds of pfSense. There is no hyperscan support available on 32-bit versions of pfSense. This is a limitation of the hyperscan library. If you have a 32-bit system and attempt to force hyperscan mode, it will not work. Leaving the setting in AC or Auto is suggested for 32-bit installations.
You should generally leave the Pattern Matcher setting on the INTERFACE SETTINGS tab set to "Auto". With this setting, hyperscan will be used if available; otherwise "AC" will used. For existing installations where your Pattern Matcher setting was "AC", you should change the setting to "Auto" after upgrading. I made the choice not to automatically make this change during the upgrade installation in case a user had chosen "AC" for a specific reason. "AC" is a safe default. If you have a 64-bit build of pfSense and wish to use hyperscan pattern matching, make the change on the INTERFACE SETTINGS tab, save it, and then restart Suricata on the interface.
2. Two additional hashing algorithms (SHA1 and SHA256) were added to the Tracked Files option. The old binary config parameter for switching MD5 hashing of tracked files ON or OFF is changed to a select drop-down with choices of "None", "MD5", "SHA1" and "SHA256". This option is part of the logging options on the INTERFACE SETTINGS tab. Formerly it was an On/Off checkbox to toggle MD5 hashing on or off. The option is now a select drop-down. Choose "None" if you wish to disable hashing for logged Tracked Files, otherwise choose one of the three available hashing algorithms. The default for this option is "None".
3. Two new EVE JSON logging options were added for logging SMTP traffic and DROPPED traffic. These are enabled by default when EVE JSON logging is enabled. Note that the DROPPED traffic option can consume quite a large amount of disk space on a busy network. This option logs all packets that are dropped when using inline IPS mode in JSON format. The DROPPED traffic option is hidden and not used if Legacy Mode is chosen for the IPS Mode.
Bill
-
Thanks!
What kind of noticeable improvements can we expect to see (in general) switching to Auto that will use hyperscan?
Is it something noticeable at a user level? Will small networks notice it at all? Will it reduce CPU usage at all?
I see Intel's slides on Hyperscan in Suricata (slides 15-21),
https://suricon.net/wp-content/uploads/2016/11/SuriCon2016_GeoffLangdale.pdfBut it starts with a bunch of disclaimers, is filled with lots of things I don't understand, and ends with:
You can integrate a free (as in speech, and as in beer) library into Suricata …
- … and roughly double your performance
I was hoping you could translate into real world improvements we could see in pfSense using Suricata in layman's terms?
Because from Intel's slides I can't tell whether this is a big deal or a non-event. -
Thanks!
What kind of noticeable improvements can we expect to see (in general) switching to Auto that will use hyperscan?
Is it something noticeable at a user level? Will small networks notice it at all? Will it reduce CPU usage at all?
I see Intel's slides on Hyperscan in Suricata (slides 15-21),
https://suricon.net/wp-content/uploads/2016/11/SuriCon2016_GeoffLangdale.pdfBut it starts with a bunch of disclaimers, is filled with lots of things I don't understand, and ends with:
You can integrate a free (as in speech, and as in beer) library into Suricata …
- … and roughly double your performance
I was hoping you could translate into real world improvements we could see in pfSense using Suricata in layman's terms?
Because from Intel's slides I can't tell whether this is a big deal or a non-event.To be honest, for home users, there will be not one bit of noticeable difference in performance. This is because home networks and very small business networks just don't generate enough traffic to tax an IDS/IPS running on modern hardware.
Large corporate network users and others with heavily loaded gigabit and above connections will notice some performance improvement with hyperscan enabled. This would show up as fewer dropped packets under heavy loads. An IPS will drop packets at very heavy load rates. That's how you know you need to put more iron under the IPS.
Now one thing hyperscan can help with is increased performance with DPI (deep packet inspection). Again, though, this will be more noticeable to someone with a symmetrical Gigabit connection or higher that is running at 80% or higher utilization continuously.
In summary, I added hyperscan support to the pfSense package because it was added upstream, and there is no downside to using it. For 32-bit installations you would just continue to use "AC" or one of the other two options for MPM (multi pattern matcher). For 64-bit installations, hyperscan is better, but it's just not "wow" sort of better for most users.
Bill
-
Hyperscan gave me a noticeable improvement. When using AC, my measured WAN speed was usually ~300 Mbps. Now it exceeds 350 Mbps using HS. The WAN speed is usually between 350-400 Mbps, so this is a welcome addition. Thank you for the update!
-
Hyperscan gave me a noticeable improvement. When using AC, my measured WAN speed was usually ~300 Mbps. Now it exceeds 350 Mbps using HS. The WAN speed is usually between 350-400 Mbps, so this is a welcome addition. Thank you for the update!
That's a happy surprise. I would not have expected hyperscan to start making appreciable performance improvments until you got to Gigabit territory. I have no way of adequately testing as my Internet connection here in the boonies is not that fast. I did get an upgrade to 100 megabits/sec down and 10 megabits/sec up last week, though. I was formerly limited to 24 meg down and 2 up via cable modem where I live.
Bill
-
I'm also happy to report that transferring a sample .ISO file of a few gigabytes over HTTP with pattern matcher being set to AC resulted to 15-20M/s speeds with my connection. With HyperScan I'm seeing 24-29+M/s which is very close to the line speed.
I'm on a 250/50 fiber.
