Suricata 7.0.0 Package Update for DEVEL Snapshots -- Release Notes

[bmeeks]

Suricata 7.0.0

An update for the Suricata package has been submitted for the DEVEL Snapshots branch of pfSense. This new package should show up in the DEVEL branch package repos for snapshot testers once the repos are reactivated for testing and the packages rebuild using the latest development kernel. After adequate testing in the DEVEL branch, and if no major issues surface, then Suricata 7.0.0 will be posted for the RELEASE branches of pfSense CE and Plus.

This package update provides support for the latest 7.0.0 version of the binary from upstream. Additionally, a few Memory Cap defaults have been increased to match the new OISF suggested default values.

Note: these changes to the default Memcap values for Streams and Flows will increase the memory footprint of Suricata when running. On hardware with less than 4GB of RAM, you may hit memory issues when using a very large ruleset. This may become acute on systems with 2GB of RAM or less.

New Features:

1. Support for new Exception Policies for flows, streams, and app parsers. These new policy settings default to "ignore". With the default settings, Suricata 7.x running with Inline IPS Mode will behave the same as older versions. But the new Exception Policies allow an administrator much greater flexibility in securing the IDS/IPS when off-normal situations are encountered, including configuring a "fail closed" setup. More details on Exception Polices can be found at: https://docs.suricata.io/en/suricata-7.0.0/configuration/exception-policies.html. Be sure to read and understand the settings as described in the linked documentation before you change any Exception Policy settings!
2. The default Stream Memcap setting is increased from 32 MB to 256 MB to prevent most common startup failures due to insufficient stream memory space. Any user setting below 256 MB will be automatically updated to 256 MB during package installation/upgrade.
3. The default Flow Memcap setting is increased from 32 MB to 128 MB. Any user setting below 128 MB will be automatically adjusted to 128 MB during package installation/upgrade.
4. Additional parameters were added for configurating conditional PCAP logging. The user can now select when Suricata will obtain and store PCAP files of traffic. The conditions are ALERT, ALL, or TAG. The packet captures are stored in a pcaps subdirectory of the logging directory for the interface.
5. The location of the Pass List drop-down selector has been moved into the ALERTS and BLOCKS section of the Interface Settings tab along with the other blocking configuration settings. This was done to group the Legacy Blocking family of parameters all together on the page.

Bug Fixes:

1. Fix Redmine Issue #14530, TCP Stream Memcap parameter help text on FLOW/STREAM tab incorrectly converts MB to bytes.
2. Fix Redmine Issue #12899, Suricata doesn't honor Pass List. Several changes were made to the Pass List logic within the custom Legacy Blocking Module used by Suricata on pfSense that hopefully squash this long-standing and very random bug.

[NRgia]

Just as a feedback, I have updated from 23.05.1 to 23.09:

• Suricata package updated automatically to 7.0
• no issues for now

[michmoor]

@bmeeks Regarding the conditional pcap logging, is it possible to do a TAG of all alerts set to Drop?

The documentation has the following
"If the variable is set to alerts then only the flow with alerts will be logged. If the variable is set to tag then only packets tagged by signatures using the tag keyword will be logged to the pcap file. "

So my thinking is that within the GUI, set the rules you want to Drop using SID MGMT while also adding the 'tag' to amend those rules.
This will go a long way in helping us analyze rules that specifically trigger.

[bmeeks]

@michmoor said in Suricata 7.0.0 Package Update for DEVEL Snapshots -- Release Notes:

So my thinking is that within the GUI, set the rules you want to Drop using SID MGMT while also adding the 'tag' to amend those rules.
This will go a long way in helping us analyze rules that specifically trigger.

The upstream Suricata documentation for the tag rule keyword is very sparse it appears. I did not stumble upon any full explanation nor even examples in my brief searches.

There is currently no method easily available within the SID MGMT logic in the package to add a tag. You might be able to do that using the modify.conf file logic, but selecting the rules to modify may prove challenging. You could always create your own set of custom rules containing the tag keyword, though, and use those.

[michmoor]

@bmeeks said in Suricata 7.0.0 Package Update for DEVEL Snapshots -- Release Notes:

You could always create your own set of custom rules containing the tag keyword, though, and use those.

That was going to be my fallback method if a better way wasnt found. Just a pain if you are dropping on quite a few rule sets.

[NollipfSense]

@bmeeks said in Suricata 7.0.0 Package Update for DEVEL Snapshots -- Release Notes:

https://docs.suricata.io/en/suricata-7.0.0/configuration/exception-policies.html.

Clicking on the link leads to: https://redmine.pfsense.org/issues/12899 while plugging in the link leads to:

[bmeeks]

@NollipfSense:
Fixed now. Was a copy-paste error in the link code way back from when I created the post.

[NRgia]

Hello @bmeeks , wanted to ask if there are any plans to update Suricata binary to 7.0.2 . I see the port is available https://www.freshports.org/security/suricata/#history

Maybe it will be more stable. Thank you

[bmeeks]

@NRgia said in Suricata 7.0.0 Package Update for DEVEL Snapshots -- Release Notes:

Hello @bmeeks , wanted to ask if there are any plans to update Suricata binary to 7.0.2 . I see the port is available https://www.freshports.org/security/suricata/#history

Maybe it will be more stable. Thank you

I honestly do not know. My testing package builders can now no longer function as they once did due to changes Netgate made in how pfSense itself is built. There are now proprietary packages included in the standard pfSense package repo build configuration. Those proprietary packages pull their source code from a private Netgate Gitlab account. Finding all the places where there are build or runtime dependencies on those proprietary packages and removing them is cumbersome and has to be repeated each time I update my Poudriere Ports source tree from pfSense upstream.

[NRgia]

@bmeeks So, only a Netgate developer can do that now ? Or is that thing that between releases the binary versions are frozen ? Do you know to whom I can address a question?

[bmeeks]

@NRgia said in Suricata 7.0.0 Package Update for DEVEL Snapshots -- Release Notes:

@bmeeks So, only a Netgate developer can do that now ? Or is that thing that between releases the binary versions are frozen ? Do you know to whom I can address a question?

It's a complicated answer to your question. Suricata on pfSense has a custom patch compiled in to provide Legacy Mode Blocking. That patch is not part of anything that comes from upstream.

For most binary updates, the Legacy Blocking Mode module's patch applies fine. But every now and then it does not and that breaks the binary package build. I then have to rewrite portions of the blocking module's C source code and produce a new patch file. The Netgate team is not familiar with the custom patch. I wrote it and maintain it. A similar condition exists for the Snort package. And in fact, there are two custom patches for Snort: one for its Legacy Blocking Module and another in the DAQ library to enable netmap in Snort to work with host stack endpoints.

If they just pull in the latest binary update from upstream, then it is possible to break the package build when the patch is included. If they omit the patch, then Legacy Blocking Mode is unavailable and some of the PHP code might also then break as it expects the module to be there.

Edit: I don't mean to imply the custom code is my proprietary work. It is all freely posted as open-source software in the pfSense Ports GitHub repo. I'm just saying it would take someone else a bit of time to analyze and understand what the custom module code is doing and then be able to support it going forward.

[Bob.Dig]

@bmeeks That doesn't sound good.

[bmeeks]

@Bob-Dig said in Suricata 7.0.0 Package Update for DEVEL Snapshots -- Release Notes:

@bmeeks That doesn't sound good.

I'm not sure many users of some popular pfSense packages realize many of the packages are created and maintained by volunteers not associated with Netgate in any way. If the volunteer maintainer of a package leaves, that package dies on the vine unless Netgate devotes their time and resources to take over its maintenance. Obviously there are limits to how much Netgate can pour into "free packages" and still maintain rigorous development and support of core pfSense itself.

Word to the wise -- never run a business-critical component on free volunteer maintained software that could disappear on you at any time . If you can also maintain said software yourself should the maintainer disappear, then that changes the equation.

[michmoor]

@bmeeks said in Suricata 7.0.0 Package Update for DEVEL Snapshots -- Release Notes:

Those proprietary packages pull their source code

Are these packages only in the Plus version of the code or the CE?
Important to note because from my understanding from the marketing by Rubicon is that this is an Open Source Firewall.

[michmoor]

@bmeeks said in Suricata 7.0.0 Package Update for DEVEL Snapshots -- Release Notes:

I'm not sure many users of some popular pfSense packages realize many of the packages are created and maintained by volunteers not associated with Netgate in any way. If the volunteer maintainer of a package leaves, that package dies on the vine unless Netgate devotes their time and resources to take over its maintenance. Obviously there are limits to how much Netgate can pour into "free packages" and still maintain rigorous development and support of core pfSense itself.

Word to the wise -- never run a business-critical component on free volunteer maintained software that could disappear on you at any time . If you can also maintain said software yourself should the maintainer disappear, then that changes the equation.

And this is the exact reason why i caution anyone running pfsense to keep package use to a bare minium or to none at all. The core pf project has developers on payroll so you know there will always be support there. Everything else is at the mercy of a 3rd party.
For those who follow my other posts, this is why i am instructing people to stop using Squid. There is no maintainer for it at the moment. Redmines are open but no one is touching it because by all accounts there is no volunteer.
The SquidGuard project is another one. I reached out to the dev (whos name is in the package) and he responded nicely saying he hasnt been involved in that for years.
Thats just 2x packages i listed that by all accounts are not being maintained by anyone.

[bmeeks]

@michmoor said in Suricata 7.0.0 Package Update for DEVEL Snapshots -- Release Notes:

Are these packages only in the Plus version of the code or the CE?

For the moment they are in both. Definitely in CE as that's the only branch I have access to. Everything for Plus is hosted on the private Netgate GitLab I think. Periodically that GitLab is merged into the public GitHub, but certain proprietary pieces remain hosted on the private GitLab.

I first noticed this problem with 2.8 CE deveopment. The major stumbling block right now is the pfSense-repoc package that is new with the most recent 2.8 CE snapshots starting back in the late summer if I recall the date correctly. That is new binary code that handles connecting to the appropriate pkg repo to pull down updates for pfSense and any packages. The source code for that package is tucked away on the private GitLab repo. Ditto for the module that generates the NDI (Netgate Device ID), but that module is only compiled into the kernel, and since I don't build the kernel commenting it out was not a big deal. It's more burdensome to find and comment out the references to pfSense-repoc and any other similarly proprietary package code so the remaining packages can build.

There are some built-in switches originally provided by Netgate to allow you to build a pfSense kernel but only by NOT calling it "pfSense". You had to give it a different prefix name. So "non-Sense", for example. If all you want to do is build a kernel, that workaround is fine. So, the open-source moniker is still more or less appropriate. But if you want to build packages for testing directly in pfSense using the standard pfSense GUI tools, then you need to call your build "pfSense" so your packages get named correctly. But using that prefix triggers a lot of automated build options which will call in those proprietary packages unless you find and comment out each one.

[michmoor]

@bmeeks said in Suricata 7.0.0 Package Update for DEVEL Snapshots -- Release Notes:

But using that prefix triggers a lot of automated build options which will call in those proprietary packages unless you find and comment out each one.

Devils in the details i see....
But thanks for clearing that up.

[NRgia]

@michmoor said in Suricata 7.0.0 Package Update for DEVEL Snapshots -- Release Notes:

@bmeeks said in Suricata 7.0.0 Package Update for DEVEL Snapshots -- Release Notes:

I'm not sure many users of some popular pfSense packages realize many of the packages are created and maintained by volunteers not associated with Netgate in any way. If the volunteer maintainer of a package leaves, that package dies on the vine unless Netgate devotes their time and resources to take over its maintenance. Obviously there are limits to how much Netgate can pour into "free packages" and still maintain rigorous development and support of core pfSense itself.

Word to the wise -- never run a business-critical component on free volunteer maintained software that could disappear on you at any time . If you can also maintain said software yourself should the maintainer disappear, then that changes the equation.

And this is the exact reason why i caution anyone running pfsense to keep package use to a bare minium or to none at all. The core pf project has developers on payroll so you know there will always be support there. Everything else is at the mercy of a 3rd party.
For those who follow my other posts, this is why i am instructing people to stop using Squid. There is no maintainer for it at the moment. Redmines are open but no one is touching it because by all accounts there is no volunteer.
The SquidGuard project is another one. I reached out to the dev (whos name is in the package) and he responded nicely saying he hasnt been involved in that for years.
Thats just 2x packages i listed that by all accounts are not being maintained by anyone.

I see your reasoning, but then why publish this: https://www.netgate.com/blog/suricata-vs-snort and state: "...The good news is that regardless of which solution you choose, it will be compatible with pfSense Plus software..." if you don't offer the proper support to the maintainer?

Just wanted to understand what are the plans going forward with this package. If @bmeeks is hindered to build or maintain the package I wonder what will happen to pfblockerNG and other packages...Why this approach ?