After upgrade to pf+ 23.09 Surricata says it's starting but..
-
@PalisadesTahoe said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
@bmeeks Thank for taking a look. Pattern Matching is set to Auto and is indeed at version 5.4.0:
[23.09-RELEASE][admin@SRV-pfSense-PT]/root: pkg info | grep hyperscan hyperscan-5.4.0 High-performance multiple regex matching libraryI also see you are still running version 7.0.0 of the binary. There is, or will soon be posted, a new 7.0.2 version of the Suricata package. Update to that as soon as it shows up for you.
-
@PalisadesTahoe:
Check out this bug report on the upstream Suricata Redmine: https://redmine.openinfosecfoundation.org/issues/6195. There is a new 7.0.2 version of the Suricata package that should build tonight and show up. That package contains the latest 7.0.2 binary from upstream which contains a fix for the Hyperscan error.Currently you are running the 7.0.0 binary.
-
@bmeeks
I'll give it a go once it shows up, but looking at the bug, it seems related to dealing with a change in a rule pattern after an update which shouldn't be happening to us. But perhaps there are some other bug fixes in 7.0.2 that might remedy this issue. I'll report back after the update is available and applied. Thanks again. Cheers. -
@PalisadesTahoe said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
I'll give it a go once it shows up, but looking at the bug, it seems related to dealing with a change in a rule pattern after an update which shouldn't be happening to us. But perhaps there are some other bug fixes in 7.0.2 that might remedy this issue. I'll report back after the update is available and applied. Thanks again. Cheers.
Since you are using only the ET-Open P2P rules, I can try to see if this is reproducible for me in a virtual machine. Unfortunately I don't have a pfSense Plus test environment. I only have 2.7.0 CE and 2.8.0 CE Development virtual environments to test with.
There was a change made in FreeBSD itself back in August that altered how the kill states system function call works. The Legacy Blocking Code attempts to perform that system call as part of killing the states associated with any IP it adds to the blocking table, snort2c. As an experiment, you could try turning off the Kill States option for the interfaces under the INTERFACE SETTINGS tab. That would bypass calling that modified kernel code. You would need to restart Suricata on the interface after changing and saving that setting. If it works then, that gives me a strong hint of where to concentrate troubleshooting.
The consequence of disabling that option is any active states for a blocked IP will remain open and traffic would continue to flow. So, until the open states timed out, that would essentially convert the IPS to more IDS.
-
@bmeeks
Sad to report that turning off Kill States had no affect. Hyperscan still fails after a few minutes of running. Is it possible to increase debugging of the process in order to get more detail in the logs? Or if you have any other ideas, I'm happy to be the testing lab. Thanks. -
@PalisadesTahoe said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
@bmeeks
Sad to report that turning off Kill States had no affect. Hyperscan still fails after a few minutes of running. Is it possible to increase debugging of the process in order to get more detail in the logs? Or if you have any other ideas, I'm happy to be the testing lab. Thanks.No, there is nothing much to enable debugging wise unless the Suricata binary daemon is compiled with debugging enabled, and we do not do that on the production builds.
Let's see if the 7.0.2 binary makes any difference with respect to the Hyperscan error.
I really thing the Signal 11 core dump is related to killing states in the Legacy Blocking Module. Seeing that in Snort now, and it has the same code. There was a change in FreeBSD in the kill states kernel system call. A Netgate developer made some changes in the Legacy Blocking Module code to sync up with the FreeBSD changes, but I'm thinking those changes might have introduced a bug.
-
@bmeeks
More testing results. I've determined that if set the Pattern Matcher Algorithm to either 'AC' or 'AC-BS', the process runs without issue and does not crash. Detection of P2P traffic takes (unscientifically) a second-ish longer, but seems to be as effective as Hyperscan. If I set to 'AC-KS' the process errors out within minutes with the same 'Hyperscan returned fatal error' that the 'Auto' and 'Hyperscan' settings do. I poked around a little bit but couldn't find much on the differences between AC and AC-BS, or which is preferred for our situation until this gets resolved, but it's working for now. I currently have two LANs set to 'AC' and two on 'AC-BS' and can't seem to tell the difference. Thanks. -
@PalisadesTahoe said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
@bmeeks
More testing results. I've determined that if set the Pattern Matcher Algorithm to either 'AC' or 'AC-BS', the process runs without issue and does not crash. Detection of P2P traffic takes (unscientifically) a second-ish longer, but seems to be as effective as Hyperscan. If I set to 'AC-KS' the process errors out within minutes with the same 'Hyperscan returned fatal error' that the 'Auto' and 'Hyperscan' settings do. I poked around a little bit but couldn't find much on the differences between AC and AC-BS, or which is preferred for our situation until this gets resolved, but it's working for now. I currently have two LANs set to 'AC' and two on 'AC-BS' and can't seem to tell the difference. Thanks.Thank you for this feedback. This bit of info (using AC or AC-BS versus Hyperscan for the pattern matcher setting) probably needs to go back upstream. Might be something particular to FreeBSD, though. For example, I believe there is now a 5.4.1 hyperscan library version out there for other distros, but FreeBSD has (or at least had the last time I checked) only 5.4.0 in FreeBSD-ports.
-
@PalisadesTahoe Thanks for posting your detailed troubleshooting. I just tried AC in my environment and I am also having success getting blocking mode enabled again.
-
@xpxp2002 Spoke too soon. Came back to find my box running 80% CPU (four cores) and 100% mem usage and about 50% swap usage. Stopped all of the Suricata processes, and CPU and memory started recovering right away.
I have yet to see the Suricata 7.0.2 package show up in Package Manager. For now, I'm just going to run without Suricata. It seems to be ok with blocking mode disabled, but it's just too risky to keep trying these types of changes with this version and potentially have the system run out of memory and drop offline.
-
@xpxp2002 said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
I have yet to see the Suricata 7.0.2 package show up in Package Manager.
What version of pfSense are you running? I expected the new binary to be deployed to all pfSense versions by now.
Also, just to note, one of the issues with some of the non-hyperscan pattern matchers is they can become huge memory hogs. Same thing happens in Snort. The default choice is always the best choice. But I know in this case it will choose Hyperscan (as that library is available and the best and fastest) and thus trigger the Hyperscan bug.
-
@bmeeks 23.09
-
I've just sent an email to my Netgate developer contact asking that the 7.0.2 package be deployed to the pfSense Plus 23.09 branch. It was supposed to have already been deployed, so I'm not sure what held it up. Maybe some confusion around the potential Kill States bug. But that bug has no relation to the Hyperscan issue. Those are two separate bugs with separate fixes. The Kill States bug has still not been fully identified, but when it is a separate package update will be issued containing that fix.
-
Got a reply from Netgate. They are currently having issues with the pfSense Plus 23.09 package builder infrastructure. That has held up building and deploying several other package updates. Resolution is in progress, and hopefully everything is back on track later today.
Until that issue is resolved, the Suricata 7.0.2 update can't get built for 23.09
. -
@bmeeks That explains it. I'm hoping that this is the fixed Hyperscan bug that I'm encountering, so hopefully it will be resolved once the build is available. I'll schedule time to test it again tonight if the package is available by then.
-
@xpxp2002 said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
@bmeeks That explains it. I'm hoping that this is the fixed Hyperscan bug that I'm encountering, so hopefully it will be resolved once the build is available. I'll schedule time to test it again tonight if the package is available by then.
Yes, the 7.0.2 package should correct the Hyperscan fatal exit error. Here is the upstream commit (change) that fixed it: https://github.com/OISF/suricata/commit/00e00254eae205bad5d4cfbf6c9e69f944faaf69. It was fixed starting with Suricata 7.0.1 from upstream. Currently you have 7.0.0, but the update that is held up from building at the moment is 7.0.2.
This post of mine in another thread explains the Hyperscan error and the fix: https://forum.netgate.com/topic/184101/suricata-process-dying-due-to-hyperscan-problem/15?_=1700148686823.
-
@bmeeks Where to find HyperScan? Is it a kind of rule or tool used by some rules?
-
@Bob-Dig said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
@bmeeks Where to find HyperScan? Is it a kind of rule or tool used by some rules?
Hyperscan is a library that provides high-speed regex pattern matching capability. It is developed and sponsored by Intel. Have fun learning about it here: https://github.com/intel/hyperscan
. Additional info on Hyperscan can be found here: https://www.intel.com/content/www/us/en/developer/articles/technical/introduction-to-hyperscan.html.It is a technology that Suricata incorporates as a shared-library build and runtime dependency. It only works on Intel platforms (meaning you must have an AMD64 CPU). It does not work for ARM-based hardware, and thus is not compiled into nor enabled in Suricata binary builds on those platforms.
-
@bmeeks Thanks. Just wondering why I don't see those problems but I am also not watching the Suricata logfiles. Right now, there is nothing with "Hyperscan" in it.
-
@Bob-Dig said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
@bmeeks Thanks. Just wondering why I don't see those problems but I am also not watching the Suricata logfiles. Right now, there is nothing with "Hyperscan" in it.
You may not have any rules enabled that produce a particular pattern matcher sequence that trips up Hyperscan. Whether or not you see the Hyperscan bug is dependent upon the exact rules you have enabled.
It is a complicated technology. It attempts to precompile certain types of frequently used/accessed regex patterns into a memory database. That way, as rules attempt to match patterns in network packets' data, the whole process can be faster and more efficient.
But in the 5.4.0 version of Hyperscan, Intel made some changes designed to make things even faster; but those changes altered significantly how Suricata and other consumers of Hyperscan needed to function. The result was a sort of regression bug. That's what folks are seeing in 7.0.0 versions of the Suricata binary on Intel hardware when using Hyperscan. Suricata upstream made some corrections in their code starting with the 7.0.1 version.
