After upgrade to pf+ 23.09 Surricata says it's starting but..
-
@bmeeks
Sad to report that turning off Kill States had no affect. Hyperscan still fails after a few minutes of running. Is it possible to increase debugging of the process in order to get more detail in the logs? Or if you have any other ideas, I'm happy to be the testing lab. Thanks. -
@PalisadesTahoe said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
@bmeeks
Sad to report that turning off Kill States had no affect. Hyperscan still fails after a few minutes of running. Is it possible to increase debugging of the process in order to get more detail in the logs? Or if you have any other ideas, I'm happy to be the testing lab. Thanks.No, there is nothing much to enable debugging wise unless the Suricata binary daemon is compiled with debugging enabled, and we do not do that on the production builds.
Let's see if the 7.0.2 binary makes any difference with respect to the Hyperscan error.
I really thing the Signal 11 core dump is related to killing states in the Legacy Blocking Module. Seeing that in Snort now, and it has the same code. There was a change in FreeBSD in the kill states kernel system call. A Netgate developer made some changes in the Legacy Blocking Module code to sync up with the FreeBSD changes, but I'm thinking those changes might have introduced a bug.
-
@bmeeks
More testing results. I've determined that if set the Pattern Matcher Algorithm to either 'AC' or 'AC-BS', the process runs without issue and does not crash. Detection of P2P traffic takes (unscientifically) a second-ish longer, but seems to be as effective as Hyperscan. If I set to 'AC-KS' the process errors out within minutes with the same 'Hyperscan returned fatal error' that the 'Auto' and 'Hyperscan' settings do. I poked around a little bit but couldn't find much on the differences between AC and AC-BS, or which is preferred for our situation until this gets resolved, but it's working for now. I currently have two LANs set to 'AC' and two on 'AC-BS' and can't seem to tell the difference. Thanks. -
@PalisadesTahoe said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
@bmeeks
More testing results. I've determined that if set the Pattern Matcher Algorithm to either 'AC' or 'AC-BS', the process runs without issue and does not crash. Detection of P2P traffic takes (unscientifically) a second-ish longer, but seems to be as effective as Hyperscan. If I set to 'AC-KS' the process errors out within minutes with the same 'Hyperscan returned fatal error' that the 'Auto' and 'Hyperscan' settings do. I poked around a little bit but couldn't find much on the differences between AC and AC-BS, or which is preferred for our situation until this gets resolved, but it's working for now. I currently have two LANs set to 'AC' and two on 'AC-BS' and can't seem to tell the difference. Thanks.Thank you for this feedback. This bit of info (using AC or AC-BS versus Hyperscan for the pattern matcher setting) probably needs to go back upstream. Might be something particular to FreeBSD, though. For example, I believe there is now a 5.4.1 hyperscan library version out there for other distros, but FreeBSD has (or at least had the last time I checked) only 5.4.0 in FreeBSD-ports.
-
@PalisadesTahoe Thanks for posting your detailed troubleshooting. I just tried AC in my environment and I am also having success getting blocking mode enabled again.
-
@xpxp2002 Spoke too soon. Came back to find my box running 80% CPU (four cores) and 100% mem usage and about 50% swap usage. Stopped all of the Suricata processes, and CPU and memory started recovering right away.
I have yet to see the Suricata 7.0.2 package show up in Package Manager. For now, I'm just going to run without Suricata. It seems to be ok with blocking mode disabled, but it's just too risky to keep trying these types of changes with this version and potentially have the system run out of memory and drop offline.
-
@xpxp2002 said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
I have yet to see the Suricata 7.0.2 package show up in Package Manager.
What version of pfSense are you running? I expected the new binary to be deployed to all pfSense versions by now.
Also, just to note, one of the issues with some of the non-hyperscan pattern matchers is they can become huge memory hogs. Same thing happens in Snort. The default choice is always the best choice. But I know in this case it will choose Hyperscan (as that library is available and the best and fastest) and thus trigger the Hyperscan bug.
-
@bmeeks 23.09
-
I've just sent an email to my Netgate developer contact asking that the 7.0.2 package be deployed to the pfSense Plus 23.09 branch. It was supposed to have already been deployed, so I'm not sure what held it up. Maybe some confusion around the potential Kill States bug. But that bug has no relation to the Hyperscan issue. Those are two separate bugs with separate fixes. The Kill States bug has still not been fully identified, but when it is a separate package update will be issued containing that fix.
-
Got a reply from Netgate. They are currently having issues with the pfSense Plus 23.09 package builder infrastructure. That has held up building and deploying several other package updates. Resolution is in progress, and hopefully everything is back on track later today.
Until that issue is resolved, the Suricata 7.0.2 update can't get built for 23.09
. -
@bmeeks That explains it. I'm hoping that this is the fixed Hyperscan bug that I'm encountering, so hopefully it will be resolved once the build is available. I'll schedule time to test it again tonight if the package is available by then.
-
@xpxp2002 said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
@bmeeks That explains it. I'm hoping that this is the fixed Hyperscan bug that I'm encountering, so hopefully it will be resolved once the build is available. I'll schedule time to test it again tonight if the package is available by then.
Yes, the 7.0.2 package should correct the Hyperscan fatal exit error. Here is the upstream commit (change) that fixed it: https://github.com/OISF/suricata/commit/00e00254eae205bad5d4cfbf6c9e69f944faaf69. It was fixed starting with Suricata 7.0.1 from upstream. Currently you have 7.0.0, but the update that is held up from building at the moment is 7.0.2.
This post of mine in another thread explains the Hyperscan error and the fix: https://forum.netgate.com/topic/184101/suricata-process-dying-due-to-hyperscan-problem/15?_=1700148686823.
-
@bmeeks Where to find HyperScan? Is it a kind of rule or tool used by some rules?
-
@Bob-Dig said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
@bmeeks Where to find HyperScan? Is it a kind of rule or tool used by some rules?
Hyperscan is a library that provides high-speed regex pattern matching capability. It is developed and sponsored by Intel. Have fun learning about it here: https://github.com/intel/hyperscan
. Additional info on Hyperscan can be found here: https://www.intel.com/content/www/us/en/developer/articles/technical/introduction-to-hyperscan.html.It is a technology that Suricata incorporates as a shared-library build and runtime dependency. It only works on Intel platforms (meaning you must have an AMD64 CPU). It does not work for ARM-based hardware, and thus is not compiled into nor enabled in Suricata binary builds on those platforms.
-
@bmeeks Thanks. Just wondering why I don't see those problems but I am also not watching the Suricata logfiles. Right now, there is nothing with "Hyperscan" in it.
-
@Bob-Dig said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
@bmeeks Thanks. Just wondering why I don't see those problems but I am also not watching the Suricata logfiles. Right now, there is nothing with "Hyperscan" in it.
You may not have any rules enabled that produce a particular pattern matcher sequence that trips up Hyperscan. Whether or not you see the Hyperscan bug is dependent upon the exact rules you have enabled.
It is a complicated technology. It attempts to precompile certain types of frequently used/accessed regex patterns into a memory database. That way, as rules attempt to match patterns in network packets' data, the whole process can be faster and more efficient.
But in the 5.4.0 version of Hyperscan, Intel made some changes designed to make things even faster; but those changes altered significantly how Suricata and other consumers of Hyperscan needed to function. The result was a sort of regression bug. That's what folks are seeing in 7.0.0 versions of the Suricata binary on Intel hardware when using Hyperscan. Suricata upstream made some corrections in their code starting with the 7.0.1 version.
-
@bmeeks glad I found this chain. I'm on 2.7.1 (having gone through the 3 RCs). Pattern Matching set to 'Auto' fails (now with a silent error - log shows sucess and no errors but icon goes straight to not running now) - previous RC version allowed it to run for 5 mins plus and then error per above. I'm on Physical (no hypervisor). My workaround for now is put it to 'AC' and all is good... - def still an issues with Hyperscan
-
@jonatremoteeyes said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
@bmeeks glad I found this chain. I'm on 2.7.1 (having gone through the 3 RCs). Pattern Matching set to 'Auto' fails (now with a silent error - log shows sucess and no errors but icon goes straight to not running now) - previous RC version allowed it to run for 5 mins plus and then error per above. I'm on Physical (no hypervisor). My workaround for now is put it to 'AC' and all is good... - def still an issues with Hyperscan
Yes, now that I've found and fixed the Signal 11 crash bug I will start looking into the HyperScan issue in depth. Been in contact with the upstream Suricata folks about it and they are not sure what might be going on.
Very likely it's something peculiar to FreeBSD, or perhaps I left out some specific configuration required with the new Suricata binary. Still investigating.
-
Noticed this morning that Suricata 7.0.2 was now available in the packages repository. I've upgraded and switched one of my LANs back to using Hyperscan. Although it seemed to run an little bit longer before crashing, it did eventually do so with the same error: "Hyperscan returned fatal error". Not sure if we were expecting Hyperscan to also be updated, but it is still at 5.4.0, which is odd since 5.4.2 has been out since 2023-04-19.
-
@PalisadesTahoe said in After upgrade to pf+ 23.09 Surricata says it's starting but..:
Noticed this morning that Suricata 7.0.2 was now available in the packages repository. I've upgraded and switched one of my LANs back to using Hyperscan. Although it seemed to run an little bit longer before crashing, it did eventually do so with the same error: "Hyperscan returned fatal error". Not sure if we were expecting Hyperscan to also be updated, but it is still at 5.4.0, which is odd since 5.4.2 has been out since 2023-04-19.
No, no change in the HyperScan library yet. I need to first see if I can reproduce the problem. The upstream Suricata team says 5.4.0 should be okay, but that definitely 5.4.1 is broken for Suricata. The fact 5.4.0 suddenly is giving issues is puzzling to the upstream guys, too.
And just to keep things clear-- there are currently two reported issues with Suricata, and they are NOT related.
- One is the issue with a Signal 11 fault when Legacy Blocking Mode is enabled with the Kill States option checked. That bug has been hopefully identified and fixed. Some new binaries will appear soon reflecting that fix. I believe some posts in this thread are actually a result of that bug and not necessarily the HyperScan one.
- The second bug appears to revolve around the Intel HyperScan library. That one is now under investigation. I initially thought 7.0.2 would take care of that, but it apparently has not. So, now I will see about replicating the issue so a fix can be identified for it. This one may take longer to find and fix, and so is likely not to be part of the upcoming package update correcting the Signal 11 fault.
