Suricata process dying due to hyperscan problem
-
To be clear, all the errors I posted are from the suricata.log file, not from the system. Does that change things?
I've tried uninstalling, rebooting, and then installing again. I'm still having the problem.
-
@tim_co said in Suricata process dying due to hyperscan problem:
[16-Nov-2023 12:49:03 America/Denver] PHP Fatal error: Uncaught TypeError: fwrite(): Argument #1 ($stream) must be of type resource, bool given in /etc/inc/config.lib.inc:172
Stack trace:
#0 /etc/inc/config.lib.inc(172): fwrite(false, 'a:42:{s:7:"vers...')
#1 /etc/inc/config.lib.inc(147): generate_config_cache(Array)
#2 /etc/inc/config.inc(141): parse_config()
#3 /etc/inc/gwlb.inc(25): require_once('/etc/inc/config...')
#4 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...')
#5 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#6 /usr/local/pkg/nut/nut_email.php(24): require_once('/etc/inc/notice...')
#7 {main}
thrown in /etc/inc/config.lib.inc on line 172
[16-Nov-2023 12:49:03 America/Denver] PHP Fatal error: Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:101
Stack trace:
#0 /etc/inc/notices.inc(101): fopen('', 'w')
#1 /etc/inc/config.lib.inc(1148): file_notice('phperror', 'PHP ERROR: Type...', 'PHP errors')
#2 [internal function]: pfSense_clear_globals()
#3 {main}
thrown in /etc/inc/notices.inc on line 101These are the errors I am talking about. No way on Earth those came from Suricata. Those came from pfSense itself, and to me point to a problem with your configuration restore job.
-
For the record, my problems with Suricata started as soon as I upgraded to 7.0.2. I only ran the restore after i was unable to get Suricata to start and work properly.
I went through and deleted an old backup gateway I had configured or LTE backup and removed the nut package. These seemed to be contributing to the system errors.
I uninstalled Suricata, rebooted, and there was no longer a kernel error. I reinstalled Suricata, ran an update, and now it stays running but it's not detecting anything. Typically there's an alert/block at least every few minutes.
The only errors I see now in the system log are related to Suricata not being able to parse thing files and something about [100107] <Error> -- ja3(s) support is not enabled
Any more ideas?
Thanks - Tim
-
If I remove all Suricata rules, the only error I get is the "[100404 - Suricata-Main] 2023-11-16 17:17:22 Notice: threads: Threads created -> RX: 1 W: 4 FM: 1 FR: 1 Engine started.
[100759 - W#04] 2023-11-16 17:17:27 Error: spm-hs: Hyperscan returned fatal error -1."Tim
-
@tim_co said in Suricata process dying due to hyperscan problem:
The only errors I see now in the system log are related to Suricata not being able to parse thing files and something about [100107] <Error> -- ja3(s) support is not enabled
That error is harmless for now. It simply means that particular protocol parser is not enabled. It can be ignored.
@tim_co said in Suricata process dying due to hyperscan problem:
If I remove all Suricata rules, the only error I get is the "[100404 - Suricata-Main] 2023-11-16 17:17:22 Notice: threads: Threads created -> RX: 1 W: 4 FM: 1 FR: 1 Engine started.
[100759 - W#04] 2023-11-16 17:17:27 Error: spm-hs: Hyperscan returned fatal error -1."You have absolutely zero rule categories enabled and you are getting that Hyperscan error? That does not seem to be the same bug because rules with regex patterns to be compiled are required to trigger the Hyperscan bug discussed in this thread. The error occurs when the library is actively compiling regex expressions pulled from enabled rules.
-
Suricata has been rock solid for years. Its a remarkable tool. It worked great until I upgraded to 7.0.2 today.
I'm just trying to get Suricats running again by providing data that seems like it might be helpful with troubleshooting the bug and/or my environment. I clearly don't know what I'm doing.
Tim
-
Same issue here, Suricata was fine until update to 7.0.2 (HS 5.4.0), after update it was not starting. Changed Pattern Matcher Algorithm from Auto to AC and its working again. However have 2 other boxes with the 1:1 same Suricata configuration, but they start just fine, only the hardware is different there.
-
@Bismarck said in Suricata process dying due to hyperscan problem:
Same issue here, Suricata was fine until update to 7.0.2 (HS 5.4.0), after update it was not starting. Changed Pattern Matcher Algorithm from Auto to AC and its working again. However have 2 other boxes with the 1:1 same Suricata configuration, but they start just fine, only the hardware is different there.
Can you elaborate on the hardware or other platform differences? It would be helpful to me to know that.
- What processor types are in the other "working" machines?
- Are they bare metal or virtual ?
- What pfSense versions are on the other "working" machines?
- I assume "yes", but are all of the machines ("working" and "not working") running the same Suricata package version?
-
- Intel(R) Xeon(R) CPU E5-2667 and Intel(R) Core(TM) i5-4460 working, Intel(R) Celeron(R) N5105 is not.
- Yes, all bare metal
- All at 2.7.0-RELEASE (amd64)
- Yes all the same version, same config/settings/rules/sid etc.
-
@Bismarck said in Suricata process dying due to hyperscan problem:
- Intel(R) Xeon(R) CPU E5-2667 and Intel(R) Core(TM) i5-4460 working, Intel(R) Celeron(R) N5105 is not.
- Yes, all bare metal
- All at 2.7.0-RELEASE (amd64)
- Yes all the same version, same config/settings/rules/sid etc.
Thanks for the details. Curious the Celeron is not working. I do know that Hyperscan uses some specific Intel CPU instructions, that's why it does not work on ARM or other hardware platforms. But I would think Celeron would be okay.
At any rate, I am going to try and bundle an update to the Hyperscan 5.4.2 library in my next Suricata update.
-
@bmeeks said in Suricata process dying due to hyperscan problem:
At any rate, I am going to try and bundle an update to the Hyperscan 5.4.2 library in my next Suricata update.
I would be grateful to test the new version.
-
Changed mine to AC-KS and it immediately started working again, but it's putting the hurt on the CPU, temp, & memory. Time to zip-tie the fan to the fan-less system once again. Thanks for the tip!
Tim
-
@Bismarck said in Suricata process dying due to hyperscan problem:
@bmeeks said in Suricata process dying due to hyperscan problem:
At any rate, I am going to try and bundle an update to the Hyperscan 5.4.2 library in my next Suricata update.
I would be grateful to test the new version.
Unfortunately, I don't have a way to distribute it.
-
Nevermind. AC-KS also puked. AC has been running fine for a few minutes now.
-
After my device runs for half an hour or so, all of the descriptions for alerts say "SURICATA QUIC error on data". Google doesn't find anything for the string. Would somebody please explain to me what this means?
Edit - All the traffic is from the same IP address which is coming from "Zscaler, Inc." and is UDP Generic Protocol Command Decode 165.225.10.58 443 X,X,X,X 34476 1:2231001 SURICATA QUIC error on data
Thanks - Tim
-
@tim_co said in Suricata process dying due to hyperscan problem:
After my device runs for half an hour or so, all of the descriptions for alerts say "SURICATA QUIC error on data". Google doesn't find anything for the string. Would somebody please explain to me what this means?
Thanks - Tim
Tim:
QUIC is a new web transport protocol based on UDP instead of TCP. Some basic info can be found here: https://en.wikipedia.org/wiki/QUIC.You are seeing those messages because the Suricata built-in Events Rules are enabled. Those are meant to be "informational" rules. They just tell you some particular traffic type is present. That does not mean the traffic is harmful. There is a lot of QUIC out there now, and Suricata is just saying "I see it!". The QUIC rules and detection logic were added to the 7.x branch of Suricata by its upstream developers.
I recommend you go to the CATEGORIES tab and turn off nearly all of the Suricata built-in events rules. Or switch to Block on DROPs Only mode and use SID MGMT tab features to selectively set only some rules or rule categories to DROP (block) traffic and leave the others at their default of ALERT. That will prevent any nuiscan blocks from the information Events Rules. Search the forum here and you will find several posts, including some Sticky Posts at the top of this sub-forum, describing how to do this.
-
Cool. Thanks for the information. It is greatly appreciated.
-
T tim_co referenced this topic on
-
@Bismarck said in Suricata process dying due to hyperscan problem:
- Intel(R) Xeon(R) CPU E5-2667 and Intel(R) Core(TM) i5-4460 working, Intel(R) Celeron(R) N5105 is not.
I’m running a N5105, and it’s running on mine with no problem. It was yesterday on 2.7.0 and overnight on 2.7.1
-
@Vollans said in Suricata process dying due to hyperscan problem:
Intel(R) Celeron(R) N5105
The only difference is the machine with the Intel N5105 CPU also has Intel NICs (igc), the othere 2 have broadcoms (bge). I guess thats why they behave different.
Do we see a pattern here?
-
@Bismarck said in Suricata process dying due to hyperscan problem:
@Vollans said in Suricata process dying due to hyperscan problem:
Intel(R) Celeron(R) N5105
The only difference is the machine with the Intel N5105 CPU also has Intel NICs (igc), the othere 2 have broadcoms (bge). I guess thats why they behave different.
Do we see a pattern here?
I can't see any scenario where the type of NIC hardware would have anything to do with Hyperscan. That technology is purely a thing within the CPU. It pre-compiles regex patterns for faster matching using special CPU opcodes that only Intel CPUs have. That's why Hyperscan does not work on non-Intel processors.
