Suricata process dying due to hyperscan problem

tim_co

Let me preface by saying I know just enough about pfSense to be dangerous and I'm barely fluent in Unix, but I'm fairly certain I'm having the same problem that's being discussed here and it's criplling the security of my network.

Here's my system info
Version 2.7.0-RELEASE (amd64)
built on Wed Jun 28 03:53:34 UTC 2023
FreeBSD 14.0-CURRENT
CPU Type Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
Current: 3114 MHz, Max: 2700 MHz
4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
AES-NI CPU Crypto: Yes (active)
QAT Crypto: No
Hardware crypto AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS
Kernel PTI Enabled
MDS Mitigation VERW

Before I made any of the changes below I ran a backup.

I updated the Suricata package to 7.0.2 this along with System_Patches 2.2.7_2 and openvpn-client-export 1.9.2. As soon as the package updates had completed, I noticed that Suricata did not restart. I forced it to restart with the GUI and it started but wasn't detecting anything. Within a few minutes, Suricata crashed again. I poked at it a few more minutes and decided I was just going to

There wasn't much meaningful information in the Suricata.log except for "[103312 - W#04] 2023-11-16 13:04:45 Error: spm-hs: Hyperscan returned fatal error -1."

So I did a system restore and much to my surprise Suricata was still on 7.0.2 afterwards.

I tried all the normal stuff like forcing updates, restarting the service, etc. I couldn't keep Suricata running long enough to actually block anything.

I then decided to try and remove the System_Patches. As soon as it was removed, Suricata started back up again but pfSense threw a kernel error:

[16-Nov-2023 12:49:03 America/Denver] PHP Fatal error: Uncaught TypeError: fwrite(): Argument #1 ($stream) must be of type resource, bool given in /etc/inc/config.lib.inc:172
Stack trace:
#0 /etc/inc/config.lib.inc(172): fwrite(false, 'a:42:{s:7:"vers...')
#1 /etc/inc/config.lib.inc(147): generate_config_cache(Array)
#2 /etc/inc/config.inc(141): parse_config()
#3 /etc/inc/gwlb.inc(25): require_once('/etc/inc/config...')
#4 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...')
#5 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#6 /usr/local/pkg/nut/nut_email.php(24): require_once('/etc/inc/notice...')
#7 {main}
thrown in /etc/inc/config.lib.inc on line 172
[16-Nov-2023 12:49:03 America/Denver] PHP Fatal error: Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:101
Stack trace:
#0 /etc/inc/notices.inc(101): fopen('', 'w')
#1 /etc/inc/config.lib.inc(1148): file_notice('phperror', 'PHP ERROR: Type...', 'PHP errors')
#2 [internal function]: pfSense_clear_globals()
#3 {main}
thrown in /etc/inc/notices.inc on line 101

Since then I was able to get Suricata to run log enough to Alert/Block a few things but it's still trying to crash. I installed the watchdog package to keep Suricata running and it seems to be blocking things again but most of the alerts now end with "SURICATA QUIC error on data"

Also, when I go into the Service_Watchdog settings, I can select the option to monitor the Suricata service, but there's no option the save the settings and every time I refresh the page Suricate is de-selected.

I tried reinstalling System_Patches and everything continued to run for a few minutes before crashing again. I've since removed System_Patches again and also removed/reinstalled the service_watchdog to no avail.

I hope some of this information is helpful and that this group could please help me get my pfSense happy again.

I'm a retired Windows Admin & IT Security nerd turned stay at home Dad so I apologize is this information is a mess. Also, if it matters, I'm running Suricata in my private home. Mostly for my wife who frequently works remotely from here and to keep my two kids from infecting everything on the network.

Thanks - Tim in Colorado

bmeeks

@tim_co:
Yes, this looks exactly like the Hyperscan bug present in Suricata binary version 7.0.0.

First, DO NOT monitor Suricata with Service Watchdog. The Watchdog package has no idea how to properly monitor Suricata for functionality and will needlessly issue "restart" commands to Suricata even when Suricata is automatically restarting itself from certain actions such as updating rules. Immediately remove Suricata from the list of monitored processes in Service Watchdog and never ever add Suricata there again.

What version of the Suricata binary is actually on your system? I suspect it is 7.0.0. You can tell by examining the suricata.log file for an interface under LOGS VIEW. The very first line logged will give the current Suricata version.

Your errors in italics are not related to Suricata at all. Those are some kind of restore or install error, and indicate something is wrong with some critical system files.

@tim_co said in Suricata process dying due to hyperscan problem:

Also, when I go into the Service_Watchdog settings, I can select the option to monitor the Suricata service, but there's no option the save the settings and every time I refresh the page Suricate is de-selected.

This is actually a good thing. Maybe the package was modified to prevent users from shooting themselves in the foot by monitoring the Snort or Suricata packages with Service Watchdog. Never put any of the IDS/IPS packages under Service Watchdog. I've been preaching here on the forum for several years that users should never use Service Watchdog to monitor and auto-restart the IDS/IPS packages.

tim_co

@bmeeks said in Suricata process dying due to hyperscan problem:

uricata

The version info is below

[100381 - Suricata-Main] 2023-11-16 15:44:30 Notice: suricata: This is Suricata version 7.0.2 RELEASE running in SYSTEM mode

It looks like 7.0.2?

What do I do from here?

EDIT - And I removed the service_watchdog

Thanks,
Tim

tim_co

If it's helpful, here's the whole suricata.log

If it's helpful, the normal log file starts showing dozens of errors like this:

[100124 - Suricata-Main] 2023-11-16 15:57:19 Error: detect-tls-ja3-hash: ja3 support is not enabled
[100124 - Suricata-Main] 2023-11-16 15:57:19 Error: detect: error parsing signature "alert tls $HOME_NET any -> $EXTERNAL_NET any (msg:"ET JA3 Hash - Suspected Cobalt Strike Malleable C2 M1 (set)"; flow:established,to_server; ja3.hash; content:"eb88d0b3e1961a0562f006e5ce2a0b87"; ja3.string; content:"771,49192-49191-49172-49171"; flowbits:set,ET.cobaltstrike.ja3; flowbits:noalert; classtype:command-and-control; sid:2028831; rev:1; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, created_at 2019_10_15, deployment Perimeter, former_category JA3, malware_family Cobalt_Strike, confidence Low, signature_severity Major, updated_at 2019_10_15, mitre_tactic_id TA0011, mitre_tactic_name Command_And_Control, mitre_technique_id T1001, mitre_technique_name Data_Obfuscation;)" from file /usr/local/etc/suricata/suricata_32509_igb0/rules/suricata.rules at line 10762
[100124 - Suricata-Main] 2023-11-16 15:57:19 Error: detect-tls-ja3s-hash: ja3(s) support is not enabled
[100124 - Suricata-Main] 2023-11-16 15:57:19 Error: detect: error parsing signature "alert tls $EXTERNAL_NET any -> $HOME_NET any (msg:"ET JA3 HASH - Possible RustyBuer Server Response"; flowbits:isset,ET.rustybuer; ja3s.hash; content:"f6dfdd25d1522e4e1c7cd09bd37ce619"; reference:md5,ea98a9d6ca6f5b2a0820303a1d327593; classtype:bad-unknown; sid:2032960; rev:1; metadata:attack_target Client_Endpoint, created_at 2021_05_13, deployment Perimeter, former_category JA3, malware_family RustyBuer, performance_impact Low, confidence Low, signature_severity Major, updated_at 2021_05_13;)" from file /usr/local/etc/suricata/suricata_32509_igb0/rules/suricata.rules at line 10858
[100124 - Suricata-Main] 2023-11-16 15:57:32 Error: detect: previous sticky buffer has no matches
[100124 - Suricata-Main] 2023-11-16 15:57:32 Error: detect: error parsing signature "alert tcp $EXTERNAL_NET $FILE_DATA_PORTS -> $HOME_NET any (msg:"BROWSER-WEBKIT Apple Safari WebKit out-of-bounds write attempt"; flow:to_client,established; file_data; file_data; content:"try { (function () { let a = { get val() { [...{a = 1.45}] = []|3B| a.val.x|3B| }, }|3B| a.val|3B| })()|3B| } catch (e) { } </script>"; fast_pattern:only; metadata:policy balanced-ips drop, policy max-detect-ips drop, policy security-ips drop, service ftp-data, service http, service imap, service pop3; reference:cve,2017-2505; reference:url,bugs.chromium.org/p/project-zero/issues/detail?id=1137; classtype:attempted-user; sid:51391; rev:1;)" from file /usr/local/etc/suricata/suricata_32509_igb0/rules/suricata.rules at line 34646
[100124 - Suricata-Main] 2023-11-16 15:57:33 Error: detect-parse: "http_header" keyword seen with a sticky buffer still set. Reset sticky buffer with pkt_data before using the modifier.
[100124 - Suricata-Main] 2023-11-16 15:57:33 Error: detect: error parsing signature "alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"INDICATOR-COMPROMISE Potential malware download - .pdf.exe within .zip file"; flow:to_client,established; flowbits:isset,file.zip; file_data; content:".pdf.exe"; fast_pattern:only; content:"Content-Length:"; http_header; metadata:policy max-detect-ips drop, policy security-ips drop, ruleset community, service http; classtype:trojan-activity; sid:31001; rev:2;)" from file /usr/local/etc/suricata/suricata_32509_igb0/rules/suricata.rules at line 36294
[100124 - Suricata-Main] 2023-11-16 15:57:33 Error: detect-parse: "http_header" keyword seen with a sticky buffer still set. Reset sticky buffer with pkt_data before using the modifier.
[100124 - Suricata-Main] 2023-11-16 15:57:33 Error: detect: error parsing signature "alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"INDICATOR-COMPROMISE Potential malware download - .jpg.exe within .zip file"; flow:to_client,established; flowbits:isset,file.zip; file_data; content:".jpg.exe"; fast_pattern:only; content:"Content-Length:"; http_header; metadata:policy max-detect-ips drop, policy security-ips drop, ruleset community, service http; classtype:trojan-activity; sid:31000; rev:2;)" from file /usr/local/etc/suricata/suricata_32509_igb0/rules/suricata.rules at line 36295
[100124 - Suricata-Main] 2023-11-16 15:57:33 Error: detect-parse: "http_header" keyword seen with a sticky buffer still set. Reset sticky buffer with pkt_data before using the modifier.
[100124 - Suricata-Main] 2023-11-16 15:57:33 Error: detect: error parsing signature "alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"INDICATOR-COMPROMISE Potential malware download - .jpeg.exe within .zip file"; flow:to_client,established; flowbits:isset,file.zip; file_data; content:".jpeg.exe"; fast_pattern:only; content:"Content-Length:"; http_header; metadata:policy max-detect-ips drop, policy security-ips drop, ruleset community, service http; classtype:trojan-activity; sid:30999; rev:2;)" from file /usr/local/etc/suricata/suricata_32509_igb0/rules/suricata.rules at line 36296

bmeeks

@tim_co said in Suricata process dying due to hyperscan problem:

And I removed the service_watchdog

That step is good!

Those other errors are unrelated to Suricata, but they point to something getting quite messed up on your system. Restoring a config will not necessarily restore the previous package binary versions. In fact, that can make things worse because new binary versions need certain changes made in the configuration. Suricata and Snort are like this.

First thing I would do is uninstall Suricata using the option under SYSTEM > PACKAGE MANAGER.
Next, reboot the firewall so everything has a fresh clean start.
If you have errors upon the restart, then those need to get fixed before messing around any further with Suricata.
Once the firewall boots cleanly with no errors, then return to SYSTEM > PACKAGE MANAGER and reinstall Suricata.

But to be perfectly frank with you, if you are using Suricata on a home network, there is really no valued added at all. Close to 100% of network traffic these days is encrypted, and unless you have a full proxy configured for MITM, Suricata is not examining the vast majority of data flowing through your firewall. It is just seeing random encrypted bytes that it has no way to decipher and scan. All it can do is examine the source and destination IP addresses and ports, and maybe catch just a glimpse of SNI data for some traffic. It can't see a single thing in SSL web traffic nor email as that is also encrypted with TLS.

tim_co

@bmeeks

To be clear, all the errors I posted are from the suricata.log file, not from the system. Does that change things?

I've tried uninstalling, rebooting, and then installing again. I'm still having the problem.

bmeeks

@tim_co said in Suricata process dying due to hyperscan problem:

[16-Nov-2023 12:49:03 America/Denver] PHP Fatal error: Uncaught TypeError: fwrite(): Argument #1 ($stream) must be of type resource, bool given in /etc/inc/config.lib.inc:172
Stack trace:
#0 /etc/inc/config.lib.inc(172): fwrite(false, 'a:42:{s:7:"vers...')
#1 /etc/inc/config.lib.inc(147): generate_config_cache(Array)
#2 /etc/inc/config.inc(141): parse_config()
#3 /etc/inc/gwlb.inc(25): require_once('/etc/inc/config...')
#4 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...')
#5 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#6 /usr/local/pkg/nut/nut_email.php(24): require_once('/etc/inc/notice...')
#7 {main}
thrown in /etc/inc/config.lib.inc on line 172
[16-Nov-2023 12:49:03 America/Denver] PHP Fatal error: Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:101
Stack trace:
#0 /etc/inc/notices.inc(101): fopen('', 'w')
#1 /etc/inc/config.lib.inc(1148): file_notice('phperror', 'PHP ERROR: Type...', 'PHP errors')
#2 [internal function]: pfSense_clear_globals()
#3 {main}
thrown in /etc/inc/notices.inc on line 101

These are the errors I am talking about. No way on Earth those came from Suricata. Those came from pfSense itself, and to me point to a problem with your configuration restore job.

tim_co

@bmeeks

For the record, my problems with Suricata started as soon as I upgraded to 7.0.2. I only ran the restore after i was unable to get Suricata to start and work properly.

I went through and deleted an old backup gateway I had configured or LTE backup and removed the nut package. These seemed to be contributing to the system errors.

I uninstalled Suricata, rebooted, and there was no longer a kernel error. I reinstalled Suricata, ran an update, and now it stays running but it's not detecting anything. Typically there's an alert/block at least every few minutes.

The only errors I see now in the system log are related to Suricata not being able to parse thing files and something about [100107] <Error> -- ja3(s) support is not enabled

Any more ideas?

Thanks - Tim

tim_co

@bmeeks

If I remove all Suricata rules, the only error I get is the "[100404 - Suricata-Main] 2023-11-16 17:17:22 Notice: threads: Threads created -> RX: 1 W: 4 FM: 1 FR: 1 Engine started.
[100759 - W#04] 2023-11-16 17:17:27 Error: spm-hs: Hyperscan returned fatal error -1."

Tim

bmeeks

@tim_co said in Suricata process dying due to hyperscan problem:

The only errors I see now in the system log are related to Suricata not being able to parse thing files and something about [100107] <Error> -- ja3(s) support is not enabled

That error is harmless for now. It simply means that particular protocol parser is not enabled. It can be ignored.

@tim_co said in Suricata process dying due to hyperscan problem:

If I remove all Suricata rules, the only error I get is the "[100404 - Suricata-Main] 2023-11-16 17:17:22 Notice: threads: Threads created -> RX: 1 W: 4 FM: 1 FR: 1 Engine started.
[100759 - W#04] 2023-11-16 17:17:27 Error: spm-hs: Hyperscan returned fatal error -1."

You have absolutely zero rule categories enabled and you are getting that Hyperscan error? That does not seem to be the same bug because rules with regex patterns to be compiled are required to trigger the Hyperscan bug discussed in this thread. The error occurs when the library is actively compiling regex expressions pulled from enabled rules.

tim_co

Suricata has been rock solid for years. Its a remarkable tool. It worked great until I upgraded to 7.0.2 today.

I'm just trying to get Suricats running again by providing data that seems like it might be helpful with troubleshooting the bug and/or my environment. I clearly don't know what I'm doing.

Tim

Bismarck

@bmeeks

Same issue here, Suricata was fine until update to 7.0.2 (HS 5.4.0), after update it was not starting. Changed Pattern Matcher Algorithm from Auto to AC and its working again. However have 2 other boxes with the 1:1 same Suricata configuration, but they start just fine, only the hardware is different there.

bmeeks

@Bismarck said in Suricata process dying due to hyperscan problem:

@bmeeks

Same issue here, Suricata was fine until update to 7.0.2 (HS 5.4.0), after update it was not starting. Changed Pattern Matcher Algorithm from Auto to AC and its working again. However have 2 other boxes with the 1:1 same Suricata configuration, but they start just fine, only the hardware is different there.

Can you elaborate on the hardware or other platform differences? It would be helpful to me to know that.

What processor types are in the other "working" machines?
Are they bare metal or virtual ?
What pfSense versions are on the other "working" machines?
I assume "yes", but are all of the machines ("working" and "not working") running the same Suricata package version?

Bismarck

@bmeeks

Intel(R) Xeon(R) CPU E5-2667 and Intel(R) Core(TM) i5-4460 working, Intel(R) Celeron(R) N5105 is not.
Yes, all bare metal
All at 2.7.0-RELEASE (amd64)
Yes all the same version, same config/settings/rules/sid etc.

bmeeks

@Bismarck said in Suricata process dying due to hyperscan problem:

@bmeeks

Intel(R) Xeon(R) CPU E5-2667 and Intel(R) Core(TM) i5-4460 working, Intel(R) Celeron(R) N5105 is not.

Yes, all bare metal

All at 2.7.0-RELEASE (amd64)

Yes all the same version, same config/settings/rules/sid etc.

Thanks for the details. Curious the Celeron is not working. I do know that Hyperscan uses some specific Intel CPU instructions, that's why it does not work on ARM or other hardware platforms. But I would think Celeron would be okay.

At any rate, I am going to try and bundle an update to the Hyperscan 5.4.2 library in my next Suricata update.

Bismarck

@bmeeks said in Suricata process dying due to hyperscan problem:

At any rate, I am going to try and bundle an update to the Hyperscan 5.4.2 library in my next Suricata update.

I would be grateful to test the new version.

tim_co

@Bismarck

Changed mine to AC-KS and it immediately started working again, but it's putting the hurt on the CPU, temp, & memory. Time to zip-tie the fan to the fan-less system once again. Thanks for the tip!

Tim

bmeeks

@Bismarck said in Suricata process dying due to hyperscan problem:

@bmeeks said in Suricata process dying due to hyperscan problem:

At any rate, I am going to try and bundle an update to the Hyperscan 5.4.2 library in my next Suricata update.

I would be grateful to test the new version.

Unfortunately, I don't have a way to distribute it.

tim_co

@Bismarck

Nevermind. AC-KS also puked. AC has been running fine for a few minutes now.

tim_co

After my device runs for half an hour or so, all of the descriptions for alerts say "SURICATA QUIC error on data". Google doesn't find anything for the string. Would somebody please explain to me what this means?

Edit - All the traffic is from the same IP address which is coming from "Zscaler, Inc." and is UDP Generic Protocol Command Decode 165.225.10.58 443 X,X,X,X 34476 1:2231001 SURICATA QUIC error on data

Thanks - Tim