Suricata process dying due to hyperscan problem

bmeeks

@asdjklfjkdslfdsaklj said in Suricata process dying due to hyperscan problem:

@bmeeks off topic, but is there a way for the unwashed masses to see build and readiness of packages, or something beyond PRs?

No, not that I am aware of. At least nothing that is likely to be very useful. Everything for the CE versions is posted on GitHub. Here is the link to all the FreeBSD-ports in pfSense: https://github.com/pfsense/FreeBSD-ports

But the way the ports are packaged can be quite confusing to a first-time user.

tylerevers

@bmeeks would you be willing to help me understand how this process works, briefly? If it gets into the builder at a specific time, roughly when might it appear downstream in System / Package Manager? Thank you!

bmeeks

@tylerevers said in Suricata process dying due to hyperscan problem:

@bmeeks would you be willing to help me understand how this process works, briefly? If it gets into the builder at a specific time, roughly when might it appear downstream in System / Package Manager? Thank you!

I will do my best, but understand I am not a Netgate employee nor do I, nor can I, speak for them.

The entire FreeBSD-ports repo for pfSense is based on the official upstream Ports repo of FreeBSD. The pfSense fork of that repo by and large just contains the same code as upstream, but some ports (packages are really the same as ports) have changes specific to pfSense. The Snort and Suricata packages are two examples. There are also a few ports (packages) present in the pfSense fork of FreeBSD-ports that are not present upstream. Some of those are pfBlockerNG and some pfSense-specific ports (pfSense-repoc, pfSense-upgrade, etc.).

Suggested changes are posted to the FreeBSD-ports repo of pfSense by anyone. Usually it's package maintainers. The managers of the repo are Netgate employees/developers. Nothing gets merged into the official repo until it is reviewed/approved by the Netgate team. Sometimes there may be a series of back and forth revisions to pull requests (submitted changes are done as pull requests) as the Netgate developers have questions or comments.

Once a pull request is merged, that means it is officially submitted to the pfSense builder servers where the packages are created. Here is where my info gets a little fuzzy. Different pfSense familes (CE, Plus, and the various DEVEL snapshots) have different package building infrastructure and different building schedules. Some run package build jobs very frequently while others may run a job only every few days or even just manually when kicked off by Netgate package managers. The builder is the compiler/linker part that compiles and links the source code to produce binary executable files and other parts and pieces that comprise a "package". The final step is copying everything into a *.pkg file and compressing it. Once a package builder server finishes building all the packages in the FreeBSD-ports repo, it copies them via rsync to a web server. That web server is what your firewall actually talks to when querying via the SYSTEM > PACKAGE MANAGER menu.

Something important to note here is that the different pfSense versions have their package repositories served up via different URLs (and even servers, I think). Thus it is entirely possible for CE Release, CE Devel, and Plus Release and Plus Devel to all have a different version of a given package. This is not normally the case, but it certainly can occur. In fact, it is very common for the development snapshot branches of pfSense (both CE and Plus) to have the latest package versions. This is to facilitate testing of package changes. How rigorous and productive this development testing is depends upon how many users actually install the updated packages and test them in the snapshots. In truth, that is usually just a tiny number of users.

bmeeks

The more I've studied the Hyperscan issue, the more I'm starting to doubt if the 7.0.2 version will be the complete fix needed.

It appears there were issues in the older hyperscan library. Currently I see Intel has 5.4.2 as current. But FreeBSD is still using 5.4.0. There was a change in 5.4.0 (or maybe earlier) that caused the hyperscan pattern compiler bug happening in Suricata. The same bug was also manifested in Snort according to the upstream Hyperscan team. They committed a change to rollback the problematic code and it was released in the 5.4.1 version of Hyperscan.

So, long story short, it might take getting Hyperscan in FreeBSD itself updated to 5.4.1 or even 5.4.2 to solve the issue. Suricata upstream primarily targets and tests on Linux. They do some limited private testing on FreeBSD, but not with any of the cool automated tools they use on Linux. That's because the cool automated tools are not supported on FreeBSD .

tylerevers

@bmeeks Thank you for assembling your response. I appreciate the depth of your insights. Many of the details regarding diverse processes and procedures were unfamiliar to me, expanding my knowledge significantly. I'm grateful for this learning experience.

tylerevers

@bmeeks said in Suricata process dying due to hyperscan problem:

So, long story short, it might take getting Hyperscan in FreeBSD itself updated to 5.4.1 or even 5.4.2 to solve the issue. Suricata upstream primarily targets and tests on Linux. They do some limited private testing on FreeBSD, but not with any of the cool automated tools they use on Linux. That's because the cool automated tools are not supported on FreeBSD .

If I am interpreting this correctly, it means that there is nothing I can do on my end except wait for the powers that be to (hopefully) update Hyperscan in FreeBSD?

bmeeks

@tylerevers said in Suricata process dying due to hyperscan problem:

@bmeeks said in Suricata process dying due to hyperscan problem:

So, long story short, it might take getting Hyperscan in FreeBSD itself updated to 5.4.1 or even 5.4.2 to solve the issue. Suricata upstream primarily targets and tests on Linux. They do some limited private testing on FreeBSD, but not with any of the cool automated tools they use on Linux. That's because the cool automated tools are not supported on FreeBSD .

If I am interpreting this correctly, it means that there is nothing I can do on my end except wait for the powers that be to (hopefully) update Hyperscan in FreeBSD?

Yes, if I am correct only a Hyperscan library update would fix this completely and remove all errors. However, there are two levels of error happening. One fatal and the other more like a notice.

The first problem, and the more major one, was occurring with the Suricata 7.0.0 binary (which you have unless the latest 7.0.2 update has shown up for your machines now). In the 7.0.0 binary the Hyperscan failure to compile a pattern in the pattern matcher produced a Suricata Fatal Error. Fatal Errors always cause Suricata to quit and exit the current process.

But starting with Suricata 7.0.1 and later, the Suricata developers modified their code so that a Hyperscan pattern matcher compile error only produces an entry in the suricata.log and does not generate and call the Suricata Fatal Error code. Here is the upstream Suricata commit that made this change: https://github.com/OISF/suricata/commit/00e00254eae205bad5d4cfbf6c9e69f944faaf69. If you look at the branch tags in that commit, you will see it went into 7.0.1 and 7.0.2.

What I think will happen is that once you install the 7.0.2 version of the Suricata package on your machine it should stop "killing" Suricata when a Hyperscan error is detected. Instead, with the new Suricata version, it should just log an error in suricata.log and then keep on running. Of course the rule causing the Hyperscan compile error will not be parsed and loaded, but at least Suricata itself should stay running. My current belief is that once the Hyperscan 5.4.2 or later library is available in FreeBSD-ports and linked to the Suricata package, then that remaining "notice" error will also disappear.

tylerevers

@bmeeks Thank you again for being helpful, succinct, and polite. You have made my day, good sir!

tim_co

Let me preface by saying I know just enough about pfSense to be dangerous and I'm barely fluent in Unix, but I'm fairly certain I'm having the same problem that's being discussed here and it's criplling the security of my network.

Here's my system info
Version 2.7.0-RELEASE (amd64)
built on Wed Jun 28 03:53:34 UTC 2023
FreeBSD 14.0-CURRENT
CPU Type Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
Current: 3114 MHz, Max: 2700 MHz
4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
AES-NI CPU Crypto: Yes (active)
QAT Crypto: No
Hardware crypto AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS
Kernel PTI Enabled
MDS Mitigation VERW

Before I made any of the changes below I ran a backup.

I updated the Suricata package to 7.0.2 this along with System_Patches 2.2.7_2 and openvpn-client-export 1.9.2. As soon as the package updates had completed, I noticed that Suricata did not restart. I forced it to restart with the GUI and it started but wasn't detecting anything. Within a few minutes, Suricata crashed again. I poked at it a few more minutes and decided I was just going to

There wasn't much meaningful information in the Suricata.log except for "[103312 - W#04] 2023-11-16 13:04:45 Error: spm-hs: Hyperscan returned fatal error -1."

So I did a system restore and much to my surprise Suricata was still on 7.0.2 afterwards.

I tried all the normal stuff like forcing updates, restarting the service, etc. I couldn't keep Suricata running long enough to actually block anything.

I then decided to try and remove the System_Patches. As soon as it was removed, Suricata started back up again but pfSense threw a kernel error:

[16-Nov-2023 12:49:03 America/Denver] PHP Fatal error: Uncaught TypeError: fwrite(): Argument #1 ($stream) must be of type resource, bool given in /etc/inc/config.lib.inc:172
Stack trace:
#0 /etc/inc/config.lib.inc(172): fwrite(false, 'a:42:{s:7:"vers...')
#1 /etc/inc/config.lib.inc(147): generate_config_cache(Array)
#2 /etc/inc/config.inc(141): parse_config()
#3 /etc/inc/gwlb.inc(25): require_once('/etc/inc/config...')
#4 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...')
#5 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#6 /usr/local/pkg/nut/nut_email.php(24): require_once('/etc/inc/notice...')
#7 {main}
thrown in /etc/inc/config.lib.inc on line 172
[16-Nov-2023 12:49:03 America/Denver] PHP Fatal error: Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:101
Stack trace:
#0 /etc/inc/notices.inc(101): fopen('', 'w')
#1 /etc/inc/config.lib.inc(1148): file_notice('phperror', 'PHP ERROR: Type...', 'PHP errors')
#2 [internal function]: pfSense_clear_globals()
#3 {main}
thrown in /etc/inc/notices.inc on line 101

Since then I was able to get Suricata to run log enough to Alert/Block a few things but it's still trying to crash. I installed the watchdog package to keep Suricata running and it seems to be blocking things again but most of the alerts now end with "SURICATA QUIC error on data"

Also, when I go into the Service_Watchdog settings, I can select the option to monitor the Suricata service, but there's no option the save the settings and every time I refresh the page Suricate is de-selected.

I tried reinstalling System_Patches and everything continued to run for a few minutes before crashing again. I've since removed System_Patches again and also removed/reinstalled the service_watchdog to no avail.

I hope some of this information is helpful and that this group could please help me get my pfSense happy again.

I'm a retired Windows Admin & IT Security nerd turned stay at home Dad so I apologize is this information is a mess. Also, if it matters, I'm running Suricata in my private home. Mostly for my wife who frequently works remotely from here and to keep my two kids from infecting everything on the network.

Thanks - Tim in Colorado

bmeeks

@tim_co:
Yes, this looks exactly like the Hyperscan bug present in Suricata binary version 7.0.0.

First, DO NOT monitor Suricata with Service Watchdog. The Watchdog package has no idea how to properly monitor Suricata for functionality and will needlessly issue "restart" commands to Suricata even when Suricata is automatically restarting itself from certain actions such as updating rules. Immediately remove Suricata from the list of monitored processes in Service Watchdog and never ever add Suricata there again.

What version of the Suricata binary is actually on your system? I suspect it is 7.0.0. You can tell by examining the suricata.log file for an interface under LOGS VIEW. The very first line logged will give the current Suricata version.

Your errors in italics are not related to Suricata at all. Those are some kind of restore or install error, and indicate something is wrong with some critical system files.

@tim_co said in Suricata process dying due to hyperscan problem:

Also, when I go into the Service_Watchdog settings, I can select the option to monitor the Suricata service, but there's no option the save the settings and every time I refresh the page Suricate is de-selected.

This is actually a good thing. Maybe the package was modified to prevent users from shooting themselves in the foot by monitoring the Snort or Suricata packages with Service Watchdog. Never put any of the IDS/IPS packages under Service Watchdog. I've been preaching here on the forum for several years that users should never use Service Watchdog to monitor and auto-restart the IDS/IPS packages.

tim_co

@bmeeks said in Suricata process dying due to hyperscan problem:

uricata

The version info is below

[100381 - Suricata-Main] 2023-11-16 15:44:30 Notice: suricata: This is Suricata version 7.0.2 RELEASE running in SYSTEM mode

It looks like 7.0.2?

What do I do from here?

EDIT - And I removed the service_watchdog

Thanks,
Tim

tim_co

If it's helpful, here's the whole suricata.log

If it's helpful, the normal log file starts showing dozens of errors like this:

[100124 - Suricata-Main] 2023-11-16 15:57:19 Error: detect-tls-ja3-hash: ja3 support is not enabled
[100124 - Suricata-Main] 2023-11-16 15:57:19 Error: detect: error parsing signature "alert tls $HOME_NET any -> $EXTERNAL_NET any (msg:"ET JA3 Hash - Suspected Cobalt Strike Malleable C2 M1 (set)"; flow:established,to_server; ja3.hash; content:"eb88d0b3e1961a0562f006e5ce2a0b87"; ja3.string; content:"771,49192-49191-49172-49171"; flowbits:set,ET.cobaltstrike.ja3; flowbits:noalert; classtype:command-and-control; sid:2028831; rev:1; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, created_at 2019_10_15, deployment Perimeter, former_category JA3, malware_family Cobalt_Strike, confidence Low, signature_severity Major, updated_at 2019_10_15, mitre_tactic_id TA0011, mitre_tactic_name Command_And_Control, mitre_technique_id T1001, mitre_technique_name Data_Obfuscation;)" from file /usr/local/etc/suricata/suricata_32509_igb0/rules/suricata.rules at line 10762
[100124 - Suricata-Main] 2023-11-16 15:57:19 Error: detect-tls-ja3s-hash: ja3(s) support is not enabled
[100124 - Suricata-Main] 2023-11-16 15:57:19 Error: detect: error parsing signature "alert tls $EXTERNAL_NET any -> $HOME_NET any (msg:"ET JA3 HASH - Possible RustyBuer Server Response"; flowbits:isset,ET.rustybuer; ja3s.hash; content:"f6dfdd25d1522e4e1c7cd09bd37ce619"; reference:md5,ea98a9d6ca6f5b2a0820303a1d327593; classtype:bad-unknown; sid:2032960; rev:1; metadata:attack_target Client_Endpoint, created_at 2021_05_13, deployment Perimeter, former_category JA3, malware_family RustyBuer, performance_impact Low, confidence Low, signature_severity Major, updated_at 2021_05_13;)" from file /usr/local/etc/suricata/suricata_32509_igb0/rules/suricata.rules at line 10858
[100124 - Suricata-Main] 2023-11-16 15:57:32 Error: detect: previous sticky buffer has no matches
[100124 - Suricata-Main] 2023-11-16 15:57:32 Error: detect: error parsing signature "alert tcp $EXTERNAL_NET $FILE_DATA_PORTS -> $HOME_NET any (msg:"BROWSER-WEBKIT Apple Safari WebKit out-of-bounds write attempt"; flow:to_client,established; file_data; file_data; content:"try { (function () { let a = { get val() { [...{a = 1.45}] = []|3B| a.val.x|3B| }, }|3B| a.val|3B| })()|3B| } catch (e) { } </script>"; fast_pattern:only; metadata:policy balanced-ips drop, policy max-detect-ips drop, policy security-ips drop, service ftp-data, service http, service imap, service pop3; reference:cve,2017-2505; reference:url,bugs.chromium.org/p/project-zero/issues/detail?id=1137; classtype:attempted-user; sid:51391; rev:1;)" from file /usr/local/etc/suricata/suricata_32509_igb0/rules/suricata.rules at line 34646
[100124 - Suricata-Main] 2023-11-16 15:57:33 Error: detect-parse: "http_header" keyword seen with a sticky buffer still set. Reset sticky buffer with pkt_data before using the modifier.
[100124 - Suricata-Main] 2023-11-16 15:57:33 Error: detect: error parsing signature "alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"INDICATOR-COMPROMISE Potential malware download - .pdf.exe within .zip file"; flow:to_client,established; flowbits:isset,file.zip; file_data; content:".pdf.exe"; fast_pattern:only; content:"Content-Length:"; http_header; metadata:policy max-detect-ips drop, policy security-ips drop, ruleset community, service http; classtype:trojan-activity; sid:31001; rev:2;)" from file /usr/local/etc/suricata/suricata_32509_igb0/rules/suricata.rules at line 36294
[100124 - Suricata-Main] 2023-11-16 15:57:33 Error: detect-parse: "http_header" keyword seen with a sticky buffer still set. Reset sticky buffer with pkt_data before using the modifier.
[100124 - Suricata-Main] 2023-11-16 15:57:33 Error: detect: error parsing signature "alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"INDICATOR-COMPROMISE Potential malware download - .jpg.exe within .zip file"; flow:to_client,established; flowbits:isset,file.zip; file_data; content:".jpg.exe"; fast_pattern:only; content:"Content-Length:"; http_header; metadata:policy max-detect-ips drop, policy security-ips drop, ruleset community, service http; classtype:trojan-activity; sid:31000; rev:2;)" from file /usr/local/etc/suricata/suricata_32509_igb0/rules/suricata.rules at line 36295
[100124 - Suricata-Main] 2023-11-16 15:57:33 Error: detect-parse: "http_header" keyword seen with a sticky buffer still set. Reset sticky buffer with pkt_data before using the modifier.
[100124 - Suricata-Main] 2023-11-16 15:57:33 Error: detect: error parsing signature "alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"INDICATOR-COMPROMISE Potential malware download - .jpeg.exe within .zip file"; flow:to_client,established; flowbits:isset,file.zip; file_data; content:".jpeg.exe"; fast_pattern:only; content:"Content-Length:"; http_header; metadata:policy max-detect-ips drop, policy security-ips drop, ruleset community, service http; classtype:trojan-activity; sid:30999; rev:2;)" from file /usr/local/etc/suricata/suricata_32509_igb0/rules/suricata.rules at line 36296

bmeeks

@tim_co said in Suricata process dying due to hyperscan problem:

And I removed the service_watchdog

That step is good!

Those other errors are unrelated to Suricata, but they point to something getting quite messed up on your system. Restoring a config will not necessarily restore the previous package binary versions. In fact, that can make things worse because new binary versions need certain changes made in the configuration. Suricata and Snort are like this.

First thing I would do is uninstall Suricata using the option under SYSTEM > PACKAGE MANAGER.
Next, reboot the firewall so everything has a fresh clean start.
If you have errors upon the restart, then those need to get fixed before messing around any further with Suricata.
Once the firewall boots cleanly with no errors, then return to SYSTEM > PACKAGE MANAGER and reinstall Suricata.

But to be perfectly frank with you, if you are using Suricata on a home network, there is really no valued added at all. Close to 100% of network traffic these days is encrypted, and unless you have a full proxy configured for MITM, Suricata is not examining the vast majority of data flowing through your firewall. It is just seeing random encrypted bytes that it has no way to decipher and scan. All it can do is examine the source and destination IP addresses and ports, and maybe catch just a glimpse of SNI data for some traffic. It can't see a single thing in SSL web traffic nor email as that is also encrypted with TLS.

tim_co

@bmeeks

To be clear, all the errors I posted are from the suricata.log file, not from the system. Does that change things?

I've tried uninstalling, rebooting, and then installing again. I'm still having the problem.

bmeeks

@tim_co said in Suricata process dying due to hyperscan problem:

[16-Nov-2023 12:49:03 America/Denver] PHP Fatal error: Uncaught TypeError: fwrite(): Argument #1 ($stream) must be of type resource, bool given in /etc/inc/config.lib.inc:172
Stack trace:
#0 /etc/inc/config.lib.inc(172): fwrite(false, 'a:42:{s:7:"vers...')
#1 /etc/inc/config.lib.inc(147): generate_config_cache(Array)
#2 /etc/inc/config.inc(141): parse_config()
#3 /etc/inc/gwlb.inc(25): require_once('/etc/inc/config...')
#4 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...')
#5 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#6 /usr/local/pkg/nut/nut_email.php(24): require_once('/etc/inc/notice...')
#7 {main}
thrown in /etc/inc/config.lib.inc on line 172
[16-Nov-2023 12:49:03 America/Denver] PHP Fatal error: Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:101
Stack trace:
#0 /etc/inc/notices.inc(101): fopen('', 'w')
#1 /etc/inc/config.lib.inc(1148): file_notice('phperror', 'PHP ERROR: Type...', 'PHP errors')
#2 [internal function]: pfSense_clear_globals()
#3 {main}
thrown in /etc/inc/notices.inc on line 101

These are the errors I am talking about. No way on Earth those came from Suricata. Those came from pfSense itself, and to me point to a problem with your configuration restore job.

tim_co

@bmeeks

For the record, my problems with Suricata started as soon as I upgraded to 7.0.2. I only ran the restore after i was unable to get Suricata to start and work properly.

I went through and deleted an old backup gateway I had configured or LTE backup and removed the nut package. These seemed to be contributing to the system errors.

I uninstalled Suricata, rebooted, and there was no longer a kernel error. I reinstalled Suricata, ran an update, and now it stays running but it's not detecting anything. Typically there's an alert/block at least every few minutes.

The only errors I see now in the system log are related to Suricata not being able to parse thing files and something about [100107] <Error> -- ja3(s) support is not enabled

Any more ideas?

Thanks - Tim

tim_co

@bmeeks

If I remove all Suricata rules, the only error I get is the "[100404 - Suricata-Main] 2023-11-16 17:17:22 Notice: threads: Threads created -> RX: 1 W: 4 FM: 1 FR: 1 Engine started.
[100759 - W#04] 2023-11-16 17:17:27 Error: spm-hs: Hyperscan returned fatal error -1."

Tim

bmeeks

@tim_co said in Suricata process dying due to hyperscan problem:

The only errors I see now in the system log are related to Suricata not being able to parse thing files and something about [100107] <Error> -- ja3(s) support is not enabled

That error is harmless for now. It simply means that particular protocol parser is not enabled. It can be ignored.

@tim_co said in Suricata process dying due to hyperscan problem:

If I remove all Suricata rules, the only error I get is the "[100404 - Suricata-Main] 2023-11-16 17:17:22 Notice: threads: Threads created -> RX: 1 W: 4 FM: 1 FR: 1 Engine started.
[100759 - W#04] 2023-11-16 17:17:27 Error: spm-hs: Hyperscan returned fatal error -1."

You have absolutely zero rule categories enabled and you are getting that Hyperscan error? That does not seem to be the same bug because rules with regex patterns to be compiled are required to trigger the Hyperscan bug discussed in this thread. The error occurs when the library is actively compiling regex expressions pulled from enabled rules.

tim_co

Suricata has been rock solid for years. Its a remarkable tool. It worked great until I upgraded to 7.0.2 today.

I'm just trying to get Suricats running again by providing data that seems like it might be helpful with troubleshooting the bug and/or my environment. I clearly don't know what I'm doing.

Tim

Bismarck

@bmeeks

Same issue here, Suricata was fine until update to 7.0.2 (HS 5.4.0), after update it was not starting. Changed Pattern Matcher Algorithm from Auto to AC and its working again. However have 2 other boxes with the 1:1 same Suricata configuration, but they start just fine, only the hardware is different there.