Suricata process dying due to hyperscan problem

bmeeks

@kiokoman said in Suricata process dying due to hyperscan problem:

@bmeeks
yes, it's running on an interface i have without blocking mode

Please share the output of this command run from a shell prompt:

pkg info | grep suricata

Let's make sure you have the latest binary. It should show suricata-7.0.2_5.

kiokoman

@bmeeks

Shell Output - pkg info | grep suricata
pfSense-pkg-suricata-7.0.2_2 pfSense package suricata
suricata-7.0.2_5 High Performance Network IDS, IPS and Security Monitoring engine

this is pfsense 2.7.2
the strange part is that i have another pfsense 23.09.1-RELEASE that it's running on vmware with the same packages and more vlans but it have no trouble ....
the only difference is that 23.09.1 is running with 4 cpu and 2.7.2 with 8 cpu
tomorrow i can try to lower the cpu and see if there is any difference

Negan

Still broken here also..... :-(

[177766 - RX#01-ix0] 2023-12-12 06:48:02 Info: checksum: More than 1/10th of packets have an invalid checksum, assuming checksum offloading is used (193/1000)
[177768 - W#02] 2023-12-12 07:17:29 Error: spm-hs: Hyperscan returned fatal error -1.

paulp

After upgrade, the problem with the hyperscan error still occurs for me as well.

bmeeks

Well, sorry but I'm fresh out of ideas at this point. I have no clue what it could be. That fix was my last best hope.

Since I cannot reproduce the problem, it makes it practically impossible to troubleshoot and debug.

Maltz

@bmeeks I think I mentioned it somewhere else, too, but the problem only occurs for me (Netgate 2100) when I have DNSBL (pfBlocker) running also. Have you tried that combo to reproduce? (Or at least occurs much faster - I've only run it for a few days in a row with DNSBL disabled, but when both are running, Suricata dies within 5-15 minutes.)

Negan

I should have also mentioned I'm running a Netgate 8200,..

paulp

I also have pfBlockerBG active. I will uninstall it to see if the problem still occurs.

@bmeeks: On one of my pfsense servers the problem manifests itself relatively quickly - a few hours.
I would be happy if I could somehow help to identify the problem.

sgnoc

Not sure why, but I haven't had a single segfault kill my suricata interfaces running auto on my netgate 7100 for the last few hours, after the upgrade. It would have normally long since had the hyperscan error and core dumped the process. I'm keeping an eye on it, but so far so good.

Bismarck

Same here with pattern match auto or blocking enabled, suricata still won't start.

pkg info | grep suricata
pfSense-pkg-suricata-7.0.2_2   pfSense package suricata
suricata-7.0.2_5               High Performance Network IDS, IPS and Security Monitoring engine

Negan

Output from my 8200

pkg info | grep suricata

pfSense-pkg-suricata-7.0.2_2 pfSense package suricata
suricata-7.0.2_5 High Performance Network IDS, IPS and Security Monitoring engine

paulp

I uninstalled pfBlockerNG, but the error persists on the wan interface.
(It is a hyper-v machine with Pfsense 23.09.1-RELEASE with suricata-7.0.2_5)

bmeeks

@Bismarck said in Suricata process dying due to hyperscan problem:

Same here with pattern match auto or blocking enabled, suricata still won't start.

What error do you get on startup if you leave the Pattern Matcher set to AC-KS but enable only blocking? In other words, enable blocking but do not change the Pattern Matcher setting.

Show the output of the suricata.log file for the impacted interface (located under the LOGS VIEW tab) and also post anything relevant during the same time from the pfSense system log (under STATUS > SYSTEM LOGS).

Bismarck

@bmeeks no error, "pattern match auto" or "blocking enabled" kills suricata. AC-KS or "disabling blocking" is fine.

bmeeks

@Bismarck said in Suricata process dying due to hyperscan problem:

@bmeeks no error, "pattern match auto" or "blocking enabled" kills suricata. AC-KS or "disabling blocking" is fine.

Help me understand this better...

I've gotten a lot of seemingly conflicting information from the posters in this long thread. My confusion is leading to frustration .

So if you leave Pattern Matching at AC-KS and enable blocking, Suricata starts and runs with no issue. Is that correct?
The only way to make it crash is to set Pattern Matcher to Auto with blocking enabled? Is that correct?
And just to close the loop- if you set Pattern Matcher to Auto but disable blocking Suricata starts without error. Is that correct?

For the crash configurations, what is the output of the suricata.log file? Is it always the "Hyperscan returned fatal error -1" message?

Bismarck

@bmeeks

yes
yes
yes

suricata.log stops at this line

[118101 - Suricata-Main] 2023-12-12 15:10:30 Error: detect: error parsing signature "alert tcp any any -> any $HTTP_PORTS (msg:"SERVER-WEBAPP Microsoft SharePoint OAuth authentication bypass attempt"; flow:to_server,established; content:"access_token="; fast_pattern; nocase; http_client_body; base64_decode:bytes 100,relative; base64_data; content:"|22|alg|22|"; nocase; content:"|22|none|22|"; within:50; nocase; content:"/_api/"; nocase; http_uri; pcre:"/\x2f_api\x2f(web\x2f|lists\x2f|Microsoft|SP\x2e|_vti_bin|_layouts|apps\x2f|search\x2f)/Ui"; metadata:policy balanced-ips drop, policy max-detect-ips drop, policy security-ips drop, service http; reference:cve,2023-29357; reference:url,portal.msrc.microsoft.com/en-us/security-guidance/advisory/CVE-2023-29357; classtype:attempted-admin; sid:62467; rev:1;)" from file /usr/local/etc/suricata/suricata_10547_igc2/rules/suricata.rules at line 33984
[118101 - Suricata-Main] 2023-12-12 15:10:30 Error: detect-parse: "http_uri" keyword seen with a sticky buffer still set.  Reset sticky buffer with pkt_data before using the modifier.
[118101 - Suricata-Main] 2023-12-12 15:10:30 Error: detect: error parsing signature "alert tcp any any -> any $HTTP_PORTS (msg:"SERVER-WEBAPP Microsoft SharePoint OAuth authentication bypass attempt"; flow:to_server,established; content:"access_token="; nocase; http_client_body; base64_decode:bytes 100,relative; base64_data; content:"|22|alg|22|"; nocase; content:"|22|none|22|"; within:50; nocase; content:"/_layouts/15/"; fast_pattern:only; http_uri; metadata:policy balanced-ips drop, policy max-detect-ips drop, policy security-ips drop, service http; reference:cve,2023-29357; reference:url,portal.msrc.microsoft.com/en-us/security-guidance/advisory/CVE-2023-29357; classtype:attempted-admin; sid:62465; rev:1;)" from file /usr/local/etc/suricata/suricata_10547_igc2/rules/suricata.rules at line 33986
[118101 - Suricata-Main] 2023-12-12 15:10:30 Info: detect: 2 rule files processed. 34422 rules successfully loaded, 61 rules failed
[118101 - Suricata-Main] 2023-12-12 15:10:30 Warning: threshold-config: can't suppress sid 2210063, gid 1: unknown rule
[118101 - Suricata-Main] 2023-12-12 15:10:30 Info: threshold-config: Threshold config parsed: 1 rule(s) found

system log gives:

kernel pid 72109 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)

/edit for suricata.log

bmeeks

The best workaround for you guys is to select AC-CS (the normal default when Hyperscan is not present on a system) or one of the other AC-* pattern matcher settings. I have no idea what is wrong with Hyperscan at this point, and I have no reasonable hope of being able to fix it when I can't even reproduce it. It's like throwing darts at a target while blindfolded.

Hyperscan is eventually likely going to be removed from Suricata anyway. Intel seems to have begun a process of taking the library closed-source and making it proprietary. There is an open-source equivalent called VectorScan, but so far it runs only on non-Intel hardware because that's why it was created in the first place. Hyperscan was for Intel-only CPUs, so that left a number of other platforms (ARM, PowerPC, etc.) out. Thus some folks created the VectorScan project for those other CPUs. But that project was not originally created with Intel CPUs as its target, so it may take a bit for things to settle out. Here is a link about this: https://github.com/intel/hyperscan/issues/421. Follow the additional links in that GitHub discussion to learn more.

bmeeks

@Bismarck said in Suricata process dying due to hyperscan problem:

@bmeeks

yes
yes
yes

suricata.log stops at this line

[118101 - Suricata-Main] 2023-12-12 15:10:30 Error: detect: error parsing signature "alert tcp any any -> any $HTTP_PORTS (msg:"SERVER-WEBAPP Microsoft SharePoint OAuth authentication bypass attempt"; flow:to_server,established; content:"access_token="; fast_pattern; nocase; http_client_body; base64_decode:bytes 100,relative; base64_data; content:"|22|alg|22|"; nocase; content:"|22|none|22|"; within:50; nocase; content:"/_api/"; nocase; http_uri; pcre:"/\x2f_api\x2f(web\x2f|lists\x2f|Microsoft|SP\x2e|_vti_bin|_layouts|apps\x2f|search\x2f)/Ui"; metadata:policy balanced-ips drop, policy max-detect-ips drop, policy security-ips drop, service http; reference:cve,2023-29357; reference:url,portal.msrc.microsoft.com/en-us/security-guidance/advisory/CVE-2023-29357; classtype:attempted-admin; sid:62467; rev:1;)" from file /usr/local/etc/suricata/suricata_10547_igc2/rules/suricata.rules at line 33984
[118101 - Suricata-Main] 2023-12-12 15:10:30 Error: detect-parse: "http_uri" keyword seen with a sticky buffer still set.  Reset sticky buffer with pkt_data before using the modifier.
[118101 - Suricata-Main] 2023-12-12 15:10:30 Error: detect: error parsing signature "alert tcp any any -> any $HTTP_PORTS (msg:"SERVER-WEBAPP Microsoft SharePoint OAuth authentication bypass attempt"; flow:to_server,established; content:"access_token="; nocase; http_client_body; base64_decode:bytes 100,relative; base64_data; content:"|22|alg|22|"; nocase; content:"|22|none|22|"; within:50; nocase; content:"/_layouts/15/"; fast_pattern:only; http_uri; metadata:policy balanced-ips drop, policy max-detect-ips drop, policy security-ips drop, service http; reference:cve,2023-29357; reference:url,portal.msrc.microsoft.com/en-us/security-guidance/advisory/CVE-2023-29357; classtype:attempted-admin; sid:62465; rev:1;)" from file /usr/local/etc/suricata/suricata_10547_igc2/rules/suricata.rules at line 33986
[118101 - Suricata-Main] 2023-12-12 15:10:30 Info: detect: 2 rule files processed. 34422 rules successfully loaded, 61 rules failed
[118101 - Suricata-Main] 2023-12-12 15:10:30 Warning: threshold-config: can't suppress sid 2210063, gid 1: unknown rule
[118101 - Suricata-Main] 2023-12-12 15:10:30 Info: threshold-config: Threshold config parsed: 1 rule(s) found

system log gives:

kernel pid 72109 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)

/edit for suricata.log

Thanks! Can you post the entire suricata.log file for an interface when it crashes? You can scrub or x-out any IP addresses if you need to. I want to examine the whole log to see what steps are completed.

And to clarify, have you ever seen the "Hyperscan returned fatal error -1" log message?

Bismarck

@bmeeks said in Suricata process dying due to hyperscan problem:

there you go

suricata.log crash (Auto)

https://pastebin.com/GAQn5CCy

suricata.log nocrash (AC-KC)

https://pastebin.com/kRcWVNDJ

And to clarify, have you ever seen the "Hyperscan returned fatal error -1" log message?

For me no, did a log search and only came up with signal 11 core dumped messages.

tylerevers

@bmeeks said in Suricata process dying due to hyperscan problem:

My pull request containing the anticipated fix for this Hyperscan error has been merged. An updated Suricata package has built and should appear as an available update for 2.7.2 CE and 23.09.1 Plus users.

Look for an update to version 7.0.2_2 for the Suricata package. When installed, the new package should pull in version 7.0.2_5 of the Suricata binary.

Fingers crossed this fixes the Hyperscan issue. But as I mentioned previously, since I could never reproduce the error in my small test environment, I can't say with 100% certainty the bug I found and fixed is the actual Hyperscan culprit.

Nearly 20 hours since updating to 7.0.2_2 on 23.09.1 Plus with custom bare metal setup and no Hyperscan crash yet. Pattern Match set to AUTO and Blocking Mode ENABLED. Using all VLANs that traverse a LAGG in my case just as a reminder.

Thanks, Bill!