Suricata process dying due to hyperscan problem
-
@tgm-its
use AC-BS instead of Auto
-
@tgm-its said in Suricata process dying due to hyperscan problem:
And yes, is there a way to turn off e Hyperscan system so that Suricata runs without it?
Currently no patch is available because the cause of the failure has yet to be determined. I cannot reproduce it in my limited test environment. At least I have not yet encountered it. It is a seemingly random sort of bug impacting some users and not others. Very difficult to troubleshoot a failure that you cannot reliably reproduce, but I'm still trying.
You can disable the use of the Hyperscan library by going to the INTERFACE SETTINGS page for an interface and scrolling down to the Pattern Matcher Algorithm parameter and choosing something other than
Auto
orHyperscan
. Save the change and restart Suricata on the interface so that the underlying binary sees the new parameter choice. -
@bmeeks
For me, the problem always manifests itself only on the network interfaces on which there is some traffic. The problem does not appear on network interfaces with low traffic.
The problem appears after an interval between 1-8 hours, probably the higher the traffic, the faster the problem appears. -
@paulp said in Suricata process dying due to hyperscan problem:
@bmeeks
For me, the problem always manifests itself only on the network interfaces on which there is some traffic. The problem does not appear on network interfaces with low traffic.
The problem appears after an interval between 1-8 hours, probably the higher the traffic, the faster the problem appears.Hmm...that does make some sense. I am testing with very low traffic in a limited VMware Workstation virtual machine setup. Perhaps hammering a test machine with
iperf3
traffic might trigger the bug ??? -
@kiokoman Thanks for the suggestion. I'll try that.
Paal B.
-
@bmeeks
idk if iperf3 can generate enough alert to trigger the bug maybe it's more like how many alert there are more than how mutch data is in the wire -
@bmeeks I was trying to use the AC-KS pattern matcher as an alternative to hyperscan, but they must be related in some manner. This is the log I receive after just a few minutes running AC-KS. Also, I'm getting the hyperscan errors on my WAN interface, which remains busy, but also on this interface that has little traffic and does not alert all that often, so it isn't just interfaces that are more active on my network.
Here is the last few lines in suricata.log for the failed interface running with AC-KS:
[186263 - RX#01-ix1.15] 2023-12-07 09:37:41 Info: pcap: ix1.15: running in 'auto' checksum mode. Detection of interface state will require 1000 packets [186263 - RX#01-ix1.15] 2023-12-07 09:37:41 Info: pcap: ix1.15: snaplen set to 1518 [100242 - Suricata-Main] 2023-12-07 09:37:41 Notice: threads: Threads created -> RX: 1 W: 4 FM: 1 FR: 1 Engine started. [186263 - RX#01-ix1.15] 2023-12-07 09:41:10 Info: checksum: No packets with invalid checksum, assuming checksum offloading is NOT used [186267 - W#04] 2023-12-07 09:46:00 Error: spm-hs: Hyperscan returned fatal error -1.
Here are the output from the core dump:
(gdb) bt #0 0x0000000830c6834a in thr_kill () from /lib/libc.so.7 #1 0x0000000830be8344 in raise () from /lib/libc.so.7 #2 0x0000000830c8ca39 in abort () from /lib/libc.so.7 #3 0x0000000830cdbf30 in ?? () from /lib/libc.so.7 #4 0x0000000830ca8440 in ?? () from /lib/libc.so.7 #5 0x0000000830ca142c in ?? () from /lib/libc.so.7 #6 0x0000000830ca1159 in ?? () from /lib/libc.so.7 #7 0x00000000005ac55a in ConfNodeFree () #8 0x00000000005ac536 in ConfNodeFree () #9 0x00000000005ac536 in ConfNodeFree () #10 0x00000000005ad305 in ConfDeInit () #11 0x000000000058fa61 in ?? () #12 0x000000000058f03f in SuricataMain () #13 0x0000000830bbf75a in __libc_start1 () from /lib/libc.so.7 #14 0x000000000058bd40 in _start ()
(gdb) bt full #0 0x0000000830c6834a in thr_kill () from /lib/libc.so.7 No symbol table info available. #1 0x0000000830be8344 in raise () from /lib/libc.so.7 No symbol table info available. #2 0x0000000830c8ca39 in abort () from /lib/libc.so.7 No symbol table info available. #3 0x0000000830cdbf30 in ?? () from /lib/libc.so.7 No symbol table info available. #4 0x0000000830ca8440 in ?? () from /lib/libc.so.7 No symbol table info available. #5 0x0000000830ca142c in ?? () from /lib/libc.so.7 No symbol table info available. #6 0x0000000830ca1159 in ?? () from /lib/libc.so.7 No symbol table info available. #7 0x00000000005ac55a in ConfNodeFree () No symbol table info available. #8 0x00000000005ac536 in ConfNodeFree () No symbol table info available. #9 0x00000000005ac536 in ConfNodeFree () No symbol table info available. #10 0x00000000005ad305 in ConfDeInit () No symbol table info available. #11 0x000000000058fa61 in ?? () No symbol table info available. #12 0x000000000058f03f in SuricataMain () No symbol table info available. #13 0x0000000830bbf75a in __libc_start1 () from /lib/libc.so.7 No symbol table info available. #14 0x000000000058bd40 in _start () No symbol table info available.
(gdb) info threads Id Target Id Frame * 1 LWP 102018 0x0000000830c6834a in thr_kill () from /lib/libc.so.7
(gdb) thread apply all bt Thread 1 (LWP 102018): #0 0x0000000830c6834a in thr_kill () from /lib/libc.so.7 #1 0x0000000830be8344 in raise () from /lib/libc.so.7 #2 0x0000000830c8ca39 in abort () from /lib/libc.so.7 #3 0x0000000830cdbf30 in ?? () from /lib/libc.so.7 #4 0x0000000830ca8440 in ?? () from /lib/libc.so.7 #5 0x0000000830ca142c in ?? () from /lib/libc.so.7 #6 0x0000000830ca1159 in ?? () from /lib/libc.so.7 #7 0x00000000005ac55a in ConfNodeFree () #8 0x00000000005ac536 in ConfNodeFree () #9 0x00000000005ac536 in ConfNodeFree () #10 0x00000000005ad305 in ConfDeInit () #11 0x000000000058fa61 in ?? () #12 0x000000000058f03f in SuricataMain () #13 0x0000000830bbf75a in __libc_start1 () from /lib/libc.so.7 #14 0x000000000058bd40 in _start ()
(gdb) thread apply all bt full Thread 1 (LWP 102018): #0 0x0000000830c6834a in thr_kill () from /lib/libc.so.7 No symbol table info available. #1 0x0000000830be8344 in raise () from /lib/libc.so.7 No symbol table info available. #2 0x0000000830c8ca39 in abort () from /lib/libc.so.7 No symbol table info available. #3 0x0000000830cdbf30 in ?? () from /lib/libc.so.7 No symbol table info available. #4 0x0000000830ca8440 in ?? () from /lib/libc.so.7 No symbol table info available. #5 0x0000000830ca142c in ?? () from /lib/libc.so.7 No symbol table info available. #6 0x0000000830ca1159 in ?? () from /lib/libc.so.7 No symbol table info available. #7 0x00000000005ac55a in ConfNodeFree () No symbol table info available. #8 0x00000000005ac536 in ConfNodeFree () No symbol table info available. #9 0x00000000005ac536 in ConfNodeFree () No symbol table info available. #10 0x00000000005ad305 in ConfDeInit () No symbol table info available. #11 0x000000000058fa61 in ?? () No symbol table info available. #12 0x000000000058f03f in SuricataMain () No symbol table info available. #13 0x0000000830bbf75a in __libc_start1 () from /lib/libc.so.7 No symbol table info available. #14 0x000000000058bd40 in _start () No symbol table info available.
I'll try and run this interface on auto (hyperscan) and catpure the core dump to post the debug details here. Again, this is on my smaller interface that has been failing with suricata. If that doesn't prove as beneficial, I'll work on getting my wan interface to produce the core dump with hyperscan to get you that debug info.
-
@sgnoc said in Suricata process dying due to hyperscan problem:
@bmeeks I was trying to use the AC-KS pattern matcher as an alternative to hyperscan, but they must be related in some manner. This is the log I receive after just a few minutes running AC-KS. Also, I'm getting the hyperscan errors on my WAN interface, which remains busy, but also on this interface that has little traffic and does not alert all that often, so it isn't just interfaces that are more active on my network.
Here is the last few lines in suricata.log for the failed interface running with AC-KS:
[186263 - RX#01-ix1.15] 2023-12-07 09:37:41 Info: pcap: ix1.15: running in 'auto' checksum mode. Detection of interface state will require 1000 packets [186263 - RX#01-ix1.15] 2023-12-07 09:37:41 Info: pcap: ix1.15: snaplen set to 1518 [100242 - Suricata-Main] 2023-12-07 09:37:41 Notice: threads: Threads created -> RX: 1 W: 4 FM: 1 FR: 1 Engine started. [186263 - RX#01-ix1.15] 2023-12-07 09:41:10 Info: checksum: No packets with invalid checksum, assuming checksum offloading is NOT used [186267 - W#04] 2023-12-07 09:46:00 Error: spm-hs: Hyperscan returned fatal error -1.
Here are the output from the core dump:
(gdb) bt #0 0x0000000830c6834a in thr_kill () from /lib/libc.so.7 #1 0x0000000830be8344 in raise () from /lib/libc.so.7 #2 0x0000000830c8ca39 in abort () from /lib/libc.so.7 #3 0x0000000830cdbf30 in ?? () from /lib/libc.so.7 #4 0x0000000830ca8440 in ?? () from /lib/libc.so.7 #5 0x0000000830ca142c in ?? () from /lib/libc.so.7 #6 0x0000000830ca1159 in ?? () from /lib/libc.so.7 #7 0x00000000005ac55a in ConfNodeFree () #8 0x00000000005ac536 in ConfNodeFree () #9 0x00000000005ac536 in ConfNodeFree () #10 0x00000000005ad305 in ConfDeInit () #11 0x000000000058fa61 in ?? () #12 0x000000000058f03f in SuricataMain () #13 0x0000000830bbf75a in __libc_start1 () from /lib/libc.so.7 #14 0x000000000058bd40 in _start ()
(gdb) bt full #0 0x0000000830c6834a in thr_kill () from /lib/libc.so.7 No symbol table info available. #1 0x0000000830be8344 in raise () from /lib/libc.so.7 No symbol table info available. #2 0x0000000830c8ca39 in abort () from /lib/libc.so.7 No symbol table info available. #3 0x0000000830cdbf30 in ?? () from /lib/libc.so.7 No symbol table info available. #4 0x0000000830ca8440 in ?? () from /lib/libc.so.7 No symbol table info available. #5 0x0000000830ca142c in ?? () from /lib/libc.so.7 No symbol table info available. #6 0x0000000830ca1159 in ?? () from /lib/libc.so.7 No symbol table info available. #7 0x00000000005ac55a in ConfNodeFree () No symbol table info available. #8 0x00000000005ac536 in ConfNodeFree () No symbol table info available. #9 0x00000000005ac536 in ConfNodeFree () No symbol table info available. #10 0x00000000005ad305 in ConfDeInit () No symbol table info available. #11 0x000000000058fa61 in ?? () No symbol table info available. #12 0x000000000058f03f in SuricataMain () No symbol table info available. #13 0x0000000830bbf75a in __libc_start1 () from /lib/libc.so.7 No symbol table info available. #14 0x000000000058bd40 in _start () No symbol table info available.
(gdb) info threads Id Target Id Frame * 1 LWP 102018 0x0000000830c6834a in thr_kill () from /lib/libc.so.7
(gdb) thread apply all bt Thread 1 (LWP 102018): #0 0x0000000830c6834a in thr_kill () from /lib/libc.so.7 #1 0x0000000830be8344 in raise () from /lib/libc.so.7 #2 0x0000000830c8ca39 in abort () from /lib/libc.so.7 #3 0x0000000830cdbf30 in ?? () from /lib/libc.so.7 #4 0x0000000830ca8440 in ?? () from /lib/libc.so.7 #5 0x0000000830ca142c in ?? () from /lib/libc.so.7 #6 0x0000000830ca1159 in ?? () from /lib/libc.so.7 #7 0x00000000005ac55a in ConfNodeFree () #8 0x00000000005ac536 in ConfNodeFree () #9 0x00000000005ac536 in ConfNodeFree () #10 0x00000000005ad305 in ConfDeInit () #11 0x000000000058fa61 in ?? () #12 0x000000000058f03f in SuricataMain () #13 0x0000000830bbf75a in __libc_start1 () from /lib/libc.so.7 #14 0x000000000058bd40 in _start ()
(gdb) thread apply all bt full Thread 1 (LWP 102018): #0 0x0000000830c6834a in thr_kill () from /lib/libc.so.7 No symbol table info available. #1 0x0000000830be8344 in raise () from /lib/libc.so.7 No symbol table info available. #2 0x0000000830c8ca39 in abort () from /lib/libc.so.7 No symbol table info available. #3 0x0000000830cdbf30 in ?? () from /lib/libc.so.7 No symbol table info available. #4 0x0000000830ca8440 in ?? () from /lib/libc.so.7 No symbol table info available. #5 0x0000000830ca142c in ?? () from /lib/libc.so.7 No symbol table info available. #6 0x0000000830ca1159 in ?? () from /lib/libc.so.7 No symbol table info available. #7 0x00000000005ac55a in ConfNodeFree () No symbol table info available. #8 0x00000000005ac536 in ConfNodeFree () No symbol table info available. #9 0x00000000005ac536 in ConfNodeFree () No symbol table info available. #10 0x00000000005ad305 in ConfDeInit () No symbol table info available. #11 0x000000000058fa61 in ?? () No symbol table info available. #12 0x000000000058f03f in SuricataMain () No symbol table info available. #13 0x0000000830bbf75a in __libc_start1 () from /lib/libc.so.7 No symbol table info available. #14 0x000000000058bd40 in _start () No symbol table info available.
I'll try and run this interface on auto (hyperscan) and catpure the core dump to post the debug details here. Again, this is on my smaller interface that has been failing with suricata. If that doesn't prove as beneficial, I'll work on getting my wan interface to produce the core dump with hyperscan to get you that debug info.
Thank you for this info. Weird that it seems to be crashing in a section of code where Suricata is releasing memory and cleaning up after reading the configuration from the
suricata.yaml
file. Those calls toConfNodeFree()
andConfNodeInit()
are made when finishing and cleaning up from reading the YAML conf file parameters.Let's see if another crash happens in the same area of code. One thing I noticed before (but that might have been with the Kill States bug and not Hyperscan) was the crash happened in seemingly random and unrelated places.
Please continue posting the relevant sections of any future core dumps' backtrace as you did here.
-
@bmeeks I switched my WAN interface to auto. Here are the logs:
(gdb) bt #0 0x000000082ff3f34a in thr_kill () from /lib/libc.so.7 #1 0x000000082febf344 in raise () from /lib/libc.so.7 #2 0x000000082ff63a39 in abort () from /lib/libc.so.7 #3 0x000000082ffb2f30 in ?? () from /lib/libc.so.7 #4 0x000000082ff7f440 in ?? () from /lib/libc.so.7 #5 0x000000082ff7842c in ?? () from /lib/libc.so.7 #6 0x000000082ff78159 in ?? () from /lib/libc.so.7 #7 0x00000000005ac54c in ConfNodeFree () #8 0x00000000005ac536 in ConfNodeFree () #9 0x00000000005ac536 in ConfNodeFree () #10 0x00000000005ad305 in ConfDeInit () #11 0x000000000058fa61 in ?? () #12 0x000000000058f03f in SuricataMain () #13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7 #14 0x000000000058bd40 in _start ()```
(gdb) bt full #0 0x000000082ff3f34a in thr_kill () from /lib/libc.so.7 No symbol table info available. #1 0x000000082febf344 in raise () from /lib/libc.so.7 No symbol table info available. #2 0x000000082ff63a39 in abort () from /lib/libc.so.7 No symbol table info available. #3 0x000000082ffb2f30 in ?? () from /lib/libc.so.7 No symbol table info available. #4 0x000000082ff7f440 in ?? () from /lib/libc.so.7 No symbol table info available. #5 0x000000082ff7842c in ?? () from /lib/libc.so.7 No symbol table info available. #6 0x000000082ff78159 in ?? () from /lib/libc.so.7 No symbol table info available. #7 0x00000000005ac54c in ConfNodeFree () No symbol table info available. #8 0x00000000005ac536 in ConfNodeFree () No symbol table info available. #9 0x00000000005ac536 in ConfNodeFree () No symbol table info available. #10 0x00000000005ad305 in ConfDeInit () No symbol table info available. #11 0x000000000058fa61 in ?? () No symbol table info available. #12 0x000000000058f03f in SuricataMain () No symbol table info available. #13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7 No symbol table info available. #14 0x000000000058bd40 in _start () No symbol table info available.
(gdb) info threads Id Target Id Frame * 1 LWP 103117 0x000000082ff3f34a in thr_kill () from /lib/libc.so.7
(gdb) thread apply all bt Thread 1 (LWP 103117): #0 0x000000082ff3f34a in thr_kill () from /lib/libc.so.7 #1 0x000000082febf344 in raise () from /lib/libc.so.7 #2 0x000000082ff63a39 in abort () from /lib/libc.so.7 #3 0x000000082ffb2f30 in ?? () from /lib/libc.so.7 #4 0x000000082ff7f440 in ?? () from /lib/libc.so.7 #5 0x000000082ff7842c in ?? () from /lib/libc.so.7 #6 0x000000082ff78159 in ?? () from /lib/libc.so.7 #7 0x00000000005ac54c in ConfNodeFree () #8 0x00000000005ac536 in ConfNodeFree () #9 0x00000000005ac536 in ConfNodeFree () #10 0x00000000005ad305 in ConfDeInit () #11 0x000000000058fa61 in ?? () #12 0x000000000058f03f in SuricataMain () #13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7 #14 0x000000000058bd40 in _start ()
(gdb) thread apply all bt full Thread 1 (LWP 103117): #0 0x000000082ff3f34a in thr_kill () from /lib/libc.so.7 No symbol table info available. #1 0x000000082febf344 in raise () from /lib/libc.so.7 No symbol table info available. #2 0x000000082ff63a39 in abort () from /lib/libc.so.7 No symbol table info available. #3 0x000000082ffb2f30 in ?? () from /lib/libc.so.7 No symbol table info available. #4 0x000000082ff7f440 in ?? () from /lib/libc.so.7 No symbol table info available. #5 0x000000082ff7842c in ?? () from /lib/libc.so.7 No symbol table info available. #6 0x000000082ff78159 in ?? () from /lib/libc.so.7 No symbol table info available. #7 0x00000000005ac54c in ConfNodeFree () No symbol table info available. #8 0x00000000005ac536 in ConfNodeFree () No symbol table info available. #9 0x00000000005ac536 in ConfNodeFree () No symbol table info available. #10 0x00000000005ad305 in ConfDeInit () No symbol table info available. #11 0x000000000058fa61 in ?? () No symbol table info available. #12 0x000000000058f03f in SuricataMain () No symbol table info available. #13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7 No symbol table info available. #14 0x000000000058bd40 in _start () No symbol table info available.
-
@sgnoc said in Suricata process dying due to hyperscan problem:
@bmeeks I switched my WAN interface to auto. Here are the logs:
(gdb) bt #0 0x000000082ff3f34a in thr_kill () from /lib/libc.so.7 #1 0x000000082febf344 in raise () from /lib/libc.so.7 #2 0x000000082ff63a39 in abort () from /lib/libc.so.7 #3 0x000000082ffb2f30 in ?? () from /lib/libc.so.7 #4 0x000000082ff7f440 in ?? () from /lib/libc.so.7 #5 0x000000082ff7842c in ?? () from /lib/libc.so.7 #6 0x000000082ff78159 in ?? () from /lib/libc.so.7 #7 0x00000000005ac54c in ConfNodeFree () #8 0x00000000005ac536 in ConfNodeFree () #9 0x00000000005ac536 in ConfNodeFree () #10 0x00000000005ad305 in ConfDeInit () #11 0x000000000058fa61 in ?? () #12 0x000000000058f03f in SuricataMain () #13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7 #14 0x000000000058bd40 in _start ()```
(gdb) bt full #0 0x000000082ff3f34a in thr_kill () from /lib/libc.so.7 No symbol table info available. #1 0x000000082febf344 in raise () from /lib/libc.so.7 No symbol table info available. #2 0x000000082ff63a39 in abort () from /lib/libc.so.7 No symbol table info available. #3 0x000000082ffb2f30 in ?? () from /lib/libc.so.7 No symbol table info available. #4 0x000000082ff7f440 in ?? () from /lib/libc.so.7 No symbol table info available. #5 0x000000082ff7842c in ?? () from /lib/libc.so.7 No symbol table info available. #6 0x000000082ff78159 in ?? () from /lib/libc.so.7 No symbol table info available. #7 0x00000000005ac54c in ConfNodeFree () No symbol table info available. #8 0x00000000005ac536 in ConfNodeFree () No symbol table info available. #9 0x00000000005ac536 in ConfNodeFree () No symbol table info available. #10 0x00000000005ad305 in ConfDeInit () No symbol table info available. #11 0x000000000058fa61 in ?? () No symbol table info available. #12 0x000000000058f03f in SuricataMain () No symbol table info available. #13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7 No symbol table info available. #14 0x000000000058bd40 in _start () No symbol table info available.
(gdb) info threads Id Target Id Frame * 1 LWP 103117 0x000000082ff3f34a in thr_kill () from /lib/libc.so.7
(gdb) thread apply all bt Thread 1 (LWP 103117): #0 0x000000082ff3f34a in thr_kill () from /lib/libc.so.7 #1 0x000000082febf344 in raise () from /lib/libc.so.7 #2 0x000000082ff63a39 in abort () from /lib/libc.so.7 #3 0x000000082ffb2f30 in ?? () from /lib/libc.so.7 #4 0x000000082ff7f440 in ?? () from /lib/libc.so.7 #5 0x000000082ff7842c in ?? () from /lib/libc.so.7 #6 0x000000082ff78159 in ?? () from /lib/libc.so.7 #7 0x00000000005ac54c in ConfNodeFree () #8 0x00000000005ac536 in ConfNodeFree () #9 0x00000000005ac536 in ConfNodeFree () #10 0x00000000005ad305 in ConfDeInit () #11 0x000000000058fa61 in ?? () #12 0x000000000058f03f in SuricataMain () #13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7 #14 0x000000000058bd40 in _start ()
(gdb) thread apply all bt full Thread 1 (LWP 103117): #0 0x000000082ff3f34a in thr_kill () from /lib/libc.so.7 No symbol table info available. #1 0x000000082febf344 in raise () from /lib/libc.so.7 No symbol table info available. #2 0x000000082ff63a39 in abort () from /lib/libc.so.7 No symbol table info available. #3 0x000000082ffb2f30 in ?? () from /lib/libc.so.7 No symbol table info available. #4 0x000000082ff7f440 in ?? () from /lib/libc.so.7 No symbol table info available. #5 0x000000082ff7842c in ?? () from /lib/libc.so.7 No symbol table info available. #6 0x000000082ff78159 in ?? () from /lib/libc.so.7 No symbol table info available. #7 0x00000000005ac54c in ConfNodeFree () No symbol table info available. #8 0x00000000005ac536 in ConfNodeFree () No symbol table info available. #9 0x00000000005ac536 in ConfNodeFree () No symbol table info available. #10 0x00000000005ad305 in ConfDeInit () No symbol table info available. #11 0x000000000058fa61 in ?? () No symbol table info available. #12 0x000000000058f03f in SuricataMain () No symbol table info available. #13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7 No symbol table info available. #14 0x000000000058bd40 in _start () No symbol table info available.
So, it seems to be crashing in the same place. Struggling to tie this behavior back to Hyperscan and the custom Legacy Blocking Module, but perhaps there is some link there I'm missing. I'll dig around for a bit.
-
@bmeeks I can't help you with that side, but I know it keeps referencing hyperscan in the logs with the fatal error and it only occurs when I am using auto, which defaults to hyperscan, but also when I use AC-KS for the pattern matcher. It doesn't happen with AC only (I haven't tried AC-BS or specifically selecting hyperscan).
[102876 - W#02] 2023-12-07 11:27:44 Error: spm-hs: Hyperscan returned fatal error -1.
-
The strange thing.
Now after I set it to AC-BS it's been running for over 2 hours without failing.So seems to be up now.
But does it work as it should?
I checked Alert and last entry there was for like 4 minute's ago.Paal B.
-
The Pattern Matcher Algorithm is not critical to whether detection works or not. It only matters in terms of speed versus memory consumption.
All fhe pattern matcher does is execute a regular expression (regex) search against packet data looking for whatever pattern the rule specified. The different algorithms all accomplish the exact same thing. It's just that some use less memory, but may run slower. Others may be lightning fast, but that speed comes from very high RAM consumption.
Hyperscan is a specialized regex library created by Intel to take advantage of certain complex math instruction op-codes present in their CPUs. Those specialized instructions that most Intel CPUs have allow for highly optimized regex searches. But because Hyperscan depends on instruction op-codes only present in Intel CPUs, the library does not work with ARM or any other non-Intel hardware.
The
Auto
setting for the Pattern Matcher algorithm in Suricata will default to the Hyperscan library if it is present. Otherwise it will chooseAC
(Aho-Corasick). When the Suricata binary is compiled on the Netgate package builder, it automatically senses which type of CPU is targeted (Intel or something else), and then includes the Hyperscan library only for Intel CPU targets. This means if you have a Netgate firewall appliance with an ARM chip, the Hyperscan library is not even present on your system. -
FYI. 23.09.1 released with this memory allocation fix: https://www.freebsd.org/security/advisories/FreeBSD-EN-23:20.vm.asc
-
@gfeiner Unfortunately, in my case at least, Suricata still dies after this upgrade.
-
@paulp said in Suricata process dying due to hyperscan problem:
@gfeiner Unfortunately, in my case at least, Suricata still dies after this upgrade.
me too, still crashing
-
Hi all.
Just a question? Will the new update Version 2.7.2 of pFsense CE fix the Hyperscan error in Suricata?
If so I could update my Firewall if not I'll wait.BR
Paal B. -
Not sure what to make of this, but within 24 hours of updating to 23.09.1 one of my Interfaces in Suricata disappeared by itself. For reference, it was the only Interface I was having the Hyperscan issue with and it was running on AC-KS for weeks without an issue. It is the only one of three Interfaces that was using AC-KS instead of Auto.
I'm not quite sure how this happened. I woke up today and was just looking at systems to discover that one specific Suricata Interface had deleted itself.
-
@tylerevers said in Suricata process dying due to hyperscan problem:
I'm not quite sure how this happened. I woke up today and was just looking at systems to discover that one specific Suricata Interface had deleted itself.
There is no mechanism within the package code for that to happen autonomously.
I would check first the pfSense system log on the impacted system to see what may be logged there, then look in the configuration backup history of pfSense to see if someone took an action. The GUI code logs an appropriate message for all configuration changes saved to the
config.xml
fiile. You can find the configuration entire history under DIAGNOSTICS > BACKUP AND RESTORE. -
Fingers crossed, but I think I found the bug.
I definitely found an errant double
free()
of memory when processing IPv4 addresses in a Pass List. When you execute a doublefree()
of memory you will get random crashes.Since I have been unable to reproduce the problem, I can't say for sure what I found will fix the Hyperscan issue, but I am hopeful based on the fact several of you have stated that turning off Legacy Blocking Mode (in other words, running in plain IDS mode) allows Suricata to run with no issue. The double
free()
was in the custom Legacy Blocking Module, and it was located in new code that was added with the first 7.0.0 Suricata update back when 23.09 Plus was still in development mode and 2.8 CE snapshots were active. Disabling Legacy Blocking Mode means this buggy portion of the module's code is not executed.I will create a pull request and get this fix posted for Netgate to review and merge. That will not happen until the first of the coming week.
Update: the fix is posted at https://github.com/pfsense/FreeBSD-ports/pull/1333 for review and merge by the Netgate developer team.