Suricata process dying due to hyperscan problem
-
@tim_co said in Suricata process dying due to hyperscan problem:
The only errors I see now in the system log are related to Suricata not being able to parse thing files and something about [100107] <Error> -- ja3(s) support is not enabled
That error is harmless for now. It simply means that particular protocol parser is not enabled. It can be ignored.
@tim_co said in Suricata process dying due to hyperscan problem:
If I remove all Suricata rules, the only error I get is the "[100404 - Suricata-Main] 2023-11-16 17:17:22 Notice: threads: Threads created -> RX: 1 W: 4 FM: 1 FR: 1 Engine started.
[100759 - W#04] 2023-11-16 17:17:27 Error: spm-hs: Hyperscan returned fatal error -1."You have absolutely zero rule categories enabled and you are getting that Hyperscan error? That does not seem to be the same bug because rules with regex patterns to be compiled are required to trigger the Hyperscan bug discussed in this thread. The error occurs when the library is actively compiling regex expressions pulled from enabled rules.
-
Suricata has been rock solid for years. Its a remarkable tool. It worked great until I upgraded to 7.0.2 today.
I'm just trying to get Suricats running again by providing data that seems like it might be helpful with troubleshooting the bug and/or my environment. I clearly don't know what I'm doing.
Tim
-
Same issue here, Suricata was fine until update to 7.0.2 (HS 5.4.0), after update it was not starting. Changed Pattern Matcher Algorithm from Auto to AC and its working again. However have 2 other boxes with the 1:1 same Suricata configuration, but they start just fine, only the hardware is different there.
-
@Bismarck said in Suricata process dying due to hyperscan problem:
Same issue here, Suricata was fine until update to 7.0.2 (HS 5.4.0), after update it was not starting. Changed Pattern Matcher Algorithm from Auto to AC and its working again. However have 2 other boxes with the 1:1 same Suricata configuration, but they start just fine, only the hardware is different there.
Can you elaborate on the hardware or other platform differences? It would be helpful to me to know that.
- What processor types are in the other "working" machines?
- Are they bare metal or virtual ?
- What pfSense versions are on the other "working" machines?
- I assume "yes", but are all of the machines ("working" and "not working") running the same Suricata package version?
-
- Intel(R) Xeon(R) CPU E5-2667 and Intel(R) Core(TM) i5-4460 working, Intel(R) Celeron(R) N5105 is not.
- Yes, all bare metal
- All at 2.7.0-RELEASE (amd64)
- Yes all the same version, same config/settings/rules/sid etc.
-
@Bismarck said in Suricata process dying due to hyperscan problem:
- Intel(R) Xeon(R) CPU E5-2667 and Intel(R) Core(TM) i5-4460 working, Intel(R) Celeron(R) N5105 is not.
- Yes, all bare metal
- All at 2.7.0-RELEASE (amd64)
- Yes all the same version, same config/settings/rules/sid etc.
Thanks for the details. Curious the Celeron is not working. I do know that Hyperscan uses some specific Intel CPU instructions, that's why it does not work on ARM or other hardware platforms. But I would think Celeron would be okay.
At any rate, I am going to try and bundle an update to the Hyperscan 5.4.2 library in my next Suricata update.
-
@bmeeks said in Suricata process dying due to hyperscan problem:
At any rate, I am going to try and bundle an update to the Hyperscan 5.4.2 library in my next Suricata update.
I would be grateful to test the new version.
-
Changed mine to AC-KS and it immediately started working again, but it's putting the hurt on the CPU, temp, & memory. Time to zip-tie the fan to the fan-less system once again. Thanks for the tip!
Tim
-
@Bismarck said in Suricata process dying due to hyperscan problem:
@bmeeks said in Suricata process dying due to hyperscan problem:
At any rate, I am going to try and bundle an update to the Hyperscan 5.4.2 library in my next Suricata update.
I would be grateful to test the new version.
Unfortunately, I don't have a way to distribute it.
-
Nevermind. AC-KS also puked. AC has been running fine for a few minutes now.
-
After my device runs for half an hour or so, all of the descriptions for alerts say "SURICATA QUIC error on data". Google doesn't find anything for the string. Would somebody please explain to me what this means?
Edit - All the traffic is from the same IP address which is coming from "Zscaler, Inc." and is UDP Generic Protocol Command Decode 165.225.10.58 443 X,X,X,X 34476 1:2231001 SURICATA QUIC error on data
Thanks - Tim
-
@tim_co said in Suricata process dying due to hyperscan problem:
After my device runs for half an hour or so, all of the descriptions for alerts say "SURICATA QUIC error on data". Google doesn't find anything for the string. Would somebody please explain to me what this means?
Thanks - Tim
Tim:
QUIC is a new web transport protocol based on UDP instead of TCP. Some basic info can be found here: https://en.wikipedia.org/wiki/QUIC.You are seeing those messages because the Suricata built-in Events Rules are enabled. Those are meant to be "informational" rules. They just tell you some particular traffic type is present. That does not mean the traffic is harmful. There is a lot of QUIC out there now, and Suricata is just saying "I see it!". The QUIC rules and detection logic were added to the 7.x branch of Suricata by its upstream developers.
I recommend you go to the CATEGORIES tab and turn off nearly all of the Suricata built-in events rules. Or switch to Block on DROPs Only mode and use SID MGMT tab features to selectively set only some rules or rule categories to DROP (block) traffic and leave the others at their default of ALERT. That will prevent any nuiscan blocks from the information Events Rules. Search the forum here and you will find several posts, including some Sticky Posts at the top of this sub-forum, describing how to do this.
-
Cool. Thanks for the information. It is greatly appreciated.
-
-
@Bismarck said in Suricata process dying due to hyperscan problem:
- Intel(R) Xeon(R) CPU E5-2667 and Intel(R) Core(TM) i5-4460 working, Intel(R) Celeron(R) N5105 is not.
I’m running a N5105, and it’s running on mine with no problem. It was yesterday on 2.7.0 and overnight on 2.7.1
-
@Vollans said in Suricata process dying due to hyperscan problem:
Intel(R) Celeron(R) N5105
The only difference is the machine with the Intel N5105 CPU also has Intel NICs (igc), the othere 2 have broadcoms (bge). I guess thats why they behave different.
Do we see a pattern here?
-
@Bismarck said in Suricata process dying due to hyperscan problem:
@Vollans said in Suricata process dying due to hyperscan problem:
Intel(R) Celeron(R) N5105
The only difference is the machine with the Intel N5105 CPU also has Intel NICs (igc), the othere 2 have broadcoms (bge). I guess thats why they behave different.
Do we see a pattern here?
I can't see any scenario where the type of NIC hardware would have anything to do with Hyperscan. That technology is purely a thing within the CPU. It pre-compiles regex patterns for faster matching using special CPU opcodes that only Intel CPUs have. That's why Hyperscan does not work on non-Intel processors.
-
Updating this thread with the latest as of the late afternoon of November 20, 2023.
I've been mainly focusing efforts on getting the Signal 11 segfault error fixed in the custom blocking modules of both Suricata and Snort. That bug has been located, a fix identified, and Netgate is currently working through the logistics of getting that fix ready to deploy. Still working through a few kinks due to slight differences in some file versions in CE 2.7.1 and Plus 23.09. The fix builds in 2.7.1, but needs some tweaks still to build successfully for 23.09.
Now on to the Hyperscan problem in Suricata. First off, I have unfortunately been unable to duplicate this in my test environment. All I have for testing are VMware Workstation virtual machines, so I can't duplicate everyone's unique hardware. Since I have thus far been unable to reproduce the failure some users are seeing, I can't know what the root cause might be much less verify any potential fix. There will be an updated Suricata package in the very near future to address the Signal 11 segfault discussed in the above paragraph. But that update will NOT contain any known fix for the Hyperscan problem. That's because I don't really know yet what that probem is . So, you will have to be patient for a while as I try to replicate the failure. It appears a viable workaround for now is to choose a Pattern Matcher algorithm other than AUTO or HS (HyperScan). AUTO will always choose HyperScan if that library is on the system, and HS forces the HyperScan choice. So, choose one of the other AC selections if you are hitting the HyperScan error until I can get the problem identified and resolved.
-
@bmeeks said in Suricata process dying due to hyperscan problem:
It appears a viable workaround for now is to choose a Pattern Matcher algorithm other than AUTO or HS (HyperScan).
This only works temporarily for me. Eventually Suricata will still crash. I've tried, AC, AC-KS, and I'm currently trying AC-BS. I'm also trying to see if "Signature Group Header MPM Context" being changed to full affects things. Changing "Detect-Engine Profile" from High to Medium definitely helps keep things running longer across the board. I'll change it down to Low if I keep seeing problems.
Tim
-
I have the same problem under vmware starting today after ugrading to 2.7.1 and suricata to 7.0.2_1
Starting program: /usr/local/bin/suricata -i vmx0 -c /usr/local/etc/suricata/suricata_42366_vmx0/suricata.yaml --pidfile /var/run/suricata_vmx042366.pid Warning: suricata: faster capture option is available: NETMAP (--netmap=vmx0). Use --pcap=vmx0 to suppress this warning Info: conf-yaml-loader: Configuration node 'filetype' redefined. 21/11/2023 -- 23:17:56 - <Notice> -- This is Suricata version 7.0.2 RELEASE running in SYSTEM mode 21/11/2023 -- 23:17:56 - <Info> -- CPUs/cores online: 8 21/11/2023 -- 23:17:56 - <Info> -- Setting engine mode to IDS mode by default 21/11/2023 -- 23:17:56 - <Info> -- HTTP memcap: 67108864 21/11/2023 -- 23:17:56 - <Info> -- Creating automatic firewall interface IP address Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:357e to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0 IPv4 address 192.168.17.1 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0 IPv4 address 192.168.17.3 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx1 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:3560 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx1 IPv4 address ********** to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx1 IPv4 address ********** to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx1 IPv4 address ********** to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx1 IPv4 address ********** to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx1 IPv4 address ********** to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx1 IPv4 address ********** to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx1 IPv4 address ********** to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx2 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:356a to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx2 IPv4 address 192.168.8.7 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx2 IPv4 address 192.168.8.5 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx3 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:3574 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx3 IPv4 address 192.168.4.1 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface lo0 IPv6 address 0000:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface lo0 IPv6 address fe80:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface lo0 IPv4 address 127.0.0.1 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.50 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:357e to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.50 IPv4 address 192.168.12.1 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.50 IPv4 address 192.168.12.3 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.60 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:357e to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.60 IPv4 address 192.168.13.1 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.60 IPv4 address 192.168.13.3 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.70 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:357e to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.70 IPv4 address 192.168.14.1 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.70 IPv4 address 192.168.14.3 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.80 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:357e to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.80 IPv4 address 192.168.15.1 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.80 IPv4 address 192.168.15.3 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx2.100 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:356a to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx2.100 IPv4 address 192.168.16.1 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx2.100 IPv4 address 192.168.16.3 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.90 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:357e to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.90 IPv4 address 192.168.20.1 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.90 IPv4 address 192.168.20.3 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx2.110 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:356a to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx2.110 IPv4 address 192.168.110.7 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx2.110 IPv4 address 192.168.110.5 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface ovpns1 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:357e to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface ovpns1 IPv4 address 192.168.93.1 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface ovpns2 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:357e to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface ovpns2 IPv4 address 192.168.95.1 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface ovpns3 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:357e to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface ovpns3 IPv4 address 192.168.94.1 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface tun_wg0 IPv4 address 192.168.30.1 to automatic interface IP Pass List. 21/11/2023 -- 23:17:56 - <Info> -- alert-pf output device (regular) initialized: block.log 21/11/2023 -- 23:17:56 - <Info> -- Loading and parsing Pass List from: /usr/local/etc/suricata/suricata_42366_vmx0/passlist. 21/11/2023 -- 23:17:56 - <Info> -- Pass List /usr/local/etc/suricata/suricata_42366_vmx0/passlist processed: Total entries parsed: 19, IP addresses/netblocks/aliases added to No Block list: 18, IP addresses/netblocks ignored because they were covered by existing entries: 1. [New LWP 183507 of process 95785] 21/11/2023 -- 23:17:56 - <Info> -- Created Firewall Interface IP Change monitor thread for auto-whitelisting of firewall interface IP addresses. 21/11/2023 -- 23:17:56 - <Info> -- pfSense Suricata Custom Blocking Module initialized: pf-table=snort2c block-ip=both kill-state=yes block-drops-only=yes passlist-debugging=no 21/11/2023 -- 23:17:56 - <Info> -- fast output device (regular) initialized: alerts.log 21/11/2023 -- 23:17:56 - <Info> -- http-log output device (regular) initialized: http.log 21/11/2023 -- 23:17:56 - <Info> -- Firewall Interface IP Address Change monitoring thread IM#01 has successfully started. 21/11/2023 -- 23:17:56 - <Info> -- Syslog output initialized Thread 1 "Suricata-Main" received signal SIGSEGV, Segmentation fault. Address not mapped to object. 0x000000000061d52f in ?? () (gdb) bt #0 0x000000000061d52f in ?? () #1 0x000000000061d286 in AppLayerProtoDetectSupportedIpprotos () #2 0x00000000005c67a3 in ?? () #3 0x00000000005c4f0f in ?? () #4 0x00000000005c4d97 in SigInit () #5 0x00000000005c5707 in DetectEngineAppendSig () #6 0x00000000005d203d in ?? () #7 0x00000000005d1a75 in SigLoadSignatures () #8 0x000000000059c4d6 in PostConfLoadedDetectSetup () #9 0x000000000059f0aa in SuricataMain () #10 0x00000008015f06fa in __libc_start1 () from /lib/libc.so.7 #11 0x000000000059bea0 in _start ()
in my case it seems to be related to how big flowbit-required.rules and suricata.rules are
i have tested putting 2 lines inside both that files and suricata start but it crash as soon as you reload the entire rules list
@bmeeks memory limit somewhere?[2.7.1-RELEASE][root@DTLFWSRV02.dtl.local]/: truss /usr/local/bin/suricata -i vmx0 -c /usr/local/etc/suricata/suricata_42366_vmx0/suricata.yaml --pidfile /var/run/suricata_vmx042366.pid mmap(0x0,135168,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 57140792262656 (0x33f820a00000) mprotect(0xfb9245eb000,4096,PROT_READ) = 0 (0x0) issetugid() = 0 (0x0) sigfastblock(0x1,0xfb9245ee360) = 0 (0x0) __sysctl("kern.ostype",2,0xfb9245edcba,0x820d5f640,0x0,0) = 0 (0x0) __sysctl("kern.hostname",2,0xfb9245eddba,0x820d5f640,0x0,0) = 0 (0x0) __sysctl("kern.osrelease",2,0xfb9245edeba,0x820d5f640,0x0,0) = 0 (0x0) __sysctl("kern.version",2,0xfb9245edfba,0x820d5f640,0x0,0) ERR#12 'Cannot allocate memory' __sysctl("hw.machine",2,0xfb9245ee0ba,0x820d5f640,0x0,0) = 0 (0x0) ...................................... .................................. ................. fstatat(AT_FDCWD,"/usr/local/etc/suricata/suricata_42366_vmx0/rules/suricata.rules",{ mode=-rw-r--r-- ,inode=5698327,size=22184862,blksize=32768 },AT_SYMLINK_NOFOLLOW) = 0 (0x0) open("/usr/local/etc/suricata/suricata_42366_vmx0/rules/suricata.rules",O_RDONLY,0666) = 10 (0xa) fstat(10,{ mode=-rw-r--r-- ,inode=5698327,size=22184862,blksize=32768 }) = 0 (0x0) read(10,"# These rules are your current s"...,32768) = 32768 (0x8000) SIGNAL 11 (SIGSEGV) code=SEGV_MAPERR trapno=12 addr=0x33f87e3571fe read(6,0x832399760,2048) ERESTART <thread 202903 exited> process killed, signal = 11 (core dumped)
-
@kiokoman said in Suricata process dying due to hyperscan problem:
__sysctl("kern.version",2,0xfb9245edfba,0x820d5f640,0x0,0) ERR#12 'Cannot allocate memory'
How much RAM is configured for the virtual machine? That error says Suricata ran out of memory.
One change made in the Suricata 7.x series was to substantially increase the memory sizes for the TCP stream_memcap and a few related parameters. That was done to match up with the new defaults from upstream and to prevent startup failures on multi-core machines. The more CPU cores you have, the more TCP stream_memcap memory you need to allocate.
-