Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Suricata process dying due to hyperscan problem

    Scheduled Pinned Locked Moved
    IDS/IPS
    25
    295
    67.4k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • kiokomanK
      kiokoman LAYER 8 @tim_co
      last edited by kiokoman

      I have the same problem under vmware starting today after ugrading to 2.7.1 and suricata to 7.0.2_1

      Starting program: /usr/local/bin/suricata -i vmx0 -c /usr/local/etc/suricata/suricata_42366_vmx0/suricata.yaml --pidfile /var/run/suricata_vmx042366.pid


Warning: suricata: faster capture option is available: NETMAP (--netmap=vmx0). Use --pcap=vmx0 to suppress this warning
Info: conf-yaml-loader: Configuration node 'filetype' redefined.
21/11/2023 -- 23:17:56 - <Notice> -- This is Suricata version 7.0.2 RELEASE running in SYSTEM mode
21/11/2023 -- 23:17:56 - <Info> -- CPUs/cores online: 8
21/11/2023 -- 23:17:56 - <Info> -- Setting engine mode to IDS mode by default
21/11/2023 -- 23:17:56 - <Info> -- HTTP memcap: 67108864
21/11/2023 -- 23:17:56 - <Info> -- Creating automatic firewall interface IP address Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:357e to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0 IPv4 address 192.168.17.1 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0 IPv4 address 192.168.17.3 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx1 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:3560 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx1 IPv4 address  **********  to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx1 IPv4 address  ********** to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx1 IPv4 address  ********** to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx1 IPv4 address  ********** to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx1 IPv4 address  ********** to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx1 IPv4 address  ********** to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx1 IPv4 address  ********** to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx2 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:356a to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx2 IPv4 address 192.168.8.7 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx2 IPv4 address 192.168.8.5 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx3 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:3574 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx3 IPv4 address 192.168.4.1 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface lo0 IPv6 address 0000:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface lo0 IPv6 address fe80:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface lo0 IPv4 address 127.0.0.1 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.50 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:357e to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.50 IPv4 address 192.168.12.1 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.50 IPv4 address 192.168.12.3 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.60 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:357e to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.60 IPv4 address 192.168.13.1 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.60 IPv4 address 192.168.13.3 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.70 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:357e to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.70 IPv4 address 192.168.14.1 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.70 IPv4 address 192.168.14.3 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.80 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:357e to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.80 IPv4 address 192.168.15.1 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.80 IPv4 address 192.168.15.3 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx2.100 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:356a to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx2.100 IPv4 address 192.168.16.1 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx2.100 IPv4 address 192.168.16.3 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.90 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:357e to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.90 IPv4 address 192.168.20.1 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx0.90 IPv4 address 192.168.20.3 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx2.110 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:356a to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx2.110 IPv4 address 192.168.110.7 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface vmx2.110 IPv4 address 192.168.110.5 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface ovpns1 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:357e to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface ovpns1 IPv4 address 192.168.93.1 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface ovpns2 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:357e to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface ovpns2 IPv4 address 192.168.95.1 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface ovpns3 IPv6 address fe80:0000:0000:0000:020c:29ff:fe71:357e to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface ovpns3 IPv4 address 192.168.94.1 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- Adding firewall interface tun_wg0 IPv4 address 192.168.30.1 to automatic interface IP Pass List.
21/11/2023 -- 23:17:56 - <Info> -- alert-pf output device (regular) initialized: block.log
21/11/2023 -- 23:17:56 - <Info> -- Loading and parsing Pass List from: /usr/local/etc/suricata/suricata_42366_vmx0/passlist.
21/11/2023 -- 23:17:56 - <Info> -- Pass List /usr/local/etc/suricata/suricata_42366_vmx0/passlist processed: Total entries parsed: 19, IP addresses/netblocks/aliases added to No Block list: 18, IP addresses/netblocks ignored because they were covered by existing entries: 1.
[New LWP 183507 of process 95785]
21/11/2023 -- 23:17:56 - <Info> -- Created Firewall Interface IP Change monitor thread for auto-whitelisting of firewall interface IP addresses.
21/11/2023 -- 23:17:56 - <Info> -- pfSense Suricata Custom Blocking Module initialized: pf-table=snort2c  block-ip=both  kill-state=yes  block-drops-only=yes  passlist-debugging=no
21/11/2023 -- 23:17:56 - <Info> -- fast output device (regular) initialized: alerts.log
21/11/2023 -- 23:17:56 - <Info> -- http-log output device (regular) initialized: http.log
21/11/2023 -- 23:17:56 - <Info> -- Firewall Interface IP Address Change monitoring thread IM#01 has successfully started.
21/11/2023 -- 23:17:56 - <Info> -- Syslog output initialized

Thread 1 "Suricata-Main" received signal SIGSEGV, Segmentation fault.
Address not mapped to object.
0x000000000061d52f in ?? ()

(gdb) bt
#0  0x000000000061d52f in ?? ()
#1  0x000000000061d286 in AppLayerProtoDetectSupportedIpprotos ()
#2  0x00000000005c67a3 in ?? ()
#3  0x00000000005c4f0f in ?? ()
#4  0x00000000005c4d97 in SigInit ()
#5  0x00000000005c5707 in DetectEngineAppendSig ()
#6  0x00000000005d203d in ?? ()
#7  0x00000000005d1a75 in SigLoadSignatures ()
#8  0x000000000059c4d6 in PostConfLoadedDetectSetup ()
#9  0x000000000059f0aa in SuricataMain ()
#10 0x00000008015f06fa in __libc_start1 () from /lib/libc.so.7
#11 0x000000000059bea0 in _start ()

      in my case it seems to be related to how big flowbit-required.rules and suricata.rules are
      i have tested putting 2 lines inside both that files and suricata start but it crash as soon as you reload the entire rules list
      @bmeeks memory limit somewhere?

      [2.7.1-RELEASE][root@DTLFWSRV02.dtl.local]/: truss /usr/local/bin/suricata -i vmx0 -c /usr/local/etc/suricata/suricata_42366_vmx0/suricata.yaml --pidfile /var/run/suricata_vmx042366.pid
mmap(0x0,135168,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 57140792262656 (0x33f820a00000)
mprotect(0xfb9245eb000,4096,PROT_READ)           = 0 (0x0)
issetugid()                                      = 0 (0x0)
sigfastblock(0x1,0xfb9245ee360)                  = 0 (0x0)
__sysctl("kern.ostype",2,0xfb9245edcba,0x820d5f640,0x0,0) = 0 (0x0)
__sysctl("kern.hostname",2,0xfb9245eddba,0x820d5f640,0x0,0) = 0 (0x0)
__sysctl("kern.osrelease",2,0xfb9245edeba,0x820d5f640,0x0,0) = 0 (0x0)
__sysctl("kern.version",2,0xfb9245edfba,0x820d5f640,0x0,0) ERR#12 'Cannot allocate memory'
__sysctl("hw.machine",2,0xfb9245ee0ba,0x820d5f640,0x0,0) = 0 (0x0)
......................................
..................................
.................
fstatat(AT_FDCWD,"/usr/local/etc/suricata/suricata_42366_vmx0/rules/suricata.rules",{ mode=-rw-r--r-- ,inode=5698327,size=22184862,blksize=32768 },AT_SYMLINK_NOFOLLOW) = 0 (0x0)
open("/usr/local/etc/suricata/suricata_42366_vmx0/rules/suricata.rules",O_RDONLY,0666) = 10 (0xa)
fstat(10,{ mode=-rw-r--r-- ,inode=5698327,size=22184862,blksize=32768 }) = 0 (0x0)
read(10,"# These rules are your current s"...,32768) = 32768 (0x8000)
SIGNAL 11 (SIGSEGV) code=SEGV_MAPERR trapno=12 addr=0x33f87e3571fe
read(6,0x832399760,2048)                         ERESTART
<thread 202903 exited>
process killed, signal = 11 (core dumped)

      ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
      Please do not use chat/PM to ask for help
      we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
      Don't forget to Upvote with the 👍 button for any post you find to be helpful.

      bmeeksB 1 Reply Last reply Reply Quote 1
      • bmeeksB
        bmeeks @kiokoman
        last edited by bmeeks

        @kiokoman said in Suricata process dying due to hyperscan problem:

        __sysctl("kern.version",2,0xfb9245edfba,0x820d5f640,0x0,0) ERR#12 'Cannot allocate memory'

        How much RAM is configured for the virtual machine? That error says Suricata ran out of memory.

        One change made in the Suricata 7.x series was to substantially increase the memory sizes for the TCP stream_memcap and a few related parameters. That was done to match up with the new defaults from upstream and to prevent startup failures on multi-core machines. The more CPU cores you have, the more TCP stream_memcap memory you need to allocate.

        kiokomanK 1 Reply Last reply Reply Quote 0
        • S sgnoc referenced this topic on
        • S
          sgnoc
          last edited by sgnoc

          I just found this topic after creating my own topic on the same issue (searches for some reason didn't find this result before I submitted).

          I'm having the exact same issue with 2 interfaces with Suricata blocking on legacy mode. Both interfaces have the same message, one with blocks and one without blocks before they both halt with that error.

          I'm running a 7100 with 23.09 and Suricata updated to 7.0.2, but it was occurring on the previous version of Suricata. I just checked and when I updated to Suricata 7.0.2, it still shows hyperscan 5.4.0. Looks like I'll just have to wait for an update to get pushed out for 5.4.2 to resolve this problem.

          Are there any temporary config solutions to get these interfaces back online for monitoring? I'm guessing it is just specific rules that are triggering these to fail. I can go back through the suricata.log files for those 2 interfaces. There were some errors on specific rules, so if I disable each of those, might that do the trick for now? One of these 2 interfaces is my WAN, so a little more important than the others.

          I'm going to give the AC pattern matcher on these 2 interfaces and see if it still crashes. Hopefully that will be a solution for me for the meantime.

          bmeeksB 1 Reply Last reply Reply Quote 0
          • bmeeksB
            bmeeks @sgnoc
            last edited by

            @sgnoc said in Suricata process dying due to hyperscan problem:

            There were some errors on specific rules, so if I disable each of those, might that do the trick for now?

            Unlikely. If you are running any Snort VRT rules with Suricata, then several hundred of those rules (like around 700 or so last time I checked) are incompatible with Suricata and will fail to load. That's because Snort and Suricata are not 100% compatible with their rule syntax. When it encounters such rules, Suricata will log an error and ignore that rule.

            @sgnoc said in Suricata process dying due to hyperscan problem:

            Are there any temporary config solutions to get these interfaces back online for monitoring?

            Changing the MPM pattern matcher alogorithm to something other than Auto or HS should work. If the HyperScan library is present for your system, then the Auto setting will always choose HyperScan. The HS setting forces the selection of HyperScan (if it is available, but remember HyperScan is an Intel creation and only runs on AMD64 CPUs, so not on ARM hardware).

            S 1 Reply Last reply Reply Quote 0
            • bmeeksB bmeeks referenced this topic on
            • S
              sgnoc @bmeeks
              last edited by

              @bmeeks I switched the 2 interfaces with the hyperscan fault to the AC pattern matcher last night, and it has run all night without failing. Everything is still running fine. I'll continue with the AC pattern matcher for now.

              Are there any repercussions for using AC vs hyperscan? I'm guessing hyperscan is better for resources and performance, if it is the default chosen if available?

              One other note, I know you mentioned the hyperscan 5.4.0 was listed by the upstream suricata developer team to be fine, but that is the version on my system, and I'm definitely getting the hyperscan error on 2 interfaces. I'm hopeful that their 5.4.2 version will be a fix, if they believe that 5.4.0 is not affected, but it is on my system with that version.

              Thanks for all your hard work! If I can do anything to help from my end, I would be happy to try. I'm running the XG-7100 netgate hardware.

              S 1 Reply Last reply Reply Quote 0
              • S
                sgnoc @sgnoc
                last edited by sgnoc

                @bmeeks My WAN interface has halted again, but I don't see a log where it failed. These are the last few lines of suricata.log:

                [101805 - Suricata-Main] 2023-11-23 18:32:40 Warning: detect-flowbits: flowbit 'file.pdf&file.ttf' is checked but not set. Checked in 28585 and 1 other sigs
[101805 - Suricata-Main] 2023-11-23 18:32:40 Warning: detect-flowbits: flowbit 'file.xls&file.ole' is checked but not set. Checked in 30990 and 1 other sigs
[101805 - Suricata-Main] 2023-11-23 18:32:40 Warning: detect-flowbits: flowbit 'ET.gadu.loginsent' is checked but not set. Checked in 2008299 and 0 other sigs
[101805 - Suricata-Main] 2023-11-23 18:32:40 Warning: detect-flowbits: flowbit 'file.onenote' is checked but not set. Checked in 61666 and 1 other sigs
[101805 - Suricata-Main] 2023-11-23 18:33:37 Notice: detect: rule reload complete

                This triggered right after the Emerging Threats rules updated and the interface rules reloaded. The pid file is still listed in /var/run, which I guess makes sense since the process halted. There are no processes running for suricata on that interface when I check with "ps aux".

                I was able to find this in the system logs:

                2023-11-23 19:14:37.481228-05:00 	kernel 	- 	pid 4814 (suricata), jid 0, uid 0: exited on signal 10 (core dumped)
2023-11-23 18:33:08.993276-05:00 	php-cgi 	81475 	[Suricata] The Rules update has finished. 
... Other interfaces reloading
2023-11-23 18:30:30.406289-05:00 	php-cgi 	81475 	[Suricata] Suricata signalled with SIGUSR2 for 00_WAN (ix0)...
2023-11-23 18:30:30.400230-05:00 	php-cgi 	81475 	[Suricata] Live-Reload of rules from auto-update is enabled...
2023-11-23 18:30:28.809617-05:00 	php-cgi 	81475 	[Suricata] Building new sid-msg.map file for 00_WAN...
2023-11-23 18:30:28.555533-05:00 	php-cgi 	81475 	[Suricata] Enabling any flowbit-required rules for: 00_WAN...
2023-11-23 18:30:17.944583-05:00 	php-cgi 	81475 	[Suricata] Updating rules configuration for: 00_WAN ...
2023-11-23 18:30:17.493458-05:00 	php-cgi 	81475 	[Suricata] Snort GPLv2 Community Rules are up to date...
2023-11-23 18:30:17.318869-05:00 	php-cgi 	81475 	[Suricata] Snort VRT rules are up to date...
2023-11-23 18:30:17.081341-05:00 	php-cgi 	81475 	[Suricata] Emerging Threats Pro rules file update downloaded successfully.
2023-11-23 18:30:16.694776-05:00 	php-cgi 	81475 	[Suricata] There is a new set of Emerging Threats Pro rules posted. Downloading etpro.rules.tar.gz...

                What troubleshooting options do I have? Is this still related to the same problem, or is this a separate problem from the hyperscan issue that I need to switch to my own topic?

                bmeeksB 2 Replies Last reply Reply Quote 0
                • kiokomanK
                  kiokoman LAYER 8 @bmeeks
                  last edited by

                  @bmeeks
                  8vcpu
                  16gb ram
                  increasing stream memory cap up to 2.147.483.648 didn't help

                  ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                  Please do not use chat/PM to ask for help
                  we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                  Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                  1 Reply Last reply Reply Quote 0
                  • bmeeksB
                    bmeeks @sgnoc
                    last edited by

                    @sgnoc said in Suricata process dying due to hyperscan problem:

                    2023-11-23 19:14:37.481228-05:00 kernel - pid 4814 (suricata), jid 0, uid 0: exited on signal 10 (core dumped)

                    Signal 10 is a bus error normally associated with ARM-based hardware. What kind of machine are you running Suricata on? The Signal 10 error is more commonly associated with a non-aligned memory access, and that really can't happen on anything but ARM hardware these days.

                    S 1 Reply Last reply Reply Quote 0
                    • S
                      sgnoc @bmeeks
                      last edited by

                      @bmeeks I'm using a netgate xg-7100-u, which has an Intel x64 processor, and I have 24 GB of ram installed. It's only the one interface on suricata that has had that error, so I wouldn't think failing memory or other services should be having issues, I would think?

                      1 Reply Last reply Reply Quote 0
                      • bmeeksB
                        bmeeks @sgnoc
                        last edited by bmeeks

                        @sgnoc said in Suricata process dying due to hyperscan problem:

                        This triggered right after the Emerging Threats rules updated and the interface rules reloaded.

                        By my calculations using the log timestamps, Suricata finished the rules update and ran for 41 minutes before crashing, so "right after the rules update" is not entirely correct.

                        2023-11-23 19:14:37.481228-05:00 	kernel 	- 	pid 4814 (suricata), jid 0, uid 0: exited on signal 10 (core dumped)
2023-11-23 18:33:08.993276-05:00 	php-cgi 	81475 	[Suricata] The Rules update has finished.

                        Rules update completed at 18:33:08. That crash happened at 19:14:37, or 41 minutes later.

                        Other helpful information the next time this happens would be the content of the suricata.log file around the same time interval. You would need to capture that log BEFORE you restarted Suricata because that log is wiped clean each time Suricata is started or restarted in the GUI.

                        S 1 Reply Last reply Reply Quote 0
                        • S
                          sgnoc @bmeeks
                          last edited by

                          @bmeeks These are the only logs available in the suricata.log file, and the immediately was reference to it being the next log in line. There was nothing else before the core dump other than the rukes reloading. It has not yet occurred again, so hopefully it is an isolated incident and won't occur again.

                          bmeeksB 1 Reply Last reply Reply Quote 0
                          • bmeeksB
                            bmeeks
                            last edited by

                            For those of you having the Signal 11 or Signal 10 crashes, it would perhaps be useful if you can submit the core dump backtrace.

                            The command to execute at a shell prompt is:

                            gdb /usr/local/bin/suricata /root/suricata.core

                            Then execute these commands within the gdb prompt:

                            (gdb) bt
(gdb) bt full
(gdb) info threads
(gdb) thread apply all bt
(gdb) thread apply all bt full

                            Capture the output of those commands and post it back here.

                            1 Reply Last reply Reply Quote 0
                            • bmeeksB
                              bmeeks @sgnoc
                              last edited by

                              @sgnoc said in Suricata process dying due to hyperscan problem:

                              It has not yet occurred again, so hopefully it is an isolated incident and won't occur again.

                              No, I don't think that is a true statement. It should never have occurred in the first place. The fact it did indicates there is a problem, and so it will happen again. It's only the "when" that is unknown.

                              S 1 Reply Last reply Reply Quote 0
                              • S
                                sgnoc @bmeeks
                                last edited by

                                @bmeeks I know it isnt likely, but can still be hopeful. I'll run the core dump commands on the next crash so I can provide them the next time it happens. Thanks for your help!

                                kiokomanK 1 Reply Last reply Reply Quote 0
                                • kiokomanK
                                  kiokoman LAYER 8 @sgnoc
                                  last edited by

                                  After the last error i decided to uninstall everything and reconfigure from scratch
                                  maybe some configuration didn't migrate correctly
                                  now i'm unable to reproduce the error at start

                                  ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                                  Please do not use chat/PM to ask for help
                                  we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                                  Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                                  bmeeksB S 2 Replies Last reply Reply Quote 0
                                  • bmeeksB
                                    bmeeks @kiokoman
                                    last edited by

                                    @kiokoman said in Suricata process dying due to hyperscan problem:

                                    After the last error i decided to uninstall everything and reconfigure from scratch
                                    maybe some configuration didn't migrate correctly
                                    now i'm unable to reproduce the error at start

                                    This has been the experience of a few other users as well all the way back to the original release of 7.x Suricata in pfSense. That's what makes this such a maddeningly difficult thing to debug 🤔.

                                    1 Reply Last reply Reply Quote 0
                                    • bmeeksB bmeeks referenced this topic on
                                    • bmeeksB
                                      bmeeks
                                      last edited by

                                      I am continuing to look into this issue. Just sent a new batch of emails to the Suricata development team with questions about some recent changes in this area of the Suricata binary's code.

                                      Still would be nice if I could reliably reproduce this in my test machines with a debug image running.

                                      1 Reply Last reply Reply Quote 0
                                      • bmeeksB
                                        bmeeks
                                        last edited by bmeeks

                                        Attention Users hitting the Suricata Hyperscan problem (or other mysterious Suricata stoppages):

                                        To help in pinning down what this problem is, please collect the following information for me when you experience the crash and include it in your post or feedback.

                                        1. Are you seeing a Signal 11 or Signal 10 error fault logged in the pfSense system log (under STATUS > SYSTEM LOGS) around the time Suricata crashed? If so, include those log entries in your report.

                                        2. Before attempting to restart Suricata after finding it stopped or crashed, examine the suricata.log for the interface under the LOGS VIEW tab in the Suricata GUI. Examine that log for any errors mentioning "hyperscan". Include those in your report.

                                        I am trying to determine if a Signal 11 or Signal 10 happens each time Suricata crashes, or if Suricata is sometimes just stopping on its own when it encounters an internal hyperscan error.

                                        Please provide the information requested above when posting about this issue. It is not helpful at all to simply create a reply saying "I'm having this problem, too" with no additional helpful information.

                                        And at this time there is no indication at all the hyperscan crash issue is related to the Legacy Blocking Mode bug shared with Snort. That bug has, I'm fairly confident, been fixed. I think the issue in this thread is something different.

                                        1 Reply Last reply Reply Quote 0
                                        • S
                                          SteveITS Rebel Alliance @kiokoman
                                          last edited by

                                          @kiokoman Do have a backup config 1) from before upgrading, 2) that wasn’t working and 3) after rebuilding? Might be interesting to compare the Suricata section to see if anything is different across those.

                                          (I usually save one just before upgrading and immediately after)

                                          Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                                          When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                                          Upvote 👍 helpful posts!

                                          kiokomanK 1 Reply Last reply Reply Quote 0
                                          • kiokomanK
                                            kiokoman LAYER 8 @SteveITS
                                            last edited by kiokoman

                                            @SteveITS
                                            i have the backup history,
                                            the only difference after reconfiguration was

                                            old not working config:

                                            <stream_bypass>off</stream_bypass>
                                            <stream_drop_invalid>off</stream_drop_invalid>

                                            vs
                                            new config
                                            <stream_bypass>no</stream_bypass>
                                            <stream_drop_invalid>no</stream_drop_invalid>

                                            everything else is the same but i don't have the old generated suricata.yaml

                                            anyway i have a new problem now, before it was not even starting, now i have this after some hours, and only on one interface (i have suricata running on wan and lan, wan(vmx1) is still running ok)

                                            [843086 - W#01-vmx2] 2023-11-25 12:03:58 Info: pcap: vmx2: running in 'auto' checksum mode. Detection of interface state will require 1000 packets
[843086 - W#01-vmx2] 2023-11-25 12:03:58 Info: pcap: vmx2: snaplen set to 1518
[100515 - Suricata-Main] 2023-11-25 12:03:58 Notice: threads: Threads created -> W: 1 FM: 1 FR: 1   Engine started.
[843086 - W#01-vmx2] 2023-11-25 12:03:59 Info: checksum: No packets with invalid checksum, assuming checksum offloading is NOT used
[843086 - W#01-vmx2] 2023-11-25 12:05:02 Error: spm-hs: Hyperscan returned fatal error -1.

                                            ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                                            Please do not use chat/PM to ask for help
                                            we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                                            Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                                            bmeeksB 2 Replies Last reply Reply Quote 0
                                            • First post
                                              Last post