Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Suricata process dying due to hyperscan problem

    IDS/IPS
    25
    295
    64.9k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      chrysmon @ajohnson353
      last edited by

      @ajohnson353 said in Suricata process dying due to hyperscan problem:

      @chrysmon I am seeing the same thing in AC mode. It has yet to die since making the switch.

      If I remember well, mine was not working in AC mode. Let it run for longer time to be sure.

      1 Reply Last reply Reply Quote 0
      • bmeeksB
        bmeeks
        last edited by bmeeks

        Wonder if this might be the source of the mysterious Hyperscan bug we are seeing in Suricata?

        https://www.freebsd.org/security/advisories/FreeBSD-EN-23:15.sanitizer.asc

        If so, that would explain a lot of the weirdness. I will keep tabs on this. Thanks to @RobbieTT for the link in another thread unreleated to Suricata.

        M S 2 Replies Last reply Reply Quote 1
        • M
          masons @bmeeks
          last edited by

          @bmeeks said in Suricata process dying due to hyperscan problem:

          Wonder if this might be the source of the mysterious Hyperscan bug we are seeing in Suricata?

          https://www.freebsd.org/security/advisories/FreeBSD-EN-23:15.sanitizer.asc

          @bmeeks,

          The two machines I posted about earlier, are both running with the default hyperscan enabled and with legacy blocking mode enabled. Both machines have not experienced a Suricata core dump since I disabled ASLR for the Suricata binary. Thus it seems increasingly plausible that the root of the issue is linked to ASLR and the link above about the LLVM sanitizer could certainly explain why this has suddenly happened.

          kiokomanK 1 Reply Last reply Reply Quote 0
          • kiokomanK
            kiokoman LAYER 8 @masons
            last edited by

            @bmeeks
            proccontrol -m aslr -s disable /usr/local/bin/suricata -i vmx2 -D -c /usr/local/etc/suricata/suricata_28559_vmx2/suricata.yaml --pidfile /var/run/suricata_vmx228559.pid

            [1022863 - RX#01-vmx2] 2023-12-03 20:36:32 Info: pcap: vmx2: snaplen set to 1518
            [600532 - Suricata-Main] 2023-12-03 20:36:32 Notice: threads: Threads created -> RX: 1 W: 8 FM: 1 FR: 1 Engine started.
            [1022863 - RX#01-vmx2] 2023-12-03 20:36:35 Info: checksum: No packets with invalid checksum, assuming checksum offloading is NOT used
            [1022865 - W#02] 2023-12-03 20:36:41 Error: spm-hs: Hyperscan returned fatal error -1.
            [1022866 - W#03] 2023-12-03 20:36:41 Error: spm-hs: Hyperscan returned fatal error -1.

            no luck for me

            ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
            Please do not use chat/PM to ask for help
            we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
            Don't forget to Upvote with the 👍 button for any post you find to be helpful.

            1 Reply Last reply Reply Quote 0
            • S
              sgnoc
              last edited by

              I tried to disable the ASLR on my system to test, but it caused the whole system to become unresponsive and I had to do a forced power cycle and revert back. Not sure what happened, since the logs only show suricata coming back online on the interfaces and then no logs until the reboot. I shut down the suricata processes before running "elfctl -e +noaslr /usr/local/bin/suricata" and then restarted the suricata interfaces. So no luck on my end testing with disabled ASLR.

              1 Reply Last reply Reply Quote 0
              • S
                SteveITS Rebel Alliance @bmeeks
                last edited by

                @bmeeks I noticed this in another thread:

                @jimp said in Major DNS Bug 23.01 with Quad9 on SSL:

                While we are likely to include the patch from that EN in future builds it isn't relevant to Unbound.

                They only use those sanitizers for debug/test builds and not for normal/production builds.

                Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                Upvote 👍 helpful posts!

                1 Reply Last reply Reply Quote 0
                • JonathanLeeJ
                  JonathanLee
                  last edited by JonathanLee

                  Hello fellow Netgate community members,

                  I recently learned that Snort and Suricata's maintainer does all this work you what you see here unpaid. I opened a ticket to have a Wikipedia type donate to maintainer button on all 3rd party packages. I personally want to send some money to maintainers. If you also feel the same way please respond to this Redmine.

                  https://redmine.pfsense.org/issues/15056

                  Happy Holidays.

                  Make sure to upvote

                  1 Reply Last reply Reply Quote 1
                  • P
                    paulp
                    last edited by

                    Hello, I also have the same problem, I tried all the recommendations proposed on this topic, but without success.
                    However, I noticed that the problem only occurs on the network interfaces with higher traffic and not on the interfaces with low traffic.

                    I have a Pfsense 23.09 installed on a Hyper-V machine, so no VLANS.
                    There are 6 network cards (without VLANS).

                    Initially Suricata was active only on the WAN and it deactivates after an interval between 1 hour and 7-8 hours.
                    For testing I enabled Suricata on all interfaces. And I noticed that now it stops only on the WAN interface and the LAN interface on which the network traffic is higher. Suricata never stops on the interfaces where the traffic is very low.

                    1 Reply Last reply Reply Quote 0
                    • tgm-itsT
                      tgm-its
                      last edited by tgm-its

                      Hi all.
                      It seems I get the same bugger of an error and Suricata stopes running on (yes you guessed it) one interface.
                      I've read through most of the post here without finding a solution.
                      Is there a solution? Or?

                      So what is next? A patch? Or an Update?

                      I'm on pFSense 2.7.1 and Suricata 7.0.2_1.

                      And yes, is there a way to turn off e Hyperscan system so that Suricata runs without it?

                      Paal B.

                      kiokomanK bmeeksB 2 Replies Last reply Reply Quote 1
                      • kiokomanK
                        kiokoman LAYER 8 @tgm-its
                        last edited by

                        @tgm-its
                        use AC-BS instead of Auto
                        37ffe775-ae7f-406c-93a3-3fbc0b0b271e-image.png

                        ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                        Please do not use chat/PM to ask for help
                        we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                        Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                        tgm-itsT 1 Reply Last reply Reply Quote 1
                        • bmeeksB
                          bmeeks @tgm-its
                          last edited by

                          @tgm-its said in Suricata process dying due to hyperscan problem:

                          And yes, is there a way to turn off e Hyperscan system so that Suricata runs without it?

                          Currently no patch is available because the cause of the failure has yet to be determined. I cannot reproduce it in my limited test environment. At least I have not yet encountered it. It is a seemingly random sort of bug impacting some users and not others. Very difficult to troubleshoot a failure that you cannot reliably reproduce, but I'm still trying.

                          You can disable the use of the Hyperscan library by going to the INTERFACE SETTINGS page for an interface and scrolling down to the Pattern Matcher Algorithm parameter and choosing something other than Auto or Hyperscan. Save the change and restart Suricata on the interface so that the underlying binary sees the new parameter choice.

                          P 1 Reply Last reply Reply Quote 0
                          • P
                            paulp @bmeeks
                            last edited by

                            @bmeeks
                            For me, the problem always manifests itself only on the network interfaces on which there is some traffic. The problem does not appear on network interfaces with low traffic.
                            The problem appears after an interval between 1-8 hours, probably the higher the traffic, the faster the problem appears.

                            bmeeksB 1 Reply Last reply Reply Quote 1
                            • bmeeksB
                              bmeeks @paulp
                              last edited by

                              @paulp said in Suricata process dying due to hyperscan problem:

                              @bmeeks
                              For me, the problem always manifests itself only on the network interfaces on which there is some traffic. The problem does not appear on network interfaces with low traffic.
                              The problem appears after an interval between 1-8 hours, probably the higher the traffic, the faster the problem appears.

                              Hmm...that does make some sense. I am testing with very low traffic in a limited VMware Workstation virtual machine setup. Perhaps hammering a test machine with iperf3 traffic might trigger the bug ???

                              kiokomanK 1 Reply Last reply Reply Quote 0
                              • tgm-itsT
                                tgm-its @kiokoman
                                last edited by

                                @kiokoman Thanks for the suggestion. I'll try that.

                                Paal B.

                                1 Reply Last reply Reply Quote 0
                                • kiokomanK
                                  kiokoman LAYER 8 @bmeeks
                                  last edited by

                                  @bmeeks
                                  idk if iperf3 can generate enough alert to trigger the bug maybe it's more like how many alert there are more than how mutch data is in the wire

                                  ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                                  Please do not use chat/PM to ask for help
                                  we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                                  Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                                  1 Reply Last reply Reply Quote 0
                                  • S
                                    sgnoc
                                    last edited by

                                    @bmeeks I was trying to use the AC-KS pattern matcher as an alternative to hyperscan, but they must be related in some manner. This is the log I receive after just a few minutes running AC-KS. Also, I'm getting the hyperscan errors on my WAN interface, which remains busy, but also on this interface that has little traffic and does not alert all that often, so it isn't just interfaces that are more active on my network.

                                    Here is the last few lines in suricata.log for the failed interface running with AC-KS:

                                    [186263 - RX#01-ix1.15] 2023-12-07 09:37:41 Info: pcap: ix1.15: running in 'auto' checksum mode. Detection of interface state will require 1000 packets
[186263 - RX#01-ix1.15] 2023-12-07 09:37:41 Info: pcap: ix1.15: snaplen set to 1518
[100242 - Suricata-Main] 2023-12-07 09:37:41 Notice: threads: Threads created -> RX: 1 W: 4 FM: 1 FR: 1   Engine started.
[186263 - RX#01-ix1.15] 2023-12-07 09:41:10 Info: checksum: No packets with invalid checksum, assuming checksum offloading is NOT used
[186267 - W#04] 2023-12-07 09:46:00 Error: spm-hs: Hyperscan returned fatal error -1.

                                    Here are the output from the core dump:

                                    (gdb) bt
#0  0x0000000830c6834a in thr_kill () from /lib/libc.so.7
#1  0x0000000830be8344 in raise () from /lib/libc.so.7
#2  0x0000000830c8ca39 in abort () from /lib/libc.so.7
#3  0x0000000830cdbf30 in ?? () from /lib/libc.so.7
#4  0x0000000830ca8440 in ?? () from /lib/libc.so.7
#5  0x0000000830ca142c in ?? () from /lib/libc.so.7
#6  0x0000000830ca1159 in ?? () from /lib/libc.so.7
#7  0x00000000005ac55a in ConfNodeFree ()
#8  0x00000000005ac536 in ConfNodeFree ()
#9  0x00000000005ac536 in ConfNodeFree ()
#10 0x00000000005ad305 in ConfDeInit ()
#11 0x000000000058fa61 in ?? ()
#12 0x000000000058f03f in SuricataMain ()
#13 0x0000000830bbf75a in __libc_start1 () from /lib/libc.so.7
#14 0x000000000058bd40 in _start ()

                                    (gdb) bt full
#0  0x0000000830c6834a in thr_kill () from /lib/libc.so.7
No symbol table info available.
#1  0x0000000830be8344 in raise () from /lib/libc.so.7
No symbol table info available.
#2  0x0000000830c8ca39 in abort () from /lib/libc.so.7
No symbol table info available.
#3  0x0000000830cdbf30 in ?? () from /lib/libc.so.7
No symbol table info available.
#4  0x0000000830ca8440 in ?? () from /lib/libc.so.7
No symbol table info available.
#5  0x0000000830ca142c in ?? () from /lib/libc.so.7
No symbol table info available.
#6  0x0000000830ca1159 in ?? () from /lib/libc.so.7
No symbol table info available.
#7  0x00000000005ac55a in ConfNodeFree ()
No symbol table info available.
#8  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#9  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#10 0x00000000005ad305 in ConfDeInit ()
No symbol table info available.
#11 0x000000000058fa61 in ?? ()
No symbol table info available.
#12 0x000000000058f03f in SuricataMain ()
No symbol table info available.
#13 0x0000000830bbf75a in __libc_start1 () from /lib/libc.so.7
No symbol table info available.
#14 0x000000000058bd40 in _start ()
No symbol table info available.

                                    (gdb) info threads
  Id   Target Id         Frame
* 1    LWP 102018        0x0000000830c6834a in thr_kill () from /lib/libc.so.7

                                    (gdb) thread apply all bt

Thread 1 (LWP 102018):
#0  0x0000000830c6834a in thr_kill () from /lib/libc.so.7
#1  0x0000000830be8344 in raise () from /lib/libc.so.7
#2  0x0000000830c8ca39 in abort () from /lib/libc.so.7
#3  0x0000000830cdbf30 in ?? () from /lib/libc.so.7
#4  0x0000000830ca8440 in ?? () from /lib/libc.so.7
#5  0x0000000830ca142c in ?? () from /lib/libc.so.7
#6  0x0000000830ca1159 in ?? () from /lib/libc.so.7
#7  0x00000000005ac55a in ConfNodeFree ()
#8  0x00000000005ac536 in ConfNodeFree ()
#9  0x00000000005ac536 in ConfNodeFree ()
#10 0x00000000005ad305 in ConfDeInit ()
#11 0x000000000058fa61 in ?? ()
#12 0x000000000058f03f in SuricataMain ()
#13 0x0000000830bbf75a in __libc_start1 () from /lib/libc.so.7
#14 0x000000000058bd40 in _start ()

                                    (gdb) thread apply all bt full

Thread 1 (LWP 102018):
#0  0x0000000830c6834a in thr_kill () from /lib/libc.so.7
No symbol table info available.
#1  0x0000000830be8344 in raise () from /lib/libc.so.7
No symbol table info available.
#2  0x0000000830c8ca39 in abort () from /lib/libc.so.7
No symbol table info available.
#3  0x0000000830cdbf30 in ?? () from /lib/libc.so.7
No symbol table info available.
#4  0x0000000830ca8440 in ?? () from /lib/libc.so.7
No symbol table info available.
#5  0x0000000830ca142c in ?? () from /lib/libc.so.7
No symbol table info available.
#6  0x0000000830ca1159 in ?? () from /lib/libc.so.7
No symbol table info available.
#7  0x00000000005ac55a in ConfNodeFree ()
No symbol table info available.
#8  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#9  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#10 0x00000000005ad305 in ConfDeInit ()
No symbol table info available.
#11 0x000000000058fa61 in ?? ()
No symbol table info available.
#12 0x000000000058f03f in SuricataMain ()
No symbol table info available.
#13 0x0000000830bbf75a in __libc_start1 () from /lib/libc.so.7
No symbol table info available.
#14 0x000000000058bd40 in _start ()
No symbol table info available.

                                    I'll try and run this interface on auto (hyperscan) and catpure the core dump to post the debug details here. Again, this is on my smaller interface that has been failing with suricata. If that doesn't prove as beneficial, I'll work on getting my wan interface to produce the core dump with hyperscan to get you that debug info.

                                    bmeeksB 1 Reply Last reply Reply Quote 0
                                    • bmeeksB
                                      bmeeks @sgnoc
                                      last edited by bmeeks

                                      @sgnoc said in Suricata process dying due to hyperscan problem:

                                      @bmeeks I was trying to use the AC-KS pattern matcher as an alternative to hyperscan, but they must be related in some manner. This is the log I receive after just a few minutes running AC-KS. Also, I'm getting the hyperscan errors on my WAN interface, which remains busy, but also on this interface that has little traffic and does not alert all that often, so it isn't just interfaces that are more active on my network.

                                      Here is the last few lines in suricata.log for the failed interface running with AC-KS:

                                      [186263 - RX#01-ix1.15] 2023-12-07 09:37:41 Info: pcap: ix1.15: running in 'auto' checksum mode. Detection of interface state will require 1000 packets
[186263 - RX#01-ix1.15] 2023-12-07 09:37:41 Info: pcap: ix1.15: snaplen set to 1518
[100242 - Suricata-Main] 2023-12-07 09:37:41 Notice: threads: Threads created -> RX: 1 W: 4 FM: 1 FR: 1   Engine started.
[186263 - RX#01-ix1.15] 2023-12-07 09:41:10 Info: checksum: No packets with invalid checksum, assuming checksum offloading is NOT used
[186267 - W#04] 2023-12-07 09:46:00 Error: spm-hs: Hyperscan returned fatal error -1.

                                      Here are the output from the core dump:

                                      (gdb) bt
#0  0x0000000830c6834a in thr_kill () from /lib/libc.so.7
#1  0x0000000830be8344 in raise () from /lib/libc.so.7
#2  0x0000000830c8ca39 in abort () from /lib/libc.so.7
#3  0x0000000830cdbf30 in ?? () from /lib/libc.so.7
#4  0x0000000830ca8440 in ?? () from /lib/libc.so.7
#5  0x0000000830ca142c in ?? () from /lib/libc.so.7
#6  0x0000000830ca1159 in ?? () from /lib/libc.so.7
#7  0x00000000005ac55a in ConfNodeFree ()
#8  0x00000000005ac536 in ConfNodeFree ()
#9  0x00000000005ac536 in ConfNodeFree ()
#10 0x00000000005ad305 in ConfDeInit ()
#11 0x000000000058fa61 in ?? ()
#12 0x000000000058f03f in SuricataMain ()
#13 0x0000000830bbf75a in __libc_start1 () from /lib/libc.so.7
#14 0x000000000058bd40 in _start ()

                                      (gdb) bt full
#0  0x0000000830c6834a in thr_kill () from /lib/libc.so.7
No symbol table info available.
#1  0x0000000830be8344 in raise () from /lib/libc.so.7
No symbol table info available.
#2  0x0000000830c8ca39 in abort () from /lib/libc.so.7
No symbol table info available.
#3  0x0000000830cdbf30 in ?? () from /lib/libc.so.7
No symbol table info available.
#4  0x0000000830ca8440 in ?? () from /lib/libc.so.7
No symbol table info available.
#5  0x0000000830ca142c in ?? () from /lib/libc.so.7
No symbol table info available.
#6  0x0000000830ca1159 in ?? () from /lib/libc.so.7
No symbol table info available.
#7  0x00000000005ac55a in ConfNodeFree ()
No symbol table info available.
#8  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#9  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#10 0x00000000005ad305 in ConfDeInit ()
No symbol table info available.
#11 0x000000000058fa61 in ?? ()
No symbol table info available.
#12 0x000000000058f03f in SuricataMain ()
No symbol table info available.
#13 0x0000000830bbf75a in __libc_start1 () from /lib/libc.so.7
No symbol table info available.
#14 0x000000000058bd40 in _start ()
No symbol table info available.

                                      (gdb) info threads
  Id   Target Id         Frame
* 1    LWP 102018        0x0000000830c6834a in thr_kill () from /lib/libc.so.7

                                      (gdb) thread apply all bt

Thread 1 (LWP 102018):
#0  0x0000000830c6834a in thr_kill () from /lib/libc.so.7
#1  0x0000000830be8344 in raise () from /lib/libc.so.7
#2  0x0000000830c8ca39 in abort () from /lib/libc.so.7
#3  0x0000000830cdbf30 in ?? () from /lib/libc.so.7
#4  0x0000000830ca8440 in ?? () from /lib/libc.so.7
#5  0x0000000830ca142c in ?? () from /lib/libc.so.7
#6  0x0000000830ca1159 in ?? () from /lib/libc.so.7
#7  0x00000000005ac55a in ConfNodeFree ()
#8  0x00000000005ac536 in ConfNodeFree ()
#9  0x00000000005ac536 in ConfNodeFree ()
#10 0x00000000005ad305 in ConfDeInit ()
#11 0x000000000058fa61 in ?? ()
#12 0x000000000058f03f in SuricataMain ()
#13 0x0000000830bbf75a in __libc_start1 () from /lib/libc.so.7
#14 0x000000000058bd40 in _start ()

                                      (gdb) thread apply all bt full

Thread 1 (LWP 102018):
#0  0x0000000830c6834a in thr_kill () from /lib/libc.so.7
No symbol table info available.
#1  0x0000000830be8344 in raise () from /lib/libc.so.7
No symbol table info available.
#2  0x0000000830c8ca39 in abort () from /lib/libc.so.7
No symbol table info available.
#3  0x0000000830cdbf30 in ?? () from /lib/libc.so.7
No symbol table info available.
#4  0x0000000830ca8440 in ?? () from /lib/libc.so.7
No symbol table info available.
#5  0x0000000830ca142c in ?? () from /lib/libc.so.7
No symbol table info available.
#6  0x0000000830ca1159 in ?? () from /lib/libc.so.7
No symbol table info available.
#7  0x00000000005ac55a in ConfNodeFree ()
No symbol table info available.
#8  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#9  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#10 0x00000000005ad305 in ConfDeInit ()
No symbol table info available.
#11 0x000000000058fa61 in ?? ()
No symbol table info available.
#12 0x000000000058f03f in SuricataMain ()
No symbol table info available.
#13 0x0000000830bbf75a in __libc_start1 () from /lib/libc.so.7
No symbol table info available.
#14 0x000000000058bd40 in _start ()
No symbol table info available.

                                      I'll try and run this interface on auto (hyperscan) and catpure the core dump to post the debug details here. Again, this is on my smaller interface that has been failing with suricata. If that doesn't prove as beneficial, I'll work on getting my wan interface to produce the core dump with hyperscan to get you that debug info.

                                      Thank you for this info. Weird that it seems to be crashing in a section of code where Suricata is releasing memory and cleaning up after reading the configuration from the suricata.yaml file. Those calls to ConfNodeFree() and ConfNodeInit() are made when finishing and cleaning up from reading the YAML conf file parameters.

                                      Let's see if another crash happens in the same area of code. One thing I noticed before (but that might have been with the Kill States bug and not Hyperscan) was the crash happened in seemingly random and unrelated places.

                                      Please continue posting the relevant sections of any future core dumps' backtrace as you did here.

                                      1 Reply Last reply Reply Quote 0
                                      • S
                                        sgnoc
                                        last edited by

                                        @bmeeks I switched my WAN interface to auto. Here are the logs:

                                        (gdb) bt
#0  0x000000082ff3f34a in thr_kill () from /lib/libc.so.7
#1  0x000000082febf344 in raise () from /lib/libc.so.7
#2  0x000000082ff63a39 in abort () from /lib/libc.so.7
#3  0x000000082ffb2f30 in ?? () from /lib/libc.so.7
#4  0x000000082ff7f440 in ?? () from /lib/libc.so.7
#5  0x000000082ff7842c in ?? () from /lib/libc.so.7
#6  0x000000082ff78159 in ?? () from /lib/libc.so.7
#7  0x00000000005ac54c in ConfNodeFree ()
#8  0x00000000005ac536 in ConfNodeFree ()
#9  0x00000000005ac536 in ConfNodeFree ()
#10 0x00000000005ad305 in ConfDeInit ()
#11 0x000000000058fa61 in ?? ()
#12 0x000000000058f03f in SuricataMain ()
#13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7
#14 0x000000000058bd40 in _start ()```

                                        (gdb) bt full
#0  0x000000082ff3f34a in thr_kill () from /lib/libc.so.7
No symbol table info available.
#1  0x000000082febf344 in raise () from /lib/libc.so.7
No symbol table info available.
#2  0x000000082ff63a39 in abort () from /lib/libc.so.7
No symbol table info available.
#3  0x000000082ffb2f30 in ?? () from /lib/libc.so.7
No symbol table info available.
#4  0x000000082ff7f440 in ?? () from /lib/libc.so.7
No symbol table info available.
#5  0x000000082ff7842c in ?? () from /lib/libc.so.7
No symbol table info available.
#6  0x000000082ff78159 in ?? () from /lib/libc.so.7
No symbol table info available.
#7  0x00000000005ac54c in ConfNodeFree ()
No symbol table info available.
#8  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#9  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#10 0x00000000005ad305 in ConfDeInit ()
No symbol table info available.
#11 0x000000000058fa61 in ?? ()
No symbol table info available.
#12 0x000000000058f03f in SuricataMain ()
No symbol table info available.
#13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7
No symbol table info available.
#14 0x000000000058bd40 in _start ()
No symbol table info available.

                                        (gdb) info threads
  Id   Target Id         Frame
* 1    LWP 103117        0x000000082ff3f34a in thr_kill () from /lib/libc.so.7

                                        (gdb) thread apply all bt

Thread 1 (LWP 103117):
#0  0x000000082ff3f34a in thr_kill () from /lib/libc.so.7
#1  0x000000082febf344 in raise () from /lib/libc.so.7
#2  0x000000082ff63a39 in abort () from /lib/libc.so.7
#3  0x000000082ffb2f30 in ?? () from /lib/libc.so.7
#4  0x000000082ff7f440 in ?? () from /lib/libc.so.7
#5  0x000000082ff7842c in ?? () from /lib/libc.so.7
#6  0x000000082ff78159 in ?? () from /lib/libc.so.7
#7  0x00000000005ac54c in ConfNodeFree ()
#8  0x00000000005ac536 in ConfNodeFree ()
#9  0x00000000005ac536 in ConfNodeFree ()
#10 0x00000000005ad305 in ConfDeInit ()
#11 0x000000000058fa61 in ?? ()
#12 0x000000000058f03f in SuricataMain ()
#13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7
#14 0x000000000058bd40 in _start ()

                                        (gdb) thread apply all bt full

Thread 1 (LWP 103117):
#0  0x000000082ff3f34a in thr_kill () from /lib/libc.so.7
No symbol table info available.
#1  0x000000082febf344 in raise () from /lib/libc.so.7
No symbol table info available.
#2  0x000000082ff63a39 in abort () from /lib/libc.so.7
No symbol table info available.
#3  0x000000082ffb2f30 in ?? () from /lib/libc.so.7
No symbol table info available.
#4  0x000000082ff7f440 in ?? () from /lib/libc.so.7
No symbol table info available.
#5  0x000000082ff7842c in ?? () from /lib/libc.so.7
No symbol table info available.
#6  0x000000082ff78159 in ?? () from /lib/libc.so.7
No symbol table info available.
#7  0x00000000005ac54c in ConfNodeFree ()
No symbol table info available.
#8  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#9  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#10 0x00000000005ad305 in ConfDeInit ()
No symbol table info available.
#11 0x000000000058fa61 in ?? ()
No symbol table info available.
#12 0x000000000058f03f in SuricataMain ()
No symbol table info available.
#13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7
No symbol table info available.
#14 0x000000000058bd40 in _start ()
No symbol table info available.
                                        bmeeksB 1 Reply Last reply Reply Quote 0
                                        • bmeeksB
                                          bmeeks @sgnoc
                                          last edited by

                                          @sgnoc said in Suricata process dying due to hyperscan problem:

                                          @bmeeks I switched my WAN interface to auto. Here are the logs:

                                          (gdb) bt
#0  0x000000082ff3f34a in thr_kill () from /lib/libc.so.7
#1  0x000000082febf344 in raise () from /lib/libc.so.7
#2  0x000000082ff63a39 in abort () from /lib/libc.so.7
#3  0x000000082ffb2f30 in ?? () from /lib/libc.so.7
#4  0x000000082ff7f440 in ?? () from /lib/libc.so.7
#5  0x000000082ff7842c in ?? () from /lib/libc.so.7
#6  0x000000082ff78159 in ?? () from /lib/libc.so.7
#7  0x00000000005ac54c in ConfNodeFree ()
#8  0x00000000005ac536 in ConfNodeFree ()
#9  0x00000000005ac536 in ConfNodeFree ()
#10 0x00000000005ad305 in ConfDeInit ()
#11 0x000000000058fa61 in ?? ()
#12 0x000000000058f03f in SuricataMain ()
#13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7
#14 0x000000000058bd40 in _start ()```

                                          (gdb) bt full
#0  0x000000082ff3f34a in thr_kill () from /lib/libc.so.7
No symbol table info available.
#1  0x000000082febf344 in raise () from /lib/libc.so.7
No symbol table info available.
#2  0x000000082ff63a39 in abort () from /lib/libc.so.7
No symbol table info available.
#3  0x000000082ffb2f30 in ?? () from /lib/libc.so.7
No symbol table info available.
#4  0x000000082ff7f440 in ?? () from /lib/libc.so.7
No symbol table info available.
#5  0x000000082ff7842c in ?? () from /lib/libc.so.7
No symbol table info available.
#6  0x000000082ff78159 in ?? () from /lib/libc.so.7
No symbol table info available.
#7  0x00000000005ac54c in ConfNodeFree ()
No symbol table info available.
#8  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#9  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#10 0x00000000005ad305 in ConfDeInit ()
No symbol table info available.
#11 0x000000000058fa61 in ?? ()
No symbol table info available.
#12 0x000000000058f03f in SuricataMain ()
No symbol table info available.
#13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7
No symbol table info available.
#14 0x000000000058bd40 in _start ()
No symbol table info available.

                                          (gdb) info threads
  Id   Target Id         Frame
* 1    LWP 103117        0x000000082ff3f34a in thr_kill () from /lib/libc.so.7

                                          (gdb) thread apply all bt

Thread 1 (LWP 103117):
#0  0x000000082ff3f34a in thr_kill () from /lib/libc.so.7
#1  0x000000082febf344 in raise () from /lib/libc.so.7
#2  0x000000082ff63a39 in abort () from /lib/libc.so.7
#3  0x000000082ffb2f30 in ?? () from /lib/libc.so.7
#4  0x000000082ff7f440 in ?? () from /lib/libc.so.7
#5  0x000000082ff7842c in ?? () from /lib/libc.so.7
#6  0x000000082ff78159 in ?? () from /lib/libc.so.7
#7  0x00000000005ac54c in ConfNodeFree ()
#8  0x00000000005ac536 in ConfNodeFree ()
#9  0x00000000005ac536 in ConfNodeFree ()
#10 0x00000000005ad305 in ConfDeInit ()
#11 0x000000000058fa61 in ?? ()
#12 0x000000000058f03f in SuricataMain ()
#13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7
#14 0x000000000058bd40 in _start ()

                                          (gdb) thread apply all bt full

Thread 1 (LWP 103117):
#0  0x000000082ff3f34a in thr_kill () from /lib/libc.so.7
No symbol table info available.
#1  0x000000082febf344 in raise () from /lib/libc.so.7
No symbol table info available.
#2  0x000000082ff63a39 in abort () from /lib/libc.so.7
No symbol table info available.
#3  0x000000082ffb2f30 in ?? () from /lib/libc.so.7
No symbol table info available.
#4  0x000000082ff7f440 in ?? () from /lib/libc.so.7
No symbol table info available.
#5  0x000000082ff7842c in ?? () from /lib/libc.so.7
No symbol table info available.
#6  0x000000082ff78159 in ?? () from /lib/libc.so.7
No symbol table info available.
#7  0x00000000005ac54c in ConfNodeFree ()
No symbol table info available.
#8  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#9  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#10 0x00000000005ad305 in ConfDeInit ()
No symbol table info available.
#11 0x000000000058fa61 in ?? ()
No symbol table info available.
#12 0x000000000058f03f in SuricataMain ()
No symbol table info available.
#13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7
No symbol table info available.
#14 0x000000000058bd40 in _start ()
No symbol table info available.

                                          So, it seems to be crashing in the same place. Struggling to tie this behavior back to Hyperscan and the custom Legacy Blocking Module, but perhaps there is some link there I'm missing. I'll dig around for a bit.

                                          S 1 Reply Last reply Reply Quote 0
                                          • S
                                            sgnoc @bmeeks
                                            last edited by

                                            @bmeeks I can't help you with that side, but I know it keeps referencing hyperscan in the logs with the fatal error and it only occurs when I am using auto, which defaults to hyperscan, but also when I use AC-KS for the pattern matcher. It doesn't happen with AC only (I haven't tried AC-BS or specifically selecting hyperscan).

                                            [102876 - W#02] 2023-12-07 11:27:44 Error: spm-hs: Hyperscan returned fatal error -1.
                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post