Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Suricata process dying due to hyperscan problem

    Scheduled Pinned Locked Moved IDS/IPS
    295 Posts 25 Posters 87.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      sgnoc
      last edited by

      @bmeeks I switched my WAN interface to auto. Here are the logs:

      (gdb) bt
#0  0x000000082ff3f34a in thr_kill () from /lib/libc.so.7
#1  0x000000082febf344 in raise () from /lib/libc.so.7
#2  0x000000082ff63a39 in abort () from /lib/libc.so.7
#3  0x000000082ffb2f30 in ?? () from /lib/libc.so.7
#4  0x000000082ff7f440 in ?? () from /lib/libc.so.7
#5  0x000000082ff7842c in ?? () from /lib/libc.so.7
#6  0x000000082ff78159 in ?? () from /lib/libc.so.7
#7  0x00000000005ac54c in ConfNodeFree ()
#8  0x00000000005ac536 in ConfNodeFree ()
#9  0x00000000005ac536 in ConfNodeFree ()
#10 0x00000000005ad305 in ConfDeInit ()
#11 0x000000000058fa61 in ?? ()
#12 0x000000000058f03f in SuricataMain ()
#13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7
#14 0x000000000058bd40 in _start ()```

      (gdb) bt full
#0  0x000000082ff3f34a in thr_kill () from /lib/libc.so.7
No symbol table info available.
#1  0x000000082febf344 in raise () from /lib/libc.so.7
No symbol table info available.
#2  0x000000082ff63a39 in abort () from /lib/libc.so.7
No symbol table info available.
#3  0x000000082ffb2f30 in ?? () from /lib/libc.so.7
No symbol table info available.
#4  0x000000082ff7f440 in ?? () from /lib/libc.so.7
No symbol table info available.
#5  0x000000082ff7842c in ?? () from /lib/libc.so.7
No symbol table info available.
#6  0x000000082ff78159 in ?? () from /lib/libc.so.7
No symbol table info available.
#7  0x00000000005ac54c in ConfNodeFree ()
No symbol table info available.
#8  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#9  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#10 0x00000000005ad305 in ConfDeInit ()
No symbol table info available.
#11 0x000000000058fa61 in ?? ()
No symbol table info available.
#12 0x000000000058f03f in SuricataMain ()
No symbol table info available.
#13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7
No symbol table info available.
#14 0x000000000058bd40 in _start ()
No symbol table info available.

      (gdb) info threads
  Id   Target Id         Frame
* 1    LWP 103117        0x000000082ff3f34a in thr_kill () from /lib/libc.so.7

      (gdb) thread apply all bt

Thread 1 (LWP 103117):
#0  0x000000082ff3f34a in thr_kill () from /lib/libc.so.7
#1  0x000000082febf344 in raise () from /lib/libc.so.7
#2  0x000000082ff63a39 in abort () from /lib/libc.so.7
#3  0x000000082ffb2f30 in ?? () from /lib/libc.so.7
#4  0x000000082ff7f440 in ?? () from /lib/libc.so.7
#5  0x000000082ff7842c in ?? () from /lib/libc.so.7
#6  0x000000082ff78159 in ?? () from /lib/libc.so.7
#7  0x00000000005ac54c in ConfNodeFree ()
#8  0x00000000005ac536 in ConfNodeFree ()
#9  0x00000000005ac536 in ConfNodeFree ()
#10 0x00000000005ad305 in ConfDeInit ()
#11 0x000000000058fa61 in ?? ()
#12 0x000000000058f03f in SuricataMain ()
#13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7
#14 0x000000000058bd40 in _start ()

      (gdb) thread apply all bt full

Thread 1 (LWP 103117):
#0  0x000000082ff3f34a in thr_kill () from /lib/libc.so.7
No symbol table info available.
#1  0x000000082febf344 in raise () from /lib/libc.so.7
No symbol table info available.
#2  0x000000082ff63a39 in abort () from /lib/libc.so.7
No symbol table info available.
#3  0x000000082ffb2f30 in ?? () from /lib/libc.so.7
No symbol table info available.
#4  0x000000082ff7f440 in ?? () from /lib/libc.so.7
No symbol table info available.
#5  0x000000082ff7842c in ?? () from /lib/libc.so.7
No symbol table info available.
#6  0x000000082ff78159 in ?? () from /lib/libc.so.7
No symbol table info available.
#7  0x00000000005ac54c in ConfNodeFree ()
No symbol table info available.
#8  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#9  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#10 0x00000000005ad305 in ConfDeInit ()
No symbol table info available.
#11 0x000000000058fa61 in ?? ()
No symbol table info available.
#12 0x000000000058f03f in SuricataMain ()
No symbol table info available.
#13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7
No symbol table info available.
#14 0x000000000058bd40 in _start ()
No symbol table info available.
      bmeeksB 1 Reply Last reply Reply Quote 0
      • bmeeksB
        bmeeks @sgnoc
        last edited by

        @sgnoc said in Suricata process dying due to hyperscan problem:

        @bmeeks I switched my WAN interface to auto. Here are the logs:

        (gdb) bt
#0  0x000000082ff3f34a in thr_kill () from /lib/libc.so.7
#1  0x000000082febf344 in raise () from /lib/libc.so.7
#2  0x000000082ff63a39 in abort () from /lib/libc.so.7
#3  0x000000082ffb2f30 in ?? () from /lib/libc.so.7
#4  0x000000082ff7f440 in ?? () from /lib/libc.so.7
#5  0x000000082ff7842c in ?? () from /lib/libc.so.7
#6  0x000000082ff78159 in ?? () from /lib/libc.so.7
#7  0x00000000005ac54c in ConfNodeFree ()
#8  0x00000000005ac536 in ConfNodeFree ()
#9  0x00000000005ac536 in ConfNodeFree ()
#10 0x00000000005ad305 in ConfDeInit ()
#11 0x000000000058fa61 in ?? ()
#12 0x000000000058f03f in SuricataMain ()
#13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7
#14 0x000000000058bd40 in _start ()```

        (gdb) bt full
#0  0x000000082ff3f34a in thr_kill () from /lib/libc.so.7
No symbol table info available.
#1  0x000000082febf344 in raise () from /lib/libc.so.7
No symbol table info available.
#2  0x000000082ff63a39 in abort () from /lib/libc.so.7
No symbol table info available.
#3  0x000000082ffb2f30 in ?? () from /lib/libc.so.7
No symbol table info available.
#4  0x000000082ff7f440 in ?? () from /lib/libc.so.7
No symbol table info available.
#5  0x000000082ff7842c in ?? () from /lib/libc.so.7
No symbol table info available.
#6  0x000000082ff78159 in ?? () from /lib/libc.so.7
No symbol table info available.
#7  0x00000000005ac54c in ConfNodeFree ()
No symbol table info available.
#8  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#9  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#10 0x00000000005ad305 in ConfDeInit ()
No symbol table info available.
#11 0x000000000058fa61 in ?? ()
No symbol table info available.
#12 0x000000000058f03f in SuricataMain ()
No symbol table info available.
#13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7
No symbol table info available.
#14 0x000000000058bd40 in _start ()
No symbol table info available.

        (gdb) info threads
  Id   Target Id         Frame
* 1    LWP 103117        0x000000082ff3f34a in thr_kill () from /lib/libc.so.7

        (gdb) thread apply all bt

Thread 1 (LWP 103117):
#0  0x000000082ff3f34a in thr_kill () from /lib/libc.so.7
#1  0x000000082febf344 in raise () from /lib/libc.so.7
#2  0x000000082ff63a39 in abort () from /lib/libc.so.7
#3  0x000000082ffb2f30 in ?? () from /lib/libc.so.7
#4  0x000000082ff7f440 in ?? () from /lib/libc.so.7
#5  0x000000082ff7842c in ?? () from /lib/libc.so.7
#6  0x000000082ff78159 in ?? () from /lib/libc.so.7
#7  0x00000000005ac54c in ConfNodeFree ()
#8  0x00000000005ac536 in ConfNodeFree ()
#9  0x00000000005ac536 in ConfNodeFree ()
#10 0x00000000005ad305 in ConfDeInit ()
#11 0x000000000058fa61 in ?? ()
#12 0x000000000058f03f in SuricataMain ()
#13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7
#14 0x000000000058bd40 in _start ()

        (gdb) thread apply all bt full

Thread 1 (LWP 103117):
#0  0x000000082ff3f34a in thr_kill () from /lib/libc.so.7
No symbol table info available.
#1  0x000000082febf344 in raise () from /lib/libc.so.7
No symbol table info available.
#2  0x000000082ff63a39 in abort () from /lib/libc.so.7
No symbol table info available.
#3  0x000000082ffb2f30 in ?? () from /lib/libc.so.7
No symbol table info available.
#4  0x000000082ff7f440 in ?? () from /lib/libc.so.7
No symbol table info available.
#5  0x000000082ff7842c in ?? () from /lib/libc.so.7
No symbol table info available.
#6  0x000000082ff78159 in ?? () from /lib/libc.so.7
No symbol table info available.
#7  0x00000000005ac54c in ConfNodeFree ()
No symbol table info available.
#8  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#9  0x00000000005ac536 in ConfNodeFree ()
No symbol table info available.
#10 0x00000000005ad305 in ConfDeInit ()
No symbol table info available.
#11 0x000000000058fa61 in ?? ()
No symbol table info available.
#12 0x000000000058f03f in SuricataMain ()
No symbol table info available.
#13 0x000000082fe9675a in __libc_start1 () from /lib/libc.so.7
No symbol table info available.
#14 0x000000000058bd40 in _start ()
No symbol table info available.

        So, it seems to be crashing in the same place. Struggling to tie this behavior back to Hyperscan and the custom Legacy Blocking Module, but perhaps there is some link there I'm missing. I'll dig around for a bit.

        S 1 Reply Last reply Reply Quote 0
        • S
          sgnoc @bmeeks
          last edited by

          @bmeeks I can't help you with that side, but I know it keeps referencing hyperscan in the logs with the fatal error and it only occurs when I am using auto, which defaults to hyperscan, but also when I use AC-KS for the pattern matcher. It doesn't happen with AC only (I haven't tried AC-BS or specifically selecting hyperscan).

          [102876 - W#02] 2023-12-07 11:27:44 Error: spm-hs: Hyperscan returned fatal error -1.
          1 Reply Last reply Reply Quote 0
          • tgm-itsT
            tgm-its
            last edited by

            The strange thing.
            Now after I set it to AC-BS it's been running for over 2 hours without failing.

            So seems to be up now.
            But does it work as it should?
            I checked Alert and last entry there was for like 4 minute's ago.

            Paal B.

            1 Reply Last reply Reply Quote 0
            • bmeeksB
              bmeeks
              last edited by bmeeks

              The Pattern Matcher Algorithm is not critical to whether detection works or not. It only matters in terms of speed versus memory consumption.

              All fhe pattern matcher does is execute a regular expression (regex) search against packet data looking for whatever pattern the rule specified. The different algorithms all accomplish the exact same thing. It's just that some use less memory, but may run slower. Others may be lightning fast, but that speed comes from very high RAM consumption.

              Hyperscan is a specialized regex library created by Intel to take advantage of certain complex math instruction op-codes present in their CPUs. Those specialized instructions that most Intel CPUs have allow for highly optimized regex searches. But because Hyperscan depends on instruction op-codes only present in Intel CPUs, the library does not work with ARM or any other non-Intel hardware.

              The Auto setting for the Pattern Matcher algorithm in Suricata will default to the Hyperscan library if it is present. Otherwise it will choose AC (Aho-Corasick). When the Suricata binary is compiled on the Netgate package builder, it automatically senses which type of CPU is targeted (Intel or something else), and then includes the Hyperscan library only for Intel CPU targets. This means if you have a Netgate firewall appliance with an ARM chip, the Hyperscan library is not even present on your system.

              1 Reply Last reply Reply Quote 3
              • G
                gfeiner
                last edited by

                FYI. 23.09.1 released with this memory allocation fix: https://www.freebsd.org/security/advisories/FreeBSD-EN-23:20.vm.asc

                P 1 Reply Last reply Reply Quote 0
                • P
                  paulp @gfeiner
                  last edited by

                  @gfeiner Unfortunately, in my case at least, Suricata still dies after this upgrade.

                  kiokomanK 1 Reply Last reply Reply Quote 0
                  • kiokomanK
                    kiokoman LAYER 8 @paulp
                    last edited by

                    @paulp said in Suricata process dying due to hyperscan problem:

                    @gfeiner Unfortunately, in my case at least, Suricata still dies after this upgrade.

                    me too, still crashing

                    ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                    Please do not use chat/PM to ask for help
                    we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                    Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                    1 Reply Last reply Reply Quote 0
                    • tgm-itsT
                      tgm-its
                      last edited by

                      Hi all.
                      Just a question? Will the new update Version 2.7.2 of pFsense CE fix the Hyperscan error in Suricata?
                      If so I could update my Firewall if not I'll wait. 😌 😄

                      BR
                      Paal B.

                      1 Reply Last reply Reply Quote 0
                      • tylereversT
                        tylerevers
                        last edited by

                        Not sure what to make of this, but within 24 hours of updating to 23.09.1 one of my Interfaces in Suricata disappeared by itself. For reference, it was the only Interface I was having the Hyperscan issue with and it was running on AC-KS for weeks without an issue. It is the only one of three Interfaces that was using AC-KS instead of Auto.

                        I'm not quite sure how this happened. I woke up today and was just looking at systems to discover that one specific Suricata Interface had deleted itself.

                        bmeeksB 1 Reply Last reply Reply Quote 0
                        • bmeeksB
                          bmeeks @tylerevers
                          last edited by

                          @tylerevers said in Suricata process dying due to hyperscan problem:

                          I'm not quite sure how this happened. I woke up today and was just looking at systems to discover that one specific Suricata Interface had deleted itself.

                          There is no mechanism within the package code for that to happen autonomously.

                          I would check first the pfSense system log on the impacted system to see what may be logged there, then look in the configuration backup history of pfSense to see if someone took an action. The GUI code logs an appropriate message for all configuration changes saved to the config.xml fiile. You can find the configuration entire history under DIAGNOSTICS > BACKUP AND RESTORE.

                          tylereversT 1 Reply Last reply Reply Quote 1
                          • bmeeksB
                            bmeeks
                            last edited by bmeeks

                            Fingers crossed, but I think I found the bug.

                            I definitely found an errant double free() of memory when processing IPv4 addresses in a Pass List. When you execute a double free() of memory you will get random crashes.

                            Since I have been unable to reproduce the problem, I can't say for sure what I found will fix the Hyperscan issue, but I am hopeful based on the fact several of you have stated that turning off Legacy Blocking Mode (in other words, running in plain IDS mode) allows Suricata to run with no issue. The double free() was in the custom Legacy Blocking Module, and it was located in new code that was added with the first 7.0.0 Suricata update back when 23.09 Plus was still in development mode and 2.8 CE snapshots were active. Disabling Legacy Blocking Mode means this buggy portion of the module's code is not executed.

                            I will create a pull request and get this fix posted for Netgate to review and merge. That will not happen until the first of the coming week.

                            Update: the fix is posted at https://github.com/pfsense/FreeBSD-ports/pull/1333 for review and merge by the Netgate developer team.

                            kiokomanK 1 Reply Last reply Reply Quote 5
                            • kiokomanK
                              kiokoman LAYER 8 @bmeeks
                              last edited by kiokoman

                              @bmeeks
                              just out of curiosity, what's the difference between "free" and "SCfree" ? i understand that free() it's to deallocates the memory but i can't find reference for scfree

                              ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                              Please do not use chat/PM to ask for help
                              we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                              Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                              bmeeksB 1 Reply Last reply Reply Quote 0
                              • bmeeksB
                                bmeeks @kiokoman
                                last edited by bmeeks

                                @kiokoman said in Suricata process dying due to hyperscan problem:

                                @bmeeks
                                just out of curiosity, what's the difference between "free" and "SCfree" ?

                                Not a thing currently. The upstream Suricata developers just wrap some common C functions with their own names in case they might ever want to customize them for some reason. Today the two are exactly the same. Here are the #define preprocessor definitions currently in use by upstream:

                                #define SCMalloc malloc
#define SCCalloc calloc
#define SCRealloc realloc
#define SCFree free

                                I just fixed up my code to stay in sync. It was an overlooked typo thing that I noticed while scrutinizing the code for any possible bug.

                                Line 718 in the GitHub link I posted is where the errant double free() call happened. Notice the new revision deletes that line.

                                Line 676 was a misplaced continue statement that could result in a memory leak because it bypasses the free() call to dump the IPv4 address structure created and passed to us elsewhere in the code. Notice the continue statement was moved to be after the free() call.

                                The other changes are just cosmetic.

                                kiokomanK 1 Reply Last reply Reply Quote 1
                                • kiokomanK
                                  kiokoman LAYER 8 @bmeeks
                                  last edited by

                                  @bmeeks
                                  now i understand, thanks for the explanation

                                  ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                                  Please do not use chat/PM to ask for help
                                  we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                                  Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                                  1 Reply Last reply Reply Quote 0
                                  • tylereversT
                                    tylerevers @bmeeks
                                    last edited by

                                    @bmeeks said in Suricata process dying due to hyperscan problem:

                                    @tylerevers said in Suricata process dying due to hyperscan problem:

                                    I'm not quite sure how this happened. I woke up today and was just looking at systems to discover that one specific Suricata Interface had deleted itself.

                                    There is no mechanism within the package code for that to happen autonomously.

                                    I would check first the pfSense system log on the impacted system to see what may be logged there, then look in the configuration backup history of pfSense to see if someone took an action. The GUI code logs an appropriate message for all configuration changes saved to the config.xml fiile. You can find the configuration entire history under DIAGNOSTICS > BACKUP AND RESTORE.

                                    Thank you for your guidance. The configuration history indicated that my specific user made the change. My apologies for the red herring.

                                    1 Reply Last reply Reply Quote 0
                                    • bmeeksB
                                      bmeeks
                                      last edited by bmeeks

                                      My pull request containing the anticipated fix for this Hyperscan error has been merged. An updated Suricata package has built and should appear as an available update for 2.7.2 CE and 23.09.1 Plus users.

                                      Look for an update to version 7.0.2_2 for the Suricata package. When installed, the new package should pull in version 7.0.2_5 of the Suricata binary.

                                      Fingers crossed this fixes the Hyperscan issue. But as I mentioned previously, since I could never reproduce the error in my small test environment, I can't say with 100% certainty the bug I found and fixed is the actual Hyperscan culprit.

                                      N kiokomanK tylereversT 3 Replies Last reply Reply Quote 2
                                      • N
                                        NRgia @bmeeks
                                        last edited by NRgia

                                        @bmeeks said in Suricata process dying due to hyperscan problem:

                                        My pull request containing the anticipated fix for this Hyperscan error has been merged. An updated Suricata package has built and should appear as an available update for 2.7.2 CE and 23.09.1 Plus users.

                                        Look for an update to version 7.0.2_2 for the Suricata package. When installed, the new package should pull in version 7.0.2_5 of the Suricata binary.

                                        For 23.09.1 I can confirm that it is available.
                                        After the update I can see these packages

                                        pfSense-pkg-suricata-7.0.2_2   pfSense package suricata
suricata-7.0.2_5               High Performance Network IDS, IPS and Security Monitoring engine

                                        Thank you

                                        1 Reply Last reply Reply Quote 0
                                        • kiokomanK
                                          kiokoman LAYER 8 @bmeeks
                                          last edited by

                                          @bmeeks
                                          tested

                                          and .....

                                          dcfb4ce9-353a-4416-a9f5-14bb3e50fe54-image.png

                                          not working.. 😀

                                          [340341 - RX#01-vmx2] 2023-12-11 22:42:50 Info: checksum: No packets with invalid checksum, assuming checksum offloading is NOT used
                                          [340346 - W#05] 2023-12-11 22:42:53 Error: spm-hs: Hyperscan returned fatal error -1.
                                          [340347 - W#06] 2023-12-11 22:42:53 Error: spm-hs: Hyperscan returned fatal error -1.

                                          ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                                          Please do not use chat/PM to ask for help
                                          we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                                          Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                                          bmeeksB 1 Reply Last reply Reply Quote 0
                                          • bmeeksB
                                            bmeeks @kiokoman
                                            last edited by

                                            @kiokoman said in Suricata process dying due to hyperscan problem:

                                            @bmeeks
                                            tested

                                            and .....

                                            dcfb4ce9-353a-4416-a9f5-14bb3e50fe54-image.png

                                            not working.. 😀

                                            [340341 - RX#01-vmx2] 2023-12-11 22:42:50 Info: checksum: No packets with invalid checksum, assuming checksum offloading is NOT used
                                            [340346 - W#05] 2023-12-11 22:42:53 Error: spm-hs: Hyperscan returned fatal error -1.
                                            [340347 - W#06] 2023-12-11 22:42:53 Error: spm-hs: Hyperscan returned fatal error -1.

                                            Well, crap! I had high hopes.

                                            Does it still work if you disable blocking mode?

                                            kiokomanK 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.