Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Suricata process dying due to hyperscan problem

    Scheduled Pinned Locked Moved IDS/IPS
    295 Posts 25 Posters 87.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • bmeeksB
      bmeeks @kiokoman
      last edited by

      @kiokoman said in Suricata process dying due to hyperscan problem:

      @bmeeks
      there is already a debug txt a couple of post before this

      I can still try your version and let you know

      Sorry, I missed that earlier. It is helpful as it points to a problem somewhere in Hyperscan itself and not so much in the custom blocking module. Curious how things work when the custom blocking module is disabled, though 🤔.

      Nothing in your back trace results seems to have any relationship to the Legacy Blocking Module.

      1 Reply Last reply Reply Quote 0
      • bmeeksB
        bmeeks @kiokoman
        last edited by bmeeks

        @kiokoman said in Suricata process dying due to hyperscan problem:

        [101255 - Suricata-Main] 2023-12-17 23:41:11 Notice: threads: Threads created -> RX: 1 W: 8 FM: 1 FR: 1 Engine started.
        [368779 - RX#01-vmx2] 2023-12-17 23:41:11 Info: checksum: No packets with invalid checksum, assuming checksum offloading is NOT used
        [368805 - W#04] 2023-12-17 23:41:37 Error: spm-hs: Hyperscan returned fatal error -1.
        [368782 - W#01] 2023-12-17 23:41:37 Error: spm-hs: Hyperscan returned fatal error -1.

        crashed without generating /root/suricata.core

        I was sort of afraid that might happen -- no core dump.

        Refresh my memory on your NIC types (real or virtual) and which rule categories you are using. I want to try once more to duplicate the crash. I really should be able to since users are reporting it with a variety of NIC types (real and virtual), so I'm thinking the NIC is not important to the crash.

        If you could post your suricata.yaml file for one of the crashing interfaces, that might help as well. I can import it directly into a virtual machine to see if I get the crash then. The file will be in /usr/local/etc/suricata/suricata_xxxx_yyyy/suricata.yaml where the "xxxx" and "yyyy" parts are the physical NIC name and a random UUID. And even better would be also posting the actual active rules file. It will be in /usr/local/etc/suricata/suricata_xxxx_yyyy/rules/suricata.rules.

        One other test you can try is changing the Run Mode for Suricata. The default is AutoFP. Try Workers and see if there is any change. Or if already using Workers, swap to AutoFP and test. This parameter is on the INTERFACE SETTINGS tab in the Performance section. Any change made requires Suricata to be restarted so it will see the change.

        kiokomanK 1 Reply Last reply Reply Quote 0
        • kiokomanK
          kiokoman LAYER 8 @bmeeks
          last edited by

          @bmeeks
          i was thinking the same, i see no relationship to the Legacy Blocking Module, the only reference on gdb about alert-pf is on thread 2

          Thread 2 (LWP 179039 of process 2511 "IM#01"):
          #0 0x00000008029807ea in _read () from /lib/libc.so.7
          #1 0x00000008021f4a13 in ?? () from /lib/libthr.so.3
          #2 0x0000000000d0198d in AlertPfMonitorIfaceChanges (args=0x803394ef0) at alert-pf.c:1058

          but the one throwing the error is Thread 8

          Thread 8 (LWP 187487 of process 2511 "W#05"):
          #0 0x00000008029a4454 in exit () from /lib/libc.so.7
          #1 0x0000000000e9bbb9 in HSScan (ctx=<optimized out>, thread_ctx=<optimized out>, haystack=0x8338f7800 "CONNECT wfbssvc65.icrc.trendmicro.com:443 HTTP/1.1\r\nHost: wfbssvc65.icrc.trendmicro.com:443\r\n\r\n", haystack_len=<optimized out>) at util-spm-hs.c:156
          #2 0x0000000000c8319e in AppLayerProtoDetectPMMatchSignature (s=0x80322d4e0, tctx=0x832d22080, f=0x806648a80, buf=0x8338f7800 "CONNECT wfbssvc65.icrc.trendmicro.com:443 HTTP/1.1\r\nHost: wfbssvc65.icrc.trendmicro.com:443\r\n\r\n", buflen=95, flags=<optimized out>, searchlen=<optimized out>, rflow=<optimized out>) at app-layer-detect-proto.c:215
          #3 PMGetProtoInspect (tctx=0x832d22080, pm_ctx=0x1f12c80 <alpd_ctx>, mpm_tctx=<optimized out>, f=0x806648a80, buf=0x8338f7800 "CONNECT wfbssvc65.icrc.trendmicro.com:443 HTTP/1.1\r\nHost: wfbssvc65.icrc.trendmicro.com:443\r\n\r\n", buflen=buflen@entry=95, flags=5 '\005', pm_results=0x7fffdf3f7a00, rflow=0x7fffdf3f7b0f) at app-layer-detect-proto.c:296
          #4 0x0000000000c795c8 in AppLayerProtoDetectPMGetProto (tctx=<optimized out>, f=f@entry=0x806648a80, buf=<optimized out>, buflen=buflen@entry=95, flags=flags@entry=5 '\005', pm_results=pm_results@entry=0x7fffdf3f7a00, rflow=0x7fffdf3f7b0f) at app-layer-detect-proto.c:344
          #5 0x0000000000c78731 in AppLayerProtoDetectGetProto (tctx=<optimized out>, f=f@entry=0x806648a80, buf=0x8338f7800 "CONNECT wfbssvc65.icrc.trendmicro.com:443 HTTP/1.1\r\nHost: wfbssvc65.ic--Type <RET> for more, q to quit, c to continue without paging--
          rc.trendmicro.com:443\r\n\r\n", buflen=95, ipproto=ipproto@entry=6 '\006', flags=flags@entry=5 '\005', reverse_flow=0x7fffdf3f7b0f) at app-layer-detect-proto.c:1433
          #6 0x0000000000c69296 in TCPProtoDetect (tv=tv@entry=0x80d8e0600, ra_ctx=ra_ctx@entry=0x832a00020, app_tctx=app_tctx@entry=0x832d21100, p=p@entry=0x838c33200, f=f@entry=0x806648a80, ssn=ssn@entry=0x8338d5d80, stream=0x7fffdf3f7c68, data=0x8338f7800 "CONNECT wfbssvc65.icrc.trendmicro.com:443 HTTP/1.1\r\nHost: wfbssvc65.icrc.trendmicro.com:443\r\n\r\n", data_len=95, flags=5 '\005', dir=UPDATE_DIR_OPPOSING) at app-layer.c:371
          #7 0x0000000000c68c6d in AppLayerHandleTCPData (tv=tv@entry=0x80d8e0600, ra_ctx=ra_ctx@entry=0x832a00020, p=p@entry=0x838c33200, f=0x806648a80, ssn=ssn@entry=0x8338d5d80, stream=stream@entry=0x7fffdf3f7c68, data=0x8338f7800 "CONNECT wfbssvc65.icrc.trendmicro.com:443 HTTP/1.1\r\nHost: wfbssvc65.icrc.trendmicro.com:443\r\n\r\n", data_len=95, flags=5 '\005', dir=UPDATE_DIR_OPPOSING) at app-layer.c:709
          #8 0x0000000000b62905 in ReassembleUpdateAppLayer (tv=0x80d8e0600, ra_ctx=0x832a00020, ssn=0x8338d5d80, stream=0x7fffdf3f7c68, p=0x838c33200, dir=UPDATE_DIR_OPPOSING) at stream-tcp-reassemble.c:1328
          #9 StreamTcpReassembleAppLayer (tv=tv@entry=0x80d8e0600, ra_ctx=ra_ctx@entry=0x832a00020, ssn=ssn@entry=0x8338d5d80, stream=stream@entry=0x8338d5e20, p=p@entry=0x838c33200, dir=dir@entry=UPDATE_DIR_OPPOSING) at stream-tcp-reassemble.c:1391
          #10 0x0000000000b64879 in StreamTcpReassembleHandleSegmentUpdateACK (tv=0x80d8e0600, ra_ctx=0x832a00020, ssn=0x8338d5d80, stream=0x8338d5e20, p=0x838c33200) at stream-tcp-reassemble.c:1949
          #11 StreamTcpReassembleHandleSegment (tv=0x80d8e0600, ra_ctx=0x832a00020, ssn=0x8338d5d80, stream=0x8338d5d90, p=0x838c33200) at stream-tcp-reassemble.c:1997
          #12 0x0000000000b9c789 in HandleEstablishedPacketToClient (tv=0x82e14bc4, tv@entry=0x80d8e0600, ssn=0x0, ssn@entry=0x8338d5d80, p=0x0, p@entry=0x838c33200, stt=0xe50a5969d84bc43d, stt@entry=0x832d60000) at stream-tcp.c:2811
          #13 0x0000000000b7aa4d in StreamTcpPacketStateEstablished (tv=0x80d8e0600, p=0x838c33200, stt=0x832d60000, ssn=0x8338d5d80) at stream-tcp.c:3223
          #14 StreamTcpStateDispatch (tv=tv@entry=0x80d8e0600, p=p@entry=0x838c33200, stt=stt@entry=0x832d60000, ssn=ssn@entry=0x8338d5d80, state=<optimized out>) at stream-tcp.c:5236
          #15 0x0000000000b766c0 in StreamTcpPacket (tv=tv@entry=0x80d8e0600, p=p@entry=0x838c33200, stt=stt@entry=0x832d60000, pq=<optimized out>) at stream-tcp.c:5433
          #16 0x0000000000b82781 in StreamTcp (tv=tv@entry=0x80d8e0600, p=p@entry=0x838c33200, data=0x832d60000, pq=pq@entry=0x832d18030) at stream-tcp.c:5745
          #17 0x0000000000d53774 in FlowWorkerStreamTCPUpdate (tv=0x1, tv@entry=0x80d8e0600, fw=fw@entry=0x832d18000, p=p@entry=0x838c33200, detect_thread=detect_thread@entry=0x8338d7000, timeout=false) at flow-worker.c:391
          #18 0x0000000000d52f4a in FlowWorker (tv=0x80d8e0600, p=0x838c33200, data=0x832d18000) at flow-worker.c:607
          #19 0x0000000000e33b07 in TmThreadsSlotVarRun (tv=0x80d8e0600, p=0x838c33200, slot=0x8066db440) at tm-threads.c:135
          #20 TmThreadsSlotVar (td=0x80d8e0600) at tm-threads.c:471
          #21 0x00000008021e8d25 in ?? () from /lib/libthr.so.3
          #22 0x0000000000000000 in ?? ()

          anyway give me a sec i give you the content of the yaml

          ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
          Please do not use chat/PM to ask for help
          we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
          Don't forget to Upvote with the 👍 button for any post you find to be helpful.

          kiokomanK 1 Reply Last reply Reply Quote 0
          • kiokomanK
            kiokoman LAYER 8 @kiokoman
            last edited by

            @bmeeks

            suricata.yaml.txt

            ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
            Please do not use chat/PM to ask for help
            we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
            Don't forget to Upvote with the 👍 button for any post you find to be helpful.

            bmeeksB 1 Reply Last reply Reply Quote 0
            • bmeeksB
              bmeeks @kiokoman
              last edited by bmeeks

              @kiokoman said in Suricata process dying due to hyperscan problem:

              @bmeeks

              suricata.yaml.txt

              Thanks! If you can post the suricata.rules file from the interface, that would be useful, too.

              I would really love to be able to reproduce the crash, but if not I can compile a 4.2.0 Hyperscan library that you can try just for giggles.

              The error returned by Hyperscan is actually quite specific. Here are the associated comments:

              hs_error_t err = hs_scan(sctx->db, (const char *)haystack, haystack_len, 0,
                                          scratch, MatchEvent, &match_offset);
                 if (err != HS_SUCCESS && err != HS_SCAN_TERMINATED) {
                     /* An error value (other than HS_SCAN_TERMINATED) from hs_scan()
                      * indicates that it was passed an invalid database or scratch region,
                      * which is not something we can recover from at scan time. */
                     SCLogError("Hyperscan returned fatal error %d.", err);
                     exit(EXIT_FAILURE);
                 }
              

              This indicates to me that Suricata is passing Hyperscan either an invalid database or an invalid scratch memory area. What is strange, though, is that no other users on Linux are reporting this kind of issue. At least I have not found such a report.

              kiokomanK 1 Reply Last reply Reply Quote 0
              • kiokomanK
                kiokoman LAYER 8 @bmeeks
                last edited by

                @bmeeks
                https://drive.google.com/drive/folders/1-ag4lFYM0I15IlHX3kxHoNNPV5LPX6QR?usp=sharing

                ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                Please do not use chat/PM to ask for help
                we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                bmeeksB 2 Replies Last reply Reply Quote 0
                • bmeeksB
                  bmeeks @kiokoman
                  last edited by

                  @kiokoman said in Suricata process dying due to hyperscan problem:

                  @bmeeks
                  https://drive.google.com/drive/folders/1-ag4lFYM0I15IlHX3kxHoNNPV5LPX6QR?usp=sharing

                  Got them! Thanks!

                  1 Reply Last reply Reply Quote 0
                  • bmeeksB
                    bmeeks @kiokoman
                    last edited by

                    @kiokoman:
                    Imported your suricata.yaml configuration and suricata.rules file into my virtual machine. Only edited the interface names to reflect em0 which is what I use in my virtual machine at the moment.

                    Suricata starts up and runs. No error yet. Will let it run for a while to see if a crash occurs. I suspect my little VM is not seeing the same amount of packets (traffic) as your machine, though.

                    kiokomanK 1 Reply Last reply Reply Quote 0
                    • kiokomanK
                      kiokoman LAYER 8 @bmeeks
                      last edited by

                      @bmeeks
                      i tried to compile hyperscan 5.4.2 but it does not work with suricata, coredump with "ipprotos" not found or something

                      with 5.2.1 i have assertion failed ...

                      i must have done something wrong

                      if you can compile a different hs library version i can try it

                      changing from AutoFP to Worker make no difference

                      ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                      Please do not use chat/PM to ask for help
                      we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                      Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                      1 Reply Last reply Reply Quote 0
                      • M
                        masons @bmeeks
                        last edited by

                        @bmeeks,

                        I loaded the Suricata package that you provided. I did not disable ASLR, so when I started Suricata with pattern matcher set to to "Auto", the PC interface Suricata instance immediately crashed. Below is the gdb output you asked for.

                        Core was generated by `/usr/local/bin/suricata -i vtnet0.700 -D -c /usr/local/etc/suricata/suricata_238'.
                        Program terminated with signal SIGSEGV, Segmentation fault.
                        Address not mapped to object.
                        #0  0x0000000000bf531a in AppLayerProtoDetectPMGetIpprotos (alproto=22, ipprotos=0x76a19745bd6 "") at app-layer-detect-proto.c:1177
                        1177    app-layer-detect-proto.c: No such file or directory.
                        [Current thread is 1 (LWP 100363)]
                        warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
                        of file /usr/local/bin/suricata.
                        Use `info auto-load python-scripts [REGEXP]' to list them.
                        (gdb) bt
                        #0  0x0000000000bf531a in AppLayerProtoDetectPMGetIpprotos (alproto=22, ipprotos=0x76a19745bd6 "") at app-layer-detect-proto.c:1177
                        #1  0x0000000000bf5193 in AppLayerProtoDetectSupportedIpprotos (alproto=22, ipprotos=0x76a19745bd6 "") at app-layer-detect-proto.c:2054
                        #2  0x0000000000b2c299 in SigParseProto (s=0x76a19745bc0, protostr=0x822b0ce48 "dhcp") at detect-parse.c:1107
                        #3  0x0000000000b2af88 in SigParseBasics (de_ctx=0x76a17580000, s=0x76a19745bc0, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)", parser=0x822b0ae48, addrs_direction=0 '\000') at detect-parse.c:1375
                        #4  0x0000000000b28997 in SigParse (de_ctx=0x76a17580000, s=0x76a19745bc0, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)", addrs_direction=0 '\000', parser=0x822b0ae48) at detect-parse.c:1456
                        #5  0x0000000000b27024 in SigInitHelper (de_ctx=0x76a17580000, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)", dir=0 '\000') at detect-parse.c:2147
                        #6  0x0000000000b26d86 in SigInit (de_ctx=0x76a17580000, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)") at detect-parse.c:2304
                        #7  0x0000000000b2794d in DetectEngineAppendSig (de_ctx=0x76a17580000, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)") at detect-parse.c:2602
                        #8  0x0000000000b5cd42 in DetectLoadSigFile (de_ctx=0x76a17580000, sig_file=0x76a1501c780 "/usr/local/etc/suricata/suricata_23822_vtnet0.700/rules/suricata.rules", goodsigs=0x822b1d0fc, badsigs=0x822b1d0f8) at detect-engine-loader.c:165
                        #9  0x0000000000b5bc0e in ProcessSigFiles (de_ctx=0x76a17580000, pattern=0x76a1501c6e0 "/usr/local/etc/suricata/suricata_23822_vtnet0.700/rules/suricata.rules", st=0x76a175813f0, good_sigs=0x822b1d0fc, bad_sigs=0x822b1d0f8) at detect-engine-loader.c:249
                        #10 0x0000000000b5b58c in SigLoadSignatures (de_ctx=0x76a17580000, sig_file=0x0, sig_file_exclusive=0) at detect-engine-loader.c:307
                        #11 0x0000000000ab9c1e in LoadSignatures (de_ctx=0x76a17580000, suri=0x1e27b90 <suricata>) at suricata.c:2415
                        #12 0x0000000000ab99da in PostConfLoadedDetectSetup (suri=0x1e27b90 <suricata>) at suricata.c:2562
                        #13 0x0000000000abb4ed in SuricataMain (argc=8, argv=0x822b1d2f0) at suricata.c:2971
                        #14 0x0000000000ab9222 in main (argc=8, argv=0x822b1d2f0) at main.c:22
                        

                        And the output from bt full.

                        (gdb) bt full
                        #0  0x0000000000bf531a in AppLayerProtoDetectPMGetIpprotos (alproto=22, ipprotos=0x76a19745bd6 "") at app-layer-detect-proto.c:1177
                                s = 0x76a01000000
                                x = 0
                                pm_ctx = 0x1e48de0 <alpd_ctx+144>
                                j = 0
                                ipproto = 17 '\021'
                                i = 1 '\001'
                        #1  0x0000000000bf5193 in AppLayerProtoDetectSupportedIpprotos (alproto=22, ipprotos=0x76a19745bd6 "") at app-layer-detect-proto.c:2054
                        No locals.
                        #2  0x0000000000b2c299 in SigParseProto (s=0x76a19745bc0, protostr=0x822b0ce48 "dhcp") at detect-parse.c:1107
                                r = -1
                        #3  0x0000000000b2af88 in SigParseBasics (de_ctx=0x76a17580000, s=0x76a19745bc0, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)", parser=0x822b0ae48, addrs_direction=0 '\000') at detect-parse.c:1375
                                index = 0x822b08d4f "msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;"
                                dup = "alert\000dhcp\000any\000any\000->\000any\000any\000(msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;\000\000 rev:2;\000\000\000\0002;\000\00064; rev:2;\000\000\000comman"...
                        #4  0x0000000000b28997 in SigParse (de_ctx=0x76a17580000, s=0x76a19745bc0, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)", addrs_direction=0 '\000', parser=0x822b0ae48) at detect-parse.c:1456
                                ret = 8
                        #5  0x0000000000b27024 in SigInitHelper (de_ctx=0x76a17580000, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)", dir=0 '\000') at detect-parse.c:2147
                                parser = {action = "alert", '\000' <repeats 8186 times>, protocol = "dhcp", '\000' <repeats 8187 times>, direction = "->", '\000' <repeats 8189 times>, src = "any", '\000' <repeats 8188 times>, dst = "any", '\000' <repeats 8188 times>, sp = "any", '\000' <repeats 8188 times>, dp = "any", '\000' <repeats 8188 times>,
                                  opts = "msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;", '\000' <repeats 8058 times>}
                                sig = 0x76a19745bc0
                                ret = 0
                        #6  0x0000000000b26d86 in SigInit (de_ctx=0x76a17580000, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)") at detect-parse.c:2304
                                oldsignum = 685
                                sig = 0x822b1af90
                        #7  0x0000000000b2794d in DetectEngineAppendSig (de_ctx=0x76a17580000, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)") at detect-parse.c:2602
                                sig = 0x822b1aef0
                                dup_sig = 41
                        #8  0x0000000000b5cd42 in DetectLoadSigFile (de_ctx=0x76a17580000, sig_file=0x76a1501c780 "/usr/local/etc/suricata/suricata_23822_vtnet0.700/rules/suricata.rules", goodsigs=0x822b1d0fc, badsigs=0x822b1d0f8) at detect-engine-loader.c:165
                                len = 166
                                sig = 0x76a19745a80
                                good = 685
                                bad = 0
                                line = "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)\000\000rev:2;)\000\000\000\000;)\000\0004; rev:2;)\000\000\000omman"...
                                offset = 0
                                lineno = 694
                                multiline = 0
                                fp = 0x8334c85a0
                        #9  0x0000000000b5bc0e in ProcessSigFiles (de_ctx=0x76a17580000, pattern=0x76a1501c6e0 "/usr/local/etc/suricata/suricata_23822_vtnet0.700/rules/suricata.rules", st=0x76a175813f0, good_sigs=0x822b1d0fc, bad_sigs=0x822b1d0f8) at detect-engine-loader.c:249
                                fname = 0x76a1501c780 "/usr/local/etc/suricata/suricata_23822_vtnet0.700/rules/suricata.rules"
                                i = 0
                                r = 0
                                files = {gl_pathc = 1, gl_matchc = 1, gl_offs = 0, gl_flags = 0, gl_pathv = 0x76a1509aba0, gl_errfunc = 0x0, gl_closedir = 0x1, gl_readdir = 0x76a15012000, gl_opendir = 0x822b1d080, gl_lstat = 0x82d8566f8, gl_stat = 0x76a15012000}
                        #10 0x0000000000b5b58c in SigLoadSignatures (de_ctx=0x76a17580000, sig_file=0x0, sig_file_exclusive=0) at detect-engine-loader.c:307
                                rule_files = 0x76a150b1400
                                file = 0x76a150b1480
                                sig_stat = 0x76a175813f0
                                ret = 0
                                sfile = 0x76a1501c6e0 "/usr/local/etc/suricata/suricata_23822_vtnet0.700/rules/suricata.rules"
                                varname = "rule-files", '\000' <repeats 117 times>
                                good_sigs = 0
                                bad_sigs = 0
                        #11 0x0000000000ab9c1e in LoadSignatures (de_ctx=0x76a17580000, suri=0x1e27b90 <suricata>) at suricata.c:2415
                        No locals.
                        #12 0x0000000000ab99da in PostConfLoadedDetectSetup (suri=0x1e27b90 <suricata>) at suricata.c:2562
                                mt_enabled = 0
                                default_tenant = 0
                                de_ctx = 0x76a17580000
                        #13 0x0000000000abb4ed in SuricataMain (argc=8, argv=0x822b1d2f0) at suricata.c:2971
                                tracking = 0
                                limit_nproc = 1
                        #14 0x0000000000ab9222 in main (argc=8, argv=0x822b1d2f0) at main.c:22
                        No locals.
                        
                        bmeeksB kiokomanK 2 Replies Last reply Reply Quote 0
                        • bmeeksB
                          bmeeks @masons
                          last edited by

                          @masons said in Suricata process dying due to hyperscan problem:

                          @bmeeks,

                          I loaded the Suricata package that you provided. I did not disable ASLR, so when I started Suricata with pattern matcher set to to "Auto", the PC interface Suricata instance immediately crashed. Below is the gdb output you asked for.

                          Core was generated by `/usr/local/bin/suricata -i vtnet0.700 -D -c /usr/local/etc/suricata/suricata_238'.
                          Program terminated with signal SIGSEGV, Segmentation fault.
                          Address not mapped to object.
                          #0  0x0000000000bf531a in AppLayerProtoDetectPMGetIpprotos (alproto=22, ipprotos=0x76a19745bd6 "") at app-layer-detect-proto.c:1177
                          1177    app-layer-detect-proto.c: No such file or directory.
                          [Current thread is 1 (LWP 100363)]
                          warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
                          of file /usr/local/bin/suricata.
                          Use `info auto-load python-scripts [REGEXP]' to list them.
                          (gdb) bt
                          #0  0x0000000000bf531a in AppLayerProtoDetectPMGetIpprotos (alproto=22, ipprotos=0x76a19745bd6 "") at app-layer-detect-proto.c:1177
                          #1  0x0000000000bf5193 in AppLayerProtoDetectSupportedIpprotos (alproto=22, ipprotos=0x76a19745bd6 "") at app-layer-detect-proto.c:2054
                          #2  0x0000000000b2c299 in SigParseProto (s=0x76a19745bc0, protostr=0x822b0ce48 "dhcp") at detect-parse.c:1107
                          #3  0x0000000000b2af88 in SigParseBasics (de_ctx=0x76a17580000, s=0x76a19745bc0, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)", parser=0x822b0ae48, addrs_direction=0 '\000') at detect-parse.c:1375
                          #4  0x0000000000b28997 in SigParse (de_ctx=0x76a17580000, s=0x76a19745bc0, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)", addrs_direction=0 '\000', parser=0x822b0ae48) at detect-parse.c:1456
                          #5  0x0000000000b27024 in SigInitHelper (de_ctx=0x76a17580000, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)", dir=0 '\000') at detect-parse.c:2147
                          #6  0x0000000000b26d86 in SigInit (de_ctx=0x76a17580000, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)") at detect-parse.c:2304
                          #7  0x0000000000b2794d in DetectEngineAppendSig (de_ctx=0x76a17580000, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)") at detect-parse.c:2602
                          #8  0x0000000000b5cd42 in DetectLoadSigFile (de_ctx=0x76a17580000, sig_file=0x76a1501c780 "/usr/local/etc/suricata/suricata_23822_vtnet0.700/rules/suricata.rules", goodsigs=0x822b1d0fc, badsigs=0x822b1d0f8) at detect-engine-loader.c:165
                          #9  0x0000000000b5bc0e in ProcessSigFiles (de_ctx=0x76a17580000, pattern=0x76a1501c6e0 "/usr/local/etc/suricata/suricata_23822_vtnet0.700/rules/suricata.rules", st=0x76a175813f0, good_sigs=0x822b1d0fc, bad_sigs=0x822b1d0f8) at detect-engine-loader.c:249
                          #10 0x0000000000b5b58c in SigLoadSignatures (de_ctx=0x76a17580000, sig_file=0x0, sig_file_exclusive=0) at detect-engine-loader.c:307
                          #11 0x0000000000ab9c1e in LoadSignatures (de_ctx=0x76a17580000, suri=0x1e27b90 <suricata>) at suricata.c:2415
                          #12 0x0000000000ab99da in PostConfLoadedDetectSetup (suri=0x1e27b90 <suricata>) at suricata.c:2562
                          #13 0x0000000000abb4ed in SuricataMain (argc=8, argv=0x822b1d2f0) at suricata.c:2971
                          #14 0x0000000000ab9222 in main (argc=8, argv=0x822b1d2f0) at main.c:22
                          

                          And the output from bt full.

                          (gdb) bt full
                          #0  0x0000000000bf531a in AppLayerProtoDetectPMGetIpprotos (alproto=22, ipprotos=0x76a19745bd6 "") at app-layer-detect-proto.c:1177
                                  s = 0x76a01000000
                                  x = 0
                                  pm_ctx = 0x1e48de0 <alpd_ctx+144>
                                  j = 0
                                  ipproto = 17 '\021'
                                  i = 1 '\001'
                          #1  0x0000000000bf5193 in AppLayerProtoDetectSupportedIpprotos (alproto=22, ipprotos=0x76a19745bd6 "") at app-layer-detect-proto.c:2054
                          No locals.
                          #2  0x0000000000b2c299 in SigParseProto (s=0x76a19745bc0, protostr=0x822b0ce48 "dhcp") at detect-parse.c:1107
                                  r = -1
                          #3  0x0000000000b2af88 in SigParseBasics (de_ctx=0x76a17580000, s=0x76a19745bc0, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)", parser=0x822b0ae48, addrs_direction=0 '\000') at detect-parse.c:1375
                                  index = 0x822b08d4f "msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;"
                                  dup = "alert\000dhcp\000any\000any\000->\000any\000any\000(msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;\000\000 rev:2;\000\000\000\0002;\000\00064; rev:2;\000\000\000comman"...
                          #4  0x0000000000b28997 in SigParse (de_ctx=0x76a17580000, s=0x76a19745bc0, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)", addrs_direction=0 '\000', parser=0x822b0ae48) at detect-parse.c:1456
                                  ret = 8
                          #5  0x0000000000b27024 in SigInitHelper (de_ctx=0x76a17580000, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)", dir=0 '\000') at detect-parse.c:2147
                                  parser = {action = "alert", '\000' <repeats 8186 times>, protocol = "dhcp", '\000' <repeats 8187 times>, direction = "->", '\000' <repeats 8189 times>, src = "any", '\000' <repeats 8188 times>, dst = "any", '\000' <repeats 8188 times>, sp = "any", '\000' <repeats 8188 times>, dp = "any", '\000' <repeats 8188 times>,
                                    opts = "msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;", '\000' <repeats 8058 times>}
                                  sig = 0x76a19745bc0
                                  ret = 0
                          #6  0x0000000000b26d86 in SigInit (de_ctx=0x76a17580000, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)") at detect-parse.c:2304
                                  oldsignum = 685
                                  sig = 0x822b1af90
                          #7  0x0000000000b2794d in DetectEngineAppendSig (de_ctx=0x76a17580000, sigstr=0x822b1af90 "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)") at detect-parse.c:2602
                                  sig = 0x822b1aef0
                                  dup_sig = 41
                          #8  0x0000000000b5cd42 in DetectLoadSigFile (de_ctx=0x76a17580000, sig_file=0x76a1501c780 "/usr/local/etc/suricata/suricata_23822_vtnet0.700/rules/suricata.rules", goodsigs=0x822b1d0fc, badsigs=0x822b1d0f8) at detect-engine-loader.c:165
                                  len = 166
                                  sig = 0x76a19745a80
                                  good = 685
                                  bad = 0
                                  line = "alert dhcp any any -> any any (msg:\"SURICATA DHCP malformed options\"; app-layer-event:dhcp.malformed_options; classtype:protocol-command-decode; sid:2227000; rev:1;)\000\000rev:2;)\000\000\000\000;)\000\0004; rev:2;)\000\000\000omman"...
                                  offset = 0
                                  lineno = 694
                                  multiline = 0
                                  fp = 0x8334c85a0
                          #9  0x0000000000b5bc0e in ProcessSigFiles (de_ctx=0x76a17580000, pattern=0x76a1501c6e0 "/usr/local/etc/suricata/suricata_23822_vtnet0.700/rules/suricata.rules", st=0x76a175813f0, good_sigs=0x822b1d0fc, bad_sigs=0x822b1d0f8) at detect-engine-loader.c:249
                                  fname = 0x76a1501c780 "/usr/local/etc/suricata/suricata_23822_vtnet0.700/rules/suricata.rules"
                                  i = 0
                                  r = 0
                                  files = {gl_pathc = 1, gl_matchc = 1, gl_offs = 0, gl_flags = 0, gl_pathv = 0x76a1509aba0, gl_errfunc = 0x0, gl_closedir = 0x1, gl_readdir = 0x76a15012000, gl_opendir = 0x822b1d080, gl_lstat = 0x82d8566f8, gl_stat = 0x76a15012000}
                          #10 0x0000000000b5b58c in SigLoadSignatures (de_ctx=0x76a17580000, sig_file=0x0, sig_file_exclusive=0) at detect-engine-loader.c:307
                                  rule_files = 0x76a150b1400
                                  file = 0x76a150b1480
                                  sig_stat = 0x76a175813f0
                                  ret = 0
                                  sfile = 0x76a1501c6e0 "/usr/local/etc/suricata/suricata_23822_vtnet0.700/rules/suricata.rules"
                                  varname = "rule-files", '\000' <repeats 117 times>
                                  good_sigs = 0
                                  bad_sigs = 0
                          #11 0x0000000000ab9c1e in LoadSignatures (de_ctx=0x76a17580000, suri=0x1e27b90 <suricata>) at suricata.c:2415
                          No locals.
                          #12 0x0000000000ab99da in PostConfLoadedDetectSetup (suri=0x1e27b90 <suricata>) at suricata.c:2562
                                  mt_enabled = 0
                                  default_tenant = 0
                                  de_ctx = 0x76a17580000
                          #13 0x0000000000abb4ed in SuricataMain (argc=8, argv=0x822b1d2f0) at suricata.c:2971
                                  tracking = 0
                                  limit_nproc = 1
                          #14 0x0000000000ab9222 in main (argc=8, argv=0x822b1d2f0) at main.c:22
                          No locals.
                          

                          Thank you. This crash is similar to @kiokoman's core dump. Suricata seems to be doing something improper within the App Layer protocols logic. In your case it is causing a Signal 11 segfault, but in his case it simply results in the Hyperscan library stopping on a passed-in parameter validation check which leads to Suricata shutting down with a Fatal Error. I may need to raise this again with upstream.

                          Still puzzled why I can't seem to reproduce it -- at least not yet.

                          1 Reply Last reply Reply Quote 0
                          • kiokomanK
                            kiokoman LAYER 8 @masons
                            last edited by

                            @masons

                            could you tell me if the contents of this directory are the same as mine pls?

                            ls -la /usr/local/lib/libhs*

                            -rw-r--r--  1 root wheel 12342598 Dec  8 18:06 /usr/local/lib/libhs.a
                            lrwxr-xr-x  1 root wheel       10 Dec  8 18:07 /usr/local/lib/libhs.so -> libhs.so.5
                            lrwxr-xr-x  1 root wheel       14 Dec  8 18:07 /usr/local/lib/libhs.so.5 -> libhs.so.5.4.0
                            -rwxr-xr-x  1 root wheel  4521768 Dec  8 18:07 /usr/local/lib/libhs.so.5.4.0
                            -rw-r--r--  1 root wheel  1423432 Dec  8 18:05 /usr/local/lib/libhs_runtime.a
                            lrwxr-xr-x  1 root wheel       18 Dec  8 18:07 /usr/local/lib/libhs_runtime.so -> libhs_runtime.so.5
                            lrwxr-xr-x  1 root wheel       22 Dec  8 18:07 /usr/local/lib/libhs_runtime.so.5 -> libhs_runtime.so.5.4.0
                            -rwxr-xr-x  1 root wheel  1015368 Dec  8 18:07 /usr/local/lib/libhs_runtime.so.5.4.0
                            

                            ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                            Please do not use chat/PM to ask for help
                            we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                            Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                            M 1 Reply Last reply Reply Quote 0
                            • bmeeksB
                              bmeeks
                              last edited by bmeeks

                              While not exactly the same, the failure locations I see in both of the suricata.core file back trace results look eerily like this old bug in that the failure is in the app-layer protocols section: https://redmine.openinfosecfoundation.org/issues/4273.

                              I wonder if there still might be an issue lurking in the logic even though the originally reported bug was fixed ??

                              I've sent both back trace results and @kiokoman's gdb.txt dump to the upstream developers asking for any insights they may have.

                              JonathanLeeJ 1 Reply Last reply Reply Quote 0
                              • JonathanLeeJ
                                JonathanLee @bmeeks
                                last edited by

                                @bmeeks Happy Holidays!!! I wanted to message you to say that.

                                Make sure to upvote

                                bmeeksB 1 Reply Last reply Reply Quote 0
                                • M
                                  masons @kiokoman
                                  last edited by

                                  @kiokoman,

                                  It looks like file sizes differ. I'm running pfSense 2.7.2 CE in this VM.

                                  -rw-r--r--  1 root wheel  1421072 Jun 30 08:16 /usr/local/lib/libhs_runtime.a
                                  lrwxr-xr-x  1 root wheel       18 Jun 30 08:17 /usr/local/lib/libhs_runtime.so -> libhs_runtime.so.5
                                  lrwxr-xr-x  1 root wheel       22 Jun 30 08:17 /usr/local/lib/libhs_runtime.so.5 -> libhs_runtime.so.5.4.0
                                  -rwxr-xr-x  1 root wheel  1007944 Jun 30 08:17 /usr/local/lib/libhs_runtime.so.5.4.0
                                  -rw-r--r--  1 root wheel 12452892 Jun 30 08:17 /usr/local/lib/libhs.a
                                  lrwxr-xr-x  1 root wheel       10 Jun 30 08:17 /usr/local/lib/libhs.so -> libhs.so.5
                                  lrwxr-xr-x  1 root wheel       14 Jun 30 08:17 /usr/local/lib/libhs.so.5 -> libhs.so.5.4.0
                                  -rwxr-xr-x  1 root wheel  4518952 Jun 30 08:17 /usr/local/lib/libhs.so.5.4.0
                                  
                                  bmeeksB 1 Reply Last reply Reply Quote 0
                                  • bmeeksB
                                    bmeeks @JonathanLee
                                    last edited by

                                    @JonathanLee said in Suricata process dying due to hyperscan problem:

                                    @bmeeks Happy Holidays!!! I wanted to message you to say that.

                                    Thank you. Same to you and your family.

                                    1 Reply Last reply Reply Quote 1
                                    • bmeeksB
                                      bmeeks @masons
                                      last edited by bmeeks

                                      @masons said in Suricata process dying due to hyperscan problem:

                                      @kiokoman,

                                      It looks like file sizes differ. I'm running pfSense 2.7.2 CE in this VM.

                                      -rw-r--r--  1 root wheel  1421072 Jun 30 08:16 /usr/local/lib/libhs_runtime.a
                                      lrwxr-xr-x  1 root wheel       18 Jun 30 08:17 /usr/local/lib/libhs_runtime.so -> libhs_runtime.so.5
                                      lrwxr-xr-x  1 root wheel       22 Jun 30 08:17 /usr/local/lib/libhs_runtime.so.5 -> libhs_runtime.so.5.4.0
                                      -rwxr-xr-x  1 root wheel  1007944 Jun 30 08:17 /usr/local/lib/libhs_runtime.so.5.4.0
                                      -rw-r--r--  1 root wheel 12452892 Jun 30 08:17 /usr/local/lib/libhs.a
                                      lrwxr-xr-x  1 root wheel       10 Jun 30 08:17 /usr/local/lib/libhs.so -> libhs.so.5
                                      lrwxr-xr-x  1 root wheel       14 Jun 30 08:17 /usr/local/lib/libhs.so.5 -> libhs.so.5.4.0
                                      -rwxr-xr-x  1 root wheel  4518952 Jun 30 08:17 /usr/local/lib/libhs.so.5.4.0
                                      

                                      Your file dates are weird. June 30 for 2.7.2 files is too early. Mine show December 16 dates, which coincides with the day I updated this particular VM to 2.7.2 CE.

                                      My file sizes agree 100% with those posted by @kiokoman. Your file dates of June 30 agree with the rollout of pfSense 2.7.0 CE. The 2.7.1 CE was released in November of this year.

                                      You might want to remove the Suricata package from your VM, then run this command to be sure the Hyperscan library is also removed:

                                      pkg info hyperscan
                                      

                                      If anything other than something like "not installed" comes back, then manually remove Hyperscan with this command:

                                      pkg delete hyperscan
                                      

                                      You having an older library might be why you are getting the segfault instead of the same fatal error -1 termination that @kiokoman and most others are seeing.

                                      Now reinstall Suricata and you should get the correct latest Hyperscan library.

                                      M 1 Reply Last reply Reply Quote 0
                                      • M
                                        masons @bmeeks
                                        last edited by

                                        @bmeeks, I now have the same file sizes and dates. Would you like me to again load the debug package or do you have all that you need now?

                                        bmeeksB kiokomanK 2 Replies Last reply Reply Quote 0
                                        • bmeeksB
                                          bmeeks @masons
                                          last edited by bmeeks

                                          @masons said in Suricata process dying due to hyperscan problem:

                                          @bmeeks, I now have the same file sizes and dates. Would you like me to again load the debug package or do you have all that you need now?

                                          It would be interesting to see if now instead of a segfault you get the fatal error termination that @kiokoman is seeing.

                                          You do not have to install the debug package version for that test unless you just want to. I'm hoping maybe your segfault was an issue with the library compilation (being possibly from an older pfSense kernel build) and not a different manifestation of the same Hyperscan error.

                                          So, just set the Pattern Matcher to Auto and attempt to start Suricata. If it dies immediately (or later), check the suricata.log file to see if you have the "Hyperscan returned fatal error -1" message. If you still get a Signal 11 segfault in the system log, then color me thoroughly flummoxed 😁.

                                          NeganN 1 Reply Last reply Reply Quote 0
                                          • NeganN
                                            Negan @bmeeks
                                            last edited by

                                            @bmeeks

                                            Just had Suricata crash with this messag on my Netgate 8200:

                                            Crash report begins. Anonymous machine information:

                                            amd64
                                            14.0-CURRENT
                                            FreeBSD 14.0-CURRENT amd64 1400094 #1 plus-RELENG_23_09_1-n256200-3de1e293f3a: Wed Dec 6 21:00:32 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_09_1-main/obj/amd64/Obhu6gXB/var/jenkins/workspace/pfSense-Plus-snapshots-23_09_1

                                            Crash report details:

                                            PHP Errors:
                                            [19-Dec-2023 13:50:29 Australia/Sydney] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 896747264 bytes) in /usr/local/www/suricata/suricata_logs_browser.php on line 50

                                            No FreeBSD crash data found.

                                            S bmeeksB 2 Replies Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.