Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Suricata process dying due to hyperscan problem

    IDS/IPS
    25
    295
    64.9k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jowe78J
      jowe78 @chrysmon
      last edited by

      @chrysmon said in Suricata process dying due to hyperscan problem:

      in my case suricata stopped crashing after setting the Pattern Matcher to AC-BS

      I changed back so still using hyperscan (or auto), just more RAM.

      1 Reply Last reply Reply Quote 0
      • kiokomanK
        kiokoman LAYER 8 @bmeeks
        last edited by

        @bmeeks
        turning Blocking mode to off make it stable

        ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
        Please do not use chat/PM to ask for help
        we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
        Don't forget to Upvote with the 👍 button for any post you find to be helpful.

        bmeeksB 1 Reply Last reply Reply Quote 0
        • bmeeksB
          bmeeks @kiokoman
          last edited by bmeeks

          @kiokoman said in Suricata process dying due to hyperscan problem:

          @bmeeks
          turning Blocking mode to off make it stable

          Just to be clear. I am trying to determine if there is any link between Blocking being enabled and the Hyperscan crash.

          You said your WAN was stable with blocking enabled, but your LAN (with the same rules cloned from the WAN interface and blocking enabled) would not even start. So, I want you to disable blocking mode on the LAN interface only and see if Suricata then starts on your LAN.

          Then, if you LAN interface starts up with no other changes besides disabling blocking mode, then enable blocking mode again and see if the startup crash returns.

          Perhaps I misunderstood your statement, though. Do you mean your LAN interfaces starts but then randomly crashes later, or does it fail to even start? Either way, removing Blocking Mode as a variable will be helpful.

          The idea is to see if blocking mode is in any way related to the Hyperscan crash. I don't really believe it is, but since I have not been able to reproduce this crash I must depend on the users having the issue to help me understand where it's coming from.

          kiokomanK 1 Reply Last reply Reply Quote 0
          • kiokomanK
            kiokoman LAYER 8 @bmeeks
            last edited by kiokoman

            @bmeeks
            WAN is stable with blocking enabled

            LAN interfaces starts but then randomly crashes after a few minutes (max 5 min)

            LAN interface stay up and running if i disable blocking mode

            another BT running in autofp mode

            (gdb) b exit
            Breakpoint 1 at 0x82f969454
            (gdb) continue
            Continuing.
            [New LWP 561670 of process 86533]
            [New LWP 561671 of process 86533]
            [New LWP 561672 of process 86533]
            [New LWP 561673 of process 86533]
            [New LWP 561674 of process 86533]
            [New LWP 561675 of process 86533]
            [New LWP 561676 of process 86533]
            [New LWP 561677 of process 86533]
            [New LWP 561678 of process 86533]
            [New LWP 561679 of process 86533]
            [New LWP 561680 of process 86533]
            [Switching to LWP 561677 of process 86533]

            Thread 10 "W#07" hit Breakpoint 1, 0x000000082f969454 in exit () from /lib/libc.so.7
            (gdb) bt
            #0 0x000000082f969454 in exit () from /lib/libc.so.7
            #1 0x00000000006de629 in ?? ()
            #2 0x000000000061d9ac in ?? ()
            #3 0x000000000061ac4e in AppLayerProtoDetectGetProto ()
            #4 0x00000000006197c9 in ?? ()
            #5 0x0000000000619439 in AppLayerHandleTCPData ()
            #6 0x00000000005aee4a in StreamTcpReassembleAppLayer ()
            #7 0x00000000005af9e2 in StreamTcpReassembleHandleSegment ()
            #8 0x00000000005b2b9f in ?? ()
            #9 0x00000000005b15e2 in StreamTcpPacket ()
            #10 0x00000000005b7817 in StreamTcp ()
            #11 0x00000000006731c1 in ?? ()
            #12 0x0000000000672a1a in ?? ()
            #13 0x00000000006a7fe7 in ?? ()
            #14 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            #15 0x0000000000000000 in ?? ()
            Backtrace stopped: Cannot access memory at address 0x839de8000
            (gdb) bt full
            #0 0x000000082f969454 in exit () from /lib/libc.so.7
            No symbol table info available.
            #1 0x00000000006de629 in ?? ()
            No symbol table info available.
            #2 0x000000000061d9ac in ?? ()
            No symbol table info available.
            #3 0x000000000061ac4e in AppLayerProtoDetectGetProto ()
            No symbol table info available.
            #4 0x00000000006197c9 in ?? ()
            No symbol table info available.
            #5 0x0000000000619439 in AppLayerHandleTCPData ()
            No symbol table info available.
            #6 0x00000000005aee4a in StreamTcpReassembleAppLayer ()
            No symbol table info available.
            #7 0x00000000005af9e2 in StreamTcpReassembleHandleSegment ()
            No symbol table info available.
            #8 0x00000000005b2b9f in ?? ()
            No symbol table info available.
            #9 0x00000000005b15e2 in StreamTcpPacket ()
            No symbol table info available.
            #10 0x00000000005b7817 in StreamTcp ()
            No symbol table info available.
            #11 0x00000000006731c1 in ?? ()
            No symbol table info available.
            #12 0x0000000000672a1a in ?? ()
            No symbol table info available.
            #13 0x00000000006a7fe7 in ?? ()
            No symbol table info available.
            #14 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #15 0x0000000000000000 in ?? ()
            No symbol table info available.
            Backtrace stopped: Cannot access memory at address 0x839de8000
            (gdb) info threads
            Id Target Id Frame
            1 LWP 100234 of process 86533 0x000000082f9456ea in _nanosleep () from /lib/libc.so.7
            2 LWP 561146 of process 86533 "IM#01" 0x000000082f9457ea in _read () from /lib/libc.so.7
            3 LWP 561670 of process 86533 "RX#01-vmx2" 0x000000082f9446aa in _umtx_op () from /lib/libc.so.7
            4 LWP 561671 of process 86533 "W#01" 0x000000082b0b0fdc in ?? () from /lib/libthr.so.3
            5 LWP 561672 of process 86533 "W#02" 0x000000082b0b0fdc in ?? () from /lib/libthr.so.3
            6 LWP 561673 of process 86533 "W#03" 0x0000000000672490 in ?? ()
            7 LWP 561674 of process 86533 "W#04" 0x0000000000671127 in ?? ()
            8 LWP 561675 of process 86533 "W#05" 0x000000082b0b0fdc in ?? () from /lib/libthr.so.3
            9 LWP 561676 of process 86533 "W#06" 0x000000082ddb9d48 in ?? () from /usr/local/lib/libhs.so.5

            • 10 LWP 561677 of process 86533 "W#07" 0x000000082f969454 in exit () from /lib/libc.so.7
              11 LWP 561678 of process 86533 "W#08" 0x00000000005a44d1 in DetectEnginePktInspectionRun ()
              12 LWP 561679 of process 86533 "FM#01" 0x000000082b0b0fdc in ?? () from /lib/libthr.so.3
              13 LWP 561680 of process 86533 "FR#01" 0x000000082b0b0fdc in ?? () from /lib/libthr.so.3
              (gdb) thread apply all bt

            Thread 13 (LWP 561680 of process 86533 "FR#01"):
            #0 0x000000082b0b0fdc in ?? () from /lib/libthr.so.3
            #1 0x000000082b0c1022 in ?? () from /lib/libthr.so.3
            #2 0x000000082b0b2b9d in ?? () from /lib/libthr.so.3
            #3 0x00000000005ecb12 in ?? ()
            #4 0x00000000006a87a8 in ?? ()
            #5 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            #6 0x0000000000000000 in ?? ()
            Backtrace stopped: Cannot access memory at address 0x83c301000

            Thread 12 (LWP 561679 of process 86533 "FM#01"):
            #0 0x000000082b0b0fdc in ?? () from /lib/libthr.so.3
            #1 0x000000082b0c1022 in ?? () from /lib/libthr.so.3
            #2 0x000000082b0b2b9d in ?? () from /lib/libthr.so.3
            #3 0x00000000005ec633 in ?? ()
            #4 0x00000000006a87a8 in ?? ()
            #5 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            #6 0x0000000000000000 in ?? ()
            Backtrace stopped: Cannot access memory at address 0x83b685000

            Thread 11 (LWP 561678 of process 86533 "W#08"):
            #0 0x00000000005a44d1 in DetectEnginePktInspectionRun ()
            #1 0x000000000067114b in ?? ()
            #2 0x00000000006706db in Detect ()
            #3 0x0000000000672ac9 in ?? ()
            #4 0x00000000006a7fe7 in ?? ()
            #5 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            #6 0x0000000000000000 in ?? ()
            Backtrace stopped: Cannot access memory at address 0x83a717000

            Thread 10 (LWP 561677 of process 86533 "W#07"):
            #0 0x000000082f969454 in exit () from /lib/libc.so.7
            #1 0x00000000006de629 in ?? ()
            #2 0x000000000061d9ac in ?? ()
            #3 0x000000000061ac4e in AppLayerProtoDetectGetProto ()
            #4 0x00000000006197c9 in ?? ()
            #5 0x0000000000619439 in AppLayerHandleTCPData ()
            #6 0x00000000005aee4a in StreamTcpReassembleAppLayer ()
            #7 0x00000000005af9e2 in StreamTcpReassembleHandleSegment ()
            #8 0x00000000005b2b9f in ?? ()
            #9 0x00000000005b15e2 in StreamTcpPacket ()
            #10 0x00000000005b7817 in StreamTcp ()
            #11 0x00000000006731c1 in ?? ()
            #12 0x0000000000672a1a in ?? ()
            #13 0x00000000006a7fe7 in ?? ()
            #14 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            #15 0x0000000000000000 in ?? ()
            --Type <RET> for more, q to quit, c to continue without paging--
            Backtrace stopped: Cannot access memory at address 0x839de8000

            Thread 9 (LWP 561676 of process 86533 "W#06"):
            #0 0x000000082ddb9d48 in ?? () from /usr/local/lib/libhs.so.5
            #1 0x000000082dde6ed3 in ?? () from /usr/local/lib/libhs.so.5
            #2 0x000000082dd33a87 in hs_scan () from /usr/local/lib/libhs.so.5
            #3 0x00000000006d4050 in SCHSSearch ()
            #4 0x00000000005cc6fe in Prefilter ()
            #5 0x0000000000670e15 in ?? ()
            #6 0x00000000006706db in Detect ()
            #7 0x0000000000672ac9 in ?? ()
            #8 0x00000000006a7fe7 in ?? ()
            #9 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            #10 0x0000000000000000 in ?? ()
            Backtrace stopped: Cannot access memory at address 0x839723000

            Thread 8 (LWP 561675 of process 86533 "W#05"):
            #0 0x000000082b0b0fdc in ?? () from /lib/libthr.so.3
            #1 0x000000082b0b8f56 in pthread_mutex_unlock () from /lib/libthr.so.3
            #2 0x000000082b0b2730 in __pthread_cleanup_pop_imp () from /lib/libthr.so.3
            #3 0x000000082f929e2a in vfprintf_l () from /lib/libc.so.7
            #4 0x000000082f9228df in fprintf () from /lib/libc.so.7
            #5 0x00000000006b071e in SCLogMessage ()
            #6 0x00000000006b13b6 in SCLogErr ()
            #7 0x00000000006de61f in ?? ()
            #8 0x000000000061d9ac in ?? ()
            #9 0x000000000061ac4e in AppLayerProtoDetectGetProto ()
            #10 0x00000000006197c9 in ?? ()
            #11 0x0000000000619439 in AppLayerHandleTCPData ()
            #12 0x00000000005aee4a in StreamTcpReassembleAppLayer ()
            #13 0x00000000005af9e2 in StreamTcpReassembleHandleSegment ()
            #14 0x00000000005b2b9f in ?? ()
            #15 0x00000000005b15e2 in StreamTcpPacket ()
            #16 0x00000000005b7817 in StreamTcp ()
            #17 0x00000000006731c1 in ?? ()
            #18 0x0000000000672a1a in ?? ()
            #19 0x00000000006a7fe7 in ?? ()
            #20 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            #21 0x0000000000000000 in ?? ()
            Backtrace stopped: Cannot access memory at address 0x839405000

            Thread 7 (LWP 561674 of process 86533 "W#04"):
            #0 0x0000000000671127 in ?? ()
            #1 0x00000000006706db in Detect ()
            #2 0x0000000000672ac9 in ?? ()
            #3 0x00000000006a7fe7 in ?? ()
            #4 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            #5 0x0000000000000000 in ?? ()
            --Type <RET> for more, q to quit, c to continue without paging--
            Backtrace stopped: Cannot access memory at address 0x839183000

            Thread 6 (LWP 561673 of process 86533 "W#03"):
            #0 0x0000000000672490 in ?? ()
            #1 0x0000000000671118 in ?? ()
            #2 0x00000000006706db in Detect ()
            #3 0x0000000000672ac9 in ?? ()
            #4 0x00000000006a7fe7 in ?? ()
            #5 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            #6 0x0000000000000000 in ?? ()
            Backtrace stopped: Cannot access memory at address 0x838061000

            Thread 5 (LWP 561672 of process 86533 "W#02"):
            #0 0x000000082b0b0fdc in ?? () from /lib/libthr.so.3
            #1 0x000000082b0c1022 in ?? () from /lib/libthr.so.3
            #2 0x000000082b0b2b9d in ?? () from /lib/libthr.so.3
            #3 0x00000000006a3cc9 in TmqhInputFlow ()
            #4 0x00000000006a7fb3 in ?? ()
            #5 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            #6 0x0000000000000000 in ?? ()
            Backtrace stopped: Cannot access memory at address 0x8371ba000

            Thread 4 (LWP 561671 of process 86533 "W#01"):
            #0 0x000000082b0b0fdc in ?? () from /lib/libthr.so.3
            #1 0x000000082b0c1022 in ?? () from /lib/libthr.so.3
            #2 0x000000082b0b2b9d in ?? () from /lib/libthr.so.3
            #3 0x00000000006a3cc9 in TmqhInputFlow ()
            #4 0x00000000006a7fb3 in ?? ()
            #5 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            #6 0x0000000000000000 in ?? ()
            Backtrace stopped: Cannot access memory at address 0x83620c000

            Thread 3 (LWP 561670 of process 86533 "RX#01-vmx2"):
            #0 0x000000082f9446aa in _umtx_op () from /lib/libc.so.7
            #1 0x000000082b0b8f75 in pthread_mutex_unlock () from /lib/libthr.so.3
            #2 0x00000000006740b1 in ?? ()
            #3 0x000000082c1f4ff4 in ?? () from /usr/local/lib/libpcap.so.1
            #4 0x00000000006737b7 in ?? ()
            #5 0x00000000006a83aa in ?? ()
            #6 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            #7 0x0000000000000000 in ?? ()
            Backtrace stopped: Cannot access memory at address 0x835349000

            Thread 2 (LWP 561146 of process 86533 "IM#01"):
            #0 0x000000082f9457ea in _read () from /lib/libc.so.7
            #1 0x000000082b0bfa13 in ?? () from /lib/libthr.so.3
            #2 0x00000000006355ed in AlertPfMonitorIfaceChanges ()
            #3 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            --Type <RET> for more, q to quit, c to continue without paging--
            #4 0x0000000000000000 in ?? ()
            Backtrace stopped: Cannot access memory at address 0x83423d000

            Thread 1 (LWP 100234 of process 86533):
            #0 0x000000082f9456ea in _nanosleep () from /lib/libc.so.7
            #1 0x000000082b0bf82c in ?? () from /lib/libthr.so.3
            #2 0x000000082f8c9c46 in usleep () from /lib/libc.so.7
            #3 0x000000000059fa6a in ?? ()
            #4 0x000000000059f3b4 in SuricataMain ()
            #5 0x000000082f89b6fa in __libc_start1 () from /lib/libc.so.7
            #6 0x000000000059bea0 in _start ()
            (gdb) thread apply all bt full

            Thread 13 (LWP 561680 of process 86533 "FR#01"):
            #0 0x000000082b0b0fdc in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #1 0x000000082b0c1022 in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #2 0x000000082b0b2b9d in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #3 0x00000000005ecb12 in ?? ()
            No symbol table info available.
            #4 0x00000000006a87a8 in ?? ()
            No symbol table info available.
            #5 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #6 0x0000000000000000 in ?? ()
            No symbol table info available.
            Backtrace stopped: Cannot access memory at address 0x83c301000

            Thread 12 (LWP 561679 of process 86533 "FM#01"):
            #0 0x000000082b0b0fdc in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #1 0x000000082b0c1022 in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #2 0x000000082b0b2b9d in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #3 0x00000000005ec633 in ?? ()
            No symbol table info available.
            #4 0x00000000006a87a8 in ?? ()
            No symbol table info available.
            #5 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #6 0x0000000000000000 in ?? ()
            No symbol table info available.
            Backtrace stopped: Cannot access memory at address 0x83b685000

            Thread 11 (LWP 561678 of process 86533 "W#08"):
            #0 0x00000000005a44d1 in DetectEnginePktInspectionRun ()
            No symbol table info available.
            #1 0x000000000067114b in ?? ()
            No symbol table info available.
            #2 0x00000000006706db in Detect ()
            No symbol table info available.
            #3 0x0000000000672ac9 in ?? ()
            No symbol table info available.
            #4 0x00000000006a7fe7 in ?? ()
            No symbol table info available.
            #5 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            No symbol table info available.
            --Type <RET> for more, q to quit, c to continue without paging--
            #6 0x0000000000000000 in ?? ()
            No symbol table info available.
            Backtrace stopped: Cannot access memory at address 0x83a717000

            Thread 10 (LWP 561677 of process 86533 "W#07"):
            #0 0x000000082f969454 in exit () from /lib/libc.so.7
            No symbol table info available.
            #1 0x00000000006de629 in ?? ()
            No symbol table info available.
            #2 0x000000000061d9ac in ?? ()
            No symbol table info available.
            #3 0x000000000061ac4e in AppLayerProtoDetectGetProto ()
            No symbol table info available.
            #4 0x00000000006197c9 in ?? ()
            No symbol table info available.
            #5 0x0000000000619439 in AppLayerHandleTCPData ()
            No symbol table info available.
            #6 0x00000000005aee4a in StreamTcpReassembleAppLayer ()
            No symbol table info available.
            #7 0x00000000005af9e2 in StreamTcpReassembleHandleSegment ()
            No symbol table info available.
            #8 0x00000000005b2b9f in ?? ()
            No symbol table info available.
            #9 0x00000000005b15e2 in StreamTcpPacket ()
            No symbol table info available.
            #10 0x00000000005b7817 in StreamTcp ()
            No symbol table info available.
            #11 0x00000000006731c1 in ?? ()
            No symbol table info available.
            #12 0x0000000000672a1a in ?? ()
            No symbol table info available.
            #13 0x00000000006a7fe7 in ?? ()
            No symbol table info available.
            #14 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #15 0x0000000000000000 in ?? ()
            No symbol table info available.
            Backtrace stopped: Cannot access memory at address 0x839de8000

            Thread 9 (LWP 561676 of process 86533 "W#06"):
            #0 0x000000082ddb9d48 in ?? () from /usr/local/lib/libhs.so.5
            No symbol table info available.
            #1 0x000000082dde6ed3 in ?? () from /usr/local/lib/libhs.so.5
            No symbol table info available.
            #2 0x000000082dd33a87 in hs_scan () from /usr/local/lib/libhs.so.5
            No symbol table info available.
            #3 0x00000000006d4050 in SCHSSearch ()
            No symbol table info available.
            --Type <RET> for more, q to quit, c to continue without paging--
            #4 0x00000000005cc6fe in Prefilter ()
            No symbol table info available.
            #5 0x0000000000670e15 in ?? ()
            No symbol table info available.
            #6 0x00000000006706db in Detect ()
            No symbol table info available.
            #7 0x0000000000672ac9 in ?? ()
            No symbol table info available.
            #8 0x00000000006a7fe7 in ?? ()
            No symbol table info available.
            #9 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #10 0x0000000000000000 in ?? ()
            No symbol table info available.
            Backtrace stopped: Cannot access memory at address 0x839723000

            Thread 8 (LWP 561675 of process 86533 "W#05"):
            #0 0x000000082b0b0fdc in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #1 0x000000082b0b8f56 in pthread_mutex_unlock () from /lib/libthr.so.3
            No symbol table info available.
            #2 0x000000082b0b2730 in __pthread_cleanup_pop_imp () from /lib/libthr.so.3
            No symbol table info available.
            #3 0x000000082f929e2a in vfprintf_l () from /lib/libc.so.7
            No symbol table info available.
            #4 0x000000082f9228df in fprintf () from /lib/libc.so.7
            No symbol table info available.
            #5 0x00000000006b071e in SCLogMessage ()
            No symbol table info available.
            #6 0x00000000006b13b6 in SCLogErr ()
            No symbol table info available.
            #7 0x00000000006de61f in ?? ()
            No symbol table info available.
            #8 0x000000000061d9ac in ?? ()
            No symbol table info available.
            #9 0x000000000061ac4e in AppLayerProtoDetectGetProto ()
            No symbol table info available.
            #10 0x00000000006197c9 in ?? ()
            No symbol table info available.
            #11 0x0000000000619439 in AppLayerHandleTCPData ()
            No symbol table info available.
            #12 0x00000000005aee4a in StreamTcpReassembleAppLayer ()
            No symbol table info available.
            #13 0x00000000005af9e2 in StreamTcpReassembleHandleSegment ()
            No symbol table info available.
            #14 0x00000000005b2b9f in ?? ()
            No symbol table info available.
            #15 0x00000000005b15e2 in StreamTcpPacket ()
            --Type <RET> for more, q to quit, c to continue without paging--
            No symbol table info available.
            #16 0x00000000005b7817 in StreamTcp ()
            No symbol table info available.
            #17 0x00000000006731c1 in ?? ()
            No symbol table info available.
            #18 0x0000000000672a1a in ?? ()
            No symbol table info available.
            #19 0x00000000006a7fe7 in ?? ()
            No symbol table info available.
            #20 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #21 0x0000000000000000 in ?? ()
            No symbol table info available.
            Backtrace stopped: Cannot access memory at address 0x839405000

            Thread 7 (LWP 561674 of process 86533 "W#04"):
            #0 0x0000000000671127 in ?? ()
            No symbol table info available.
            #1 0x00000000006706db in Detect ()
            No symbol table info available.
            #2 0x0000000000672ac9 in ?? ()
            No symbol table info available.
            #3 0x00000000006a7fe7 in ?? ()
            No symbol table info available.
            #4 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #5 0x0000000000000000 in ?? ()
            No symbol table info available.
            Backtrace stopped: Cannot access memory at address 0x839183000

            Thread 6 (LWP 561673 of process 86533 "W#03"):
            #0 0x0000000000672490 in ?? ()
            No symbol table info available.
            #1 0x0000000000671118 in ?? ()
            No symbol table info available.
            #2 0x00000000006706db in Detect ()
            No symbol table info available.
            #3 0x0000000000672ac9 in ?? ()
            No symbol table info available.
            #4 0x00000000006a7fe7 in ?? ()
            No symbol table info available.
            #5 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #6 0x0000000000000000 in ?? ()
            No symbol table info available.
            Backtrace stopped: Cannot access memory at address 0x838061000

            Thread 5 (LWP 561672 of process 86533 "W#02"):
            --Type <RET> for more, q to quit, c to continue without paging--
            #0 0x000000082b0b0fdc in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #1 0x000000082b0c1022 in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #2 0x000000082b0b2b9d in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #3 0x00000000006a3cc9 in TmqhInputFlow ()
            No symbol table info available.
            #4 0x00000000006a7fb3 in ?? ()
            No symbol table info available.
            #5 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #6 0x0000000000000000 in ?? ()
            No symbol table info available.
            Backtrace stopped: Cannot access memory at address 0x8371ba000

            Thread 4 (LWP 561671 of process 86533 "W#01"):
            #0 0x000000082b0b0fdc in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #1 0x000000082b0c1022 in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #2 0x000000082b0b2b9d in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #3 0x00000000006a3cc9 in TmqhInputFlow ()
            No symbol table info available.
            #4 0x00000000006a7fb3 in ?? ()
            No symbol table info available.
            #5 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #6 0x0000000000000000 in ?? ()
            No symbol table info available.
            Backtrace stopped: Cannot access memory at address 0x83620c000

            Thread 3 (LWP 561670 of process 86533 "RX#01-vmx2"):
            #0 0x000000082f9446aa in _umtx_op () from /lib/libc.so.7
            No symbol table info available.
            #1 0x000000082b0b8f75 in pthread_mutex_unlock () from /lib/libthr.so.3
            No symbol table info available.
            #2 0x00000000006740b1 in ?? ()
            No symbol table info available.
            #3 0x000000082c1f4ff4 in ?? () from /usr/local/lib/libpcap.so.1
            No symbol table info available.
            #4 0x00000000006737b7 in ?? ()
            No symbol table info available.
            #5 0x00000000006a83aa in ?? ()
            No symbol table info available.
            #6 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            No symbol table info available.
            --Type <RET> for more, q to quit, c to continue without paging--
            #7 0x0000000000000000 in ?? ()
            No symbol table info available.
            Backtrace stopped: Cannot access memory at address 0x835349000

            Thread 2 (LWP 561146 of process 86533 "IM#01"):
            #0 0x000000082f9457ea in _read () from /lib/libc.so.7
            No symbol table info available.
            #1 0x000000082b0bfa13 in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #2 0x00000000006355ed in AlertPfMonitorIfaceChanges ()
            No symbol table info available.
            #3 0x000000082b0b3d25 in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #4 0x0000000000000000 in ?? ()
            No symbol table info available.
            Backtrace stopped: Cannot access memory at address 0x83423d000

            Thread 1 (LWP 100234 of process 86533):
            #0 0x000000082f9456ea in _nanosleep () from /lib/libc.so.7
            No symbol table info available.
            #1 0x000000082b0bf82c in ?? () from /lib/libthr.so.3
            No symbol table info available.
            #2 0x000000082f8c9c46 in usleep () from /lib/libc.so.7
            No symbol table info available.
            #3 0x000000000059fa6a in ?? ()
            No symbol table info available.
            #4 0x000000000059f3b4 in SuricataMain ()
            No symbol table info available.
            #5 0x000000082f89b6fa in __libc_start1 () from /lib/libc.so.7
            No symbol table info available.
            #6 0x000000000059bea0 in _start ()
            No symbol table info available.

            ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
            Please do not use chat/PM to ask for help
            we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
            Don't forget to Upvote with the 👍 button for any post you find to be helpful.

            bmeeksB 1 Reply Last reply Reply Quote 0
            • bmeeksB
              bmeeks @kiokoman
              last edited by

              @kiokoman said in Suricata process dying due to hyperscan problem:

              LAN interfaces starts but then randomly crashes after a few minutes (max 5 min)

              LAN interface stay up and running if i disable blocking mode

              Hmm...okay, let's see how long the LAN stays up.

              Next question is "what is different in terms of VLANs and things like the Pass List for the two interfaces?"

              You said you cloned the LAN from the WAN, so I assume all the other settings are identical. The only changes would typically be the interface IP addresses and thus the composition of the resultant default Pass List.

              Are both WAN and LAN using the defaults for $HOME_NET, $EXTERNAL_NET, and Pass List?

              kiokomanK 1 Reply Last reply Reply Quote 0
              • kiokomanK
                kiokoman LAYER 8 @bmeeks
                last edited by

                @bmeeks

                wan have no vlan
                lan have 2 vlan 100 and 110

                yup WAN and LAN are using the defaults for $HOME_NET, $EXTERNAL_NET, and Pass List

                ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                Please do not use chat/PM to ask for help
                we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                bmeeksB 1 Reply Last reply Reply Quote 0
                • bmeeksB
                  bmeeks @kiokoman
                  last edited by

                  @kiokoman said in Suricata process dying due to hyperscan problem:

                  @bmeeks

                  wan have no vlan
                  lan have 2 vlan 100 and 110

                  yup WAN and LAN are using the defaults for $HOME_NET, $EXTERNAL_NET, and Pass List

                  Thank you for the info and test results.

                  Maybe VLANs or no VLANs figures in ???

                  Also a bit weird that Legacy Blocking Mode off seems to help (at least thus far in your individual testing).

                  kiokomanK 1 Reply Last reply Reply Quote 0
                  • kiokomanK
                    kiokoman LAYER 8 @bmeeks
                    last edited by kiokoman

                    @bmeeks

                    Delayed Detect Suricata will build list of signatures after packet capture threads have started. Default is Not Checked.

                    if checked, suricata crash immediately with Hyperscan returned fatal error -1.

                    now i'm trying without vlan and it's up and running for some times ....

                    i can't stay without vlan for too long max 10 min before they find out and sends some ninja to kill me

                    ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                    Please do not use chat/PM to ask for help
                    we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                    Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                    kiokomanK bmeeksB 2 Replies Last reply Reply Quote 0
                    • kiokomanK
                      kiokoman LAYER 8 @kiokoman
                      last edited by

                      @bmeeks

                      it crashed after 15 min but only after reactivating the vlan interfaces

                      maybe it's vlan

                      ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                      Please do not use chat/PM to ask for help
                      we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                      Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                      1 Reply Last reply Reply Quote 0
                      • bmeeksB
                        bmeeks @kiokoman
                        last edited by

                        @kiokoman said in Suricata process dying due to hyperscan problem:

                        @bmeeks

                        Delayed Detect Suricata will build list of signatures after packet capture threads have started. Default is Not Checked.

                        if checked, suricata crash immediately with Hyperscan returned fatal error -1.

                        now i'm trying without vlan and it's up and running for some times ....

                        i can't stay without vlan for too long max 10 min before they find out and sends some ninja to kill me

                        Thank you for that additional info. The only way I can see the Delayed Detect playing into it is maybe increased memory usage (but just a guess).

                        VLANs are one thing I did not test with in my small test environment. I run everything inside VMware Workstation and it does not support VLAN tagging.

                        1 Reply Last reply Reply Quote 0
                        • bmeeksB
                          bmeeks
                          last edited by bmeeks

                          For other users experiencing the Hyperscan crash in Suricata --

                          1. Do you have one or more VLANs configured on the interface that crashes?

                          2. Does disabling blocking mode on the crashing interface result in a difference in behavior?

                          If you have fiddled with the MPM Algorithm setting on the INTERFACE SETTINGS tab, be sure to return it to Auto and save the change before testing.

                          tylereversT NogBadTheBadN 2 Replies Last reply Reply Quote 0
                          • A
                            asdjklfjkdslfdsaklj @bmeeks
                            last edited by

                            @bmeeks anecdotally, yes. ~8hrs after disabling blocking mode and both LAN PHY Suricata instances are still up.

                            bmeeksB 1 Reply Last reply Reply Quote 0
                            • bmeeksB
                              bmeeks @asdjklfjkdslfdsaklj
                              last edited by

                              @asdjklfjkdslfdsaklj said in Suricata process dying due to hyperscan problem:

                              @bmeeks anecdotally, yes. ~8hrs after disabling blocking mode and both LAN PHY Suricata instances are still up.

                              1. I need to know if you have any VLAN configured on either LAN interface.

                              2. Try enabling Blocking Mode on just one of the LAN interfaces and see what happens then.

                              To help me troubleshoot this, I desparately need you folks having the issue to give me some explicit details when responding. For example, answer question #1 above and also try troubleshooting suggestion #2 above. Then follow up back here with detailed results for each.

                              I will repeat again for clarity: I am trying to determine if VLANs configured on the crashing interface are related or not. So, tell me if you have VLANs on the interface, and if you do, how many. Then tell me if you can relate the crash to blocking enabled or not.

                              A 1 Reply Last reply Reply Quote 1
                              • jowe78J
                                jowe78
                                last edited by jowe78

                                It was a little early to sound the all-clear...

                                The suricata interfaces died yesterday in the evening, seems to work better, but not working.

                                However something that is interesting is that I changed to Snort with 7 interfaces (AC-BNFA-NQ), all in blocking mode, and all was up and running in the morning, but ovpn went down and i can't see anything in the log related.

                                Nov 29 05:21:00 php-cgi 23 servicewatchdog_cron.php: Service Watchdog detected service openvpn stopped. Restarting openvpn (OpenVPN server:)
                                Nov 29 05:20:48 kernel ovpns1: link state changed to DOWN
                                Nov 29 05:20:48 kernel pid 35910 (openvpn), jid 0, uid 0: exited on signal 11 (core dumped)
                                Nov 29 05:19:00 sshguard 17586 Now monitoring attacks.
                                Nov 29 05:19:00 sshguard 42196 Exiting on signal.

                                1 Reply Last reply Reply Quote 0
                                • tylereversT
                                  tylerevers @bmeeks
                                  last edited by tylerevers

                                  @bmeeks said in Suricata process dying due to hyperscan problem:

                                  For other users experiencing the Hyperscan crash in Suricata --

                                  1. Do you have one or more VLANs configured on the interface that crashes?

                                  2. Does disabling blocking mode on the crashing interface result in a difference in behavior?

                                  If you have fiddled with the MPM Algorithm setting on the INTERFACE SETTINGS tab, be sure to return it to Auto and save the change before testing.

                                  Let me help you help the community, kind sir.

                                  Environment

                                  1. pfSense+ Plus 23.09-RELEASE
                                  2. suricata 7.0.2_1
                                  3. Dedicated Bare Metal pfSense+ Plus 23.09-RELEASE box acting as RoaS (Router-on-a-Stick): Xeon E5-1650 v0 @ 3.20 GHz; 40GB DDR3 ECC REG BUF; 120GB SSD boot drive
                                  4. Intel X520-DA2 with both SFP+ ports connected via LAGG to UniFi USW Pro 48 PoE on Ports 51-52 Aggregate
                                  5. I have 11 VLANs traversing the LAGG (VLAN 10 to 110 in increments of 10), but only 3 VLANs (30, 50, 60) do I have setup within Suricata.

                                  Answers to Your Questions

                                  Do you have one or more VLANs configured on the interface that crashes?
                                  As detailed above, I run Suricata on 3 out of the 11 VLANs I have in total. Each of these 3 VLANs have their own Interfaces (of course) within Suricata. Only one of these Suricata Interfaces (VLAN 30) is crashing when using Hyperscan, but has run just fine for more than a week or two with AC-KS.

                                  Does disabling blocking mode on the crashing interface result in a difference in behavior?
                                  Please help me understand where this toggle is located as I do not see an entry that says "Blocking Mode" within the GUI. I will test once I understand your request.

                                  Additional Observations

                                  Of the 3 Suricata Interfaces, all had Signature Group Header MPM Context set to Full. For the failing Suricata Interface (VLAN 30), I have set this to Auto and I have returned the Patter Matcher Algorithm to Auto. I will follow-up as soon as I see the Suricata Interface fail.

                                  jowe78J bmeeksB 2 Replies Last reply Reply Quote 0
                                  • jowe78J
                                    jowe78 @tylerevers
                                    last edited by

                                    @tylerevers said in Suricata process dying due to hyperscan problem:

                                    Please help me understand where this toggle is located

                                    Under the Suricata interface you have a box. "Block Offenders". Uncheck and you wont block, just monitor.

                                    tylereversT 1 Reply Last reply Reply Quote 1
                                    • tylereversT
                                      tylerevers @jowe78
                                      last edited by

                                      @jowe78 said in Suricata process dying due to hyperscan problem:

                                      @tylerevers said in Suricata process dying due to hyperscan problem:

                                      Please help me understand where this toggle is located

                                      Under the Suricata interface you have a box. "Block Offenders". Uncheck and you wont block, just monitor.

                                      Thank you. I will wait until I see the Suricata Interface fail with Block Offenders checked and then I shall try with it unchecked.

                                      1 Reply Last reply Reply Quote 0
                                      • bmeeksB
                                        bmeeks @tylerevers
                                        last edited by

                                        @tylerevers said in Suricata process dying due to hyperscan problem:

                                        @bmeeks said in Suricata process dying due to hyperscan problem:

                                        For other users experiencing the Hyperscan crash in Suricata --

                                        1. Do you have one or more VLANs configured on the interface that crashes?

                                        2. Does disabling blocking mode on the crashing interface result in a difference in behavior?

                                        If you have fiddled with the MPM Algorithm setting on the INTERFACE SETTINGS tab, be sure to return it to Auto and save the change before testing.

                                        Let me help you help the community, kind sir.

                                        Environment

                                        1. pfSense+ Plus 23.09-RELEASE
                                        2. suricata 7.0.2_1
                                        3. Dedicated Bare Metal pfSense+ Plus 23.09-RELEASE box acting as RoaS (Router-on-a-Stick): Xeon E5-1650 v0 @ 3.20 GHz; 40GB DDR3 ECC REG BUF; 120GB SSD boot drive
                                        4. Intel X520-DA2 with both SFP+ ports connected via LAGG to UniFi USW Pro 48 PoE on Ports 51-52 Aggregate
                                        5. I have 11 VLANs traversing the LAGG (VLAN 10 to 110 in increments of 10), but only 3 VLANs (30, 50, 60) do I have setup within Suricata.

                                        Answers to Your Questions

                                        Do you have one or more VLANs configured on the interface that crashes?
                                        As detailed above, I run Suricata on 3 out of the 11 VLANs I have in total. Each of these 3 VLANs have their own Interfaces (of course) within Suricata. Only one of these Suricata Interfaces (VLAN 30) is crashing when using Hyperscan, but has run just fine for more than a week or two with AC-KS.

                                        Does disabling blocking mode on the crashing interface result in a difference in behavior?
                                        Please help me understand where this toggle is located as I do not see an entry that says "Blocking Mode" within the GUI. I will test once I understand your request.

                                        Additional Observations

                                        Of the 3 Suricata Interfaces, all had Signature Group Header MPM Context set to Full. For the failing Suricata Interface (VLAN 30), I have set this to Auto and I have returned the Patter Matcher Algorithm to Auto. I will follow-up as soon as I see the Suricata Interface fail.

                                        Thank for the detailed reply.

                                        The "Block Mode" toggle is my generic name for the setting on the INTERFACE SETTINGS tab when you can enable or disable blocking. The setting is in the Alert and Block Settings section of the page. The checkbox is called Block Offenders. Unchecking that box removes all future blocking of offender IP addresses (it will not clear any currently existing blocks). There are also two settings for blocking offenders. One uses the netmap kernel device to implement a true inline-IPS mode of operation. But netmap will not work with VLANs or LAGG interfaces at the moment. You would need to run it on just the parent physical interface. Legacy Mode Blocking uses a custom output plugin compiled into the Suricata binary used on pfSense. This plugin calls a pfctl system function to insert offender IP addresses into a firewall table refereced in a hidden built-in blocking rule in pfSense.

                                        tylereversT 2 Replies Last reply Reply Quote 1
                                        • tylereversT
                                          tylerevers @bmeeks
                                          last edited by

                                          @bmeeks said in Suricata process dying due to hyperscan problem:

                                          The "Block Mode" toggle is my generic name for the setting on the INTERFACE SETTINGS tab when you can enable or disable blocking. The setting is in the Alert and Block Settings section of the page. The checkbox is called Block Offenders. Unchecking that box removes all future blocking of offender IP addresses (it will not clear any currently existing blocks). There are also two settings for blocking offenders. One uses the netmap kernel device to implement a true inline-IPS mode of operation. But netmap will not work with VLANs or LAGG interfaces at the moment. You would need to run it on just the parent physical interface. Legacy Mode Blocking uses a custom output plugin compiled into the Suricata binary used on pfSense. This plugin calls a pfctl system function to insert offender IP addresses into a firewall table refereced in a hidden built-in blocking rule in pfSense.

                                          Thank you for the insights and explanation. I have wanted to use true inline-IPS for some time, but I knew of the tradeoff and I simply cannot give up VLANs/LAGG.

                                          1 Reply Last reply Reply Quote 0
                                          • NogBadTheBadN
                                            NogBadTheBad @bmeeks
                                            last edited by NogBadTheBad

                                            @bmeeks said in Suricata process dying due to hyperscan problem:

                                            For other users experiencing the Hyperscan crash in Suricata --

                                            1. Do you have one or more VLANs configured on the interface that crashes?

                                            Yes on my LAN interface
                                            PPPOE on my WAN interface

                                            1. Does disabling blocking mode on the crashing interface result in a difference in behavior?

                                            Yes for no blocking, ok

                                            Set to AC and blocking ok

                                            Set to Auto and Core dumps

                                            If you have fiddled with the MPM Algorithm setting on the INTERFACE SETTINGS tab, be sure to return it to Auto and save the change before testing.

                                            Currently set not to block so I can rebaseline my suppression list.

                                            Andy

                                            1 x Netgate SG-4860 - 3 x Linksys LGS308P - 1 x Aruba InstantOn AP22

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post