Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Suricata process dying due to hyperscan problem

    Scheduled Pinned Locked Moved IDS/IPS
    295 Posts 25 Posters 89.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • NeganN
      Negan @bmeeks
      last edited by Negan

      @bmeeks

      The thing is I was not looking at any files at the time, anyway, I just Deleted Sucicata as it's been of no benefit since 23.09. so I won't need to post here again, best of luck tracking down the error..

      bmeeksB 1 Reply Last reply Reply Quote 0
      • bmeeksB
        bmeeks @Negan
        last edited by bmeeks

        @Negan said in Suricata process dying due to hyperscan problem:

        @bmeeks

        The thing is I was not looking at any files at the time, anyway, I just Deleted Sucicata as it's been of no benefit since 23.09. so I need to post here again, best of luck tracking down the error..

        You had to have been attempting to view one at some point. That's the only way that error can appear. The thing with the PHP errors, though, is that the Dashboard is the only place they will be shown. So if you tried to view a log two or three days ago, but never visited the Dashboard in the interim, then the next time you do open the Dashboard you will see the PHP error that was recorded in the past.

        According to the error, someone attempted to view a log file from the LOGS VIEW tab at this date and time: 19-Dec-2023 13:50:29 Australia/Sydney.

        NeganN 1 Reply Last reply Reply Quote 0
        • NeganN
          Negan @bmeeks
          last edited by

          @bmeeks
          Well I only installed Sucicata again today and it was crashed and the PHP Error was there, anyway, thanks for helping....

          1 Reply Last reply Reply Quote 0
          • kiokomanK
            kiokoman LAYER 8 @masons
            last edited by

            @masons
            i suspected you had a different version of hyperscan because i had that same crash when using hyperscan 5.4.2 instead of 5.4.0
            now that you have corrected the situation I expect that you no longer have the crash/core dump but only hyperscan that fails like me

            ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
            Please do not use chat/PM to ask for help
            we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
            Don't forget to Upvote with the 👍 button for any post you find to be helpful.

            bmeeksB 1 Reply Last reply Reply Quote 0
            • bmeeksB
              bmeeks @kiokoman
              last edited by

              @kiokoman and @masons:
              The upstream Suricata developer team took a look at your gdb back trace reports and has asked me to prepare a new debug Suricata package with the Address Sanitizer option enabled and let you run that for a while. That option performs a very detailed memory use analysis at runtime and can print more detailed debugging messages.

              I'm working on creating that build now and will test it on my system first to be sure it functions. Once I get everything ready, I will post a link to it back here in this thread.

              kiokomanK 1 Reply Last reply Reply Quote 0
              • kiokomanK
                kiokoman LAYER 8 @bmeeks
                last edited by

                @bmeeks
                Nice, ready when you are

                ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                Please do not use chat/PM to ask for help
                we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                bmeeksB 1 Reply Last reply Reply Quote 0
                • bmeeksB
                  bmeeks @kiokoman
                  last edited by

                  @kiokoman said in Suricata process dying due to hyperscan problem:

                  @bmeeks
                  Nice, ready when you are

                  I haven't forgotten. Ran the ASAN-enabled package first on my virtual machine and right away discovered another bug in the custom blocking module I wrote. Color me doubly embarrassed 😊. This one was a missing address-of ('&') operator in two key places. Surprised the compiler never complained about them, but they both tripped ASAN right away.

                  So, tidying up and retesting after correcting that bug before posting the ASAN-enabled package for you guys to test.

                  1 Reply Last reply Reply Quote 0
                  • bmeeksB
                    bmeeks
                    last edited by bmeeks

                    @kiokoman and @masons:

                    Here is a link to the ASAN-enabled (Adress Sanitizer) debug build of Suricata 7.0.2. Please install and try this version.

                    Here is the link to the package binary on my Google Drive: https://drive.google.com/file/d/1y3ohuWyhK3IrHb_p5nVVkc4OwJJBhg0U/view?usp=drive_link. The file has the same name (suricata-7.0.2_6.pkg).

                    Repeat the steps outlined in this post to install the debug package: https://forum.netgate.com/topic/184101/suricata-process-dying-due-to-hyperscan-problem/223.

                    You have two options for doing the initial test run. Because the ASAN runtime linked into the binary really only outputs information to the console, you must run Suricata from the command-line to get the ASAN output. But I would first just try to run it from the GUI to see if it crashes still. It is possible the last two bugs I found and fixed this morning in the custom blocking module were the cause of the Hyperscan issue. So, just first try starting this new version from the GUI. If it starts and stays running, then the bug is fixed. If it crashes, then proceed to the steps below to run it from a CLI prompt in order to capture the ASAN output.

                    To Capture ASAN Output
                    You will need to start Suricata from a shell prompt on the firewall. You will want to use PuTTY with an SSH session so that you can easily scroll back and see past console output.

                    1. Establish an SSH session to the firewall using PuTTY.
                    2. Exit the menu into a shell prompt by choosing option #8.
                    3. Identify the full path to the correct suricata.yaml configuration for the interface you are testing. Look at the subdirectories under /usr/local/etc/suricata/ and find the suricata_xxxx_yyyy directory that matches the interface you want to test.
                    4. Note the physical interface name used by pfSense (for example, em0, vmx1, etc.). The interface name will be part of the subdirectory name.
                    5. With the above two pieces of info: (1) path to suricata.yaml for the interface; and (2) the physical interface name, start Suricata using this command:
                    /usr/local/bin/suricata -i <iface> -c /usr/local/etc/suricata/<suricata_xxxx_yyyy>/suricata.yaml
                    

                    For example, here is the command for my virtual machine where I am using the em0 interface and the interface's conf directory is suricata_13473_em0:

                    /usr/local/bin/suricata -i em0 -c /usr/local/suricata/suricata_13473_em0/suricata.yaml
                    

                    Suricata should start and begin spewing data to the console. If it hits a memory issue, the ASAN module will spit out some colorful data and exit the program. If there are no ASAN-detected errors, the program will just seem to freeze as it was not started in daemon mode. A simple CTRL-C from the keyboard will break out of the program.

                    kiokomanK M 2 Replies Last reply Reply Quote 0
                    • bmeeksB bmeeks referenced this topic on
                    • kiokomanK
                      kiokoman LAYER 8 @bmeeks
                      last edited by

                      @bmeeks
                      still running man .. still running .... 🤞 🤞 🤞 🤞 🤞 🤞 🤞
                      for me it's a record, 10 min without any hyperscan error

                      ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                      Please do not use chat/PM to ask for help
                      we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                      Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                      bmeeksB 1 Reply Last reply Reply Quote 0
                      • bmeeksB
                        bmeeks @kiokoman
                        last edited by bmeeks

                        @kiokoman said in Suricata process dying due to hyperscan problem:

                        @bmeeks
                        still running man .. still running .... 🤞 🤞 🤞 🤞 🤞 🤞 🤞
                        for me it's a record, 10 min without any hyperscan error

                        This is good! Maybe I did find it this time.

                        The ASAN-enabled debug package is going to run quite a bit slower. If your results continue to be positive into the next day, then I will prepare a pull request with the fixes and get it submitted to the Netgate developer team so we can get a new package posted for production use.

                        There were two places in the custom blocking module code where an IPv4 and an IPv6 address were being copied using memcpy() and I had left off the address-of operator on one of the parameters (the destination memory address). That resulted in heap buffer overflows that would have produced random results depending on the runtime memory configuration of a particular installation. One manifestation could be corrupted pointers that were later passed to Hyperscan.

                        This would explain the connection between disabling Legacy Blocking Mode and successful execution with Hyperscan enabled.

                        1 Reply Last reply Reply Quote 1
                        • M
                          masons @bmeeks
                          last edited by

                          @bmeeks

                          My Suricata instances have mostly been running fine since I installed your latest debug package. However, I've noticed a major change in the memory usage and during a rule update tonight both instances were killed due to lack of memory. I assume the memory usage will come back down once a non-debug enabled package is re-deployed, but I just thought I should point this out in case it's helpful.

                          This graph shows memory usage for the past two weeks.
                          ea1c6cfe-6296-4227-af4b-511b08999a99-image.png

                          This graph shows memory usage before and after installing the debug package.
                          948dbf64-7f33-44cc-9ae7-20f1a7e5adcc-image.png

                          Finally, this graph shows the Suricata instances being killed following a rule update. I had to delete the PID files in order to get Suricata to start.
                          048febe4-d836-43a9-8d5b-92264c67fd0a-image.png
                          1af280d4-dccb-49cc-8799-b8b9de6fa35f-image.png

                          bmeeksB 1 Reply Last reply Reply Quote 0
                          • bmeeksB
                            bmeeks @masons
                            last edited by

                            @masons:
                            1, How much RAM is installed in the firewall?
                            2. Are you running with a ZFS or UFS installation?

                            The ASAN-enabled build will run quite a bit slower than a regular build, but I would not expect it to consume all that much more memory.

                            Changing the multi-pattern matcher algorithm can have a very large impact on memory usage, though.

                            M 1 Reply Last reply Reply Quote 0
                            • bmeeksB
                              bmeeks
                              last edited by

                              @kiokoman: do you still have good news for me about the patched Suricata code 🙂 ?

                              If you do, I will create the pull requests for the production update and get them submitted.

                              kiokomanK 1 Reply Last reply Reply Quote 0
                              • bmeeksB
                                bmeeks
                                last edited by bmeeks

                                I have submitted the pull request that hopefully contains the final fix for this Hyperscan crash. It should also address the instances of Signal 11 segfault crashes some other users have experienced in Suricata.

                                Here is the request: https://github.com/pfsense/FreeBSD-ports/pull/1337.

                                There is also a GUI package update to match: https://github.com/pfsense/FreeBSD-ports/pull/1338.

                                Look for a new 7.0.2_3 version of the Suricata package to show up soon.

                                1 Reply Last reply Reply Quote 0
                                • kiokomanK
                                  kiokoman LAYER 8 @bmeeks
                                  last edited by

                                  @bmeeks
                                  yup, still running, i didn't check the memory usage before the patch, i'm around

                                  44% of 16321 MiB

                                  maybe i have 4% 5% more ram usage in total but i have other services like telegraf / haproxy / 5 openvpn / 1 wireguard and at this time there are alot of traffic

                                  ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
                                  Please do not use chat/PM to ask for help
                                  we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
                                  Don't forget to Upvote with the 👍 button for any post you find to be helpful.

                                  bmeeksB 1 Reply Last reply Reply Quote 0
                                  • bmeeksB
                                    bmeeks @kiokoman
                                    last edited by

                                    @kiokoman said in Suricata process dying due to hyperscan problem:

                                    @bmeeks
                                    yup, still running, i didn't check the memory usage before the patch, i'm around

                                    44% of 16321 MiB

                                    maybe i have 4% 5% more ram usage in total but i have other services like telegraf / haproxy / 5 openvpn / 1 wireguard and at this time there are alot of traffic

                                    Great news!

                                    As you see immediately above, I've posted the pull request containing the bug fix for the Netgate developer team to review and merge. Also sent them an email request. Hopefully they get to it soon.

                                    The new package version will be 7.0.2_3 (for the GUI) and will be a normal non-debug production build.

                                    M 1 Reply Last reply Reply Quote 1
                                    • M
                                      michmoor LAYER 8 Rebel Alliance @bmeeks
                                      last edited by

                                      @bmeeks
                                      Great news indeed. Bill, do you have a paetron or some method that we, the community, can contribute to you? Your hard work, your efforts, are not unnoticed, and truly appreciate the efforts you place into making pfSense better with the integration of Suricata.

                                      Firewall: NetGate,Palo Alto-VM,Juniper SRX
                                      Routing: Juniper, Arista, Cisco
                                      Switching: Juniper, Arista, Cisco
                                      Wireless: Unifi, Aruba IAP
                                      JNCIP,CCNP Enterprise

                                      bmeeksB 1 Reply Last reply Reply Quote 0
                                      • bmeeksB
                                        bmeeks @michmoor
                                        last edited by

                                        @michmoor said in Suricata process dying due to hyperscan problem:

                                        @bmeeks
                                        Great news indeed. Bill, do you have a paetron or some method that we, the community, can contribute to you? Your hard work, your efforts, are not unnoticed, and truly appreciate the efforts you place into making pfSense better with the integration of Suricata.

                                        No, I do not have a Paetron account. I do this in the spirit of open-source free software, so I don't expect to make money from it. But thank you for offering.

                                        1 Reply Last reply Reply Quote 2
                                        • NeganN
                                          Negan
                                          last edited by

                                          Just installed 7.0.2_3 a few minutes ago, thanks for getting it fixed....

                                          1 Reply Last reply Reply Quote 0
                                          • B
                                            btspce
                                            last edited by

                                            Just installed 7.0.2_3 on two 6100 in HA. Suricata on three interfaces with MPM: Auto and ET Pro ruleset. No Signal 11 segfault on suricata start now but the three processes of suricata takes 100% cpu and interface goes down with failover to secondary node. After failover, primary node gui hangs and ssh reveals that the three suricata processes takes 100% cpu. Around 30 seconds after failover on secondary node gui hangs there aswell and suricata shows 100% and never let go. After gui hang traffic stops on the interfaces. Suricata 7.0.2_2 was removed and 7.0.2_3 was installed and fw was rebooted after install.

                                            Same behaviour was observed with AC-BS on 7.0.2_2.

                                            No issues with 21.05.1 and that suricata version.

                                            M 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.