Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    SG-3100 on 2.4.4 Rebooting every day at almost same time.

    Scheduled Pinned Locked Moved General pfSense Questions
    17 Posts 5 Posters 1.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator
      last edited by

      Hmm, I used to a hit that a while ago but I thought it was fixed. Certainly I haven't hit it in some time.

      That's definitely a software issue though, better to ask about it in the IDS/IPS board.

      Steve

      1 Reply Last reply Reply Quote 0
      • bmeeksB
        bmeeks @bhjitsense
        last edited by

        @bhjitsense said in SG-3100 on 2.4.4 Rebooting every day at almost same time.:

        @stephenw10 Out of curiosity; I have a 3100 and run Suricata. But I have problems where Suricata will crash and won't start (stale .pid). I've seen some discussion about it but there doesn't seem to be a resolution to this other than deleting the file and restarting. It crashes whenever I make adjustments to pfBlocker or the OpenVPN server and only if Suriata is in blocking mode. Is there going to be a fix for this? It's rather annoying.

        Specifically, what kind of pfBlocker or OpenVPN server adjustment are you making? Is there anything about the crash in the pfSense system log? Look for a Signal 10 error or fault in the pfSense system log (not in the Suricata log). Let me know what you find.

        B 1 Reply Last reply Reply Quote 0
        • B
          bhjitsense @bmeeks
          last edited by

          @bmeeks Yes, they are sig 10 errors. I recreated this one by disabling pfBlocker. I'm currently remote, but I imagine if I restart the OpenVPN server, the same thing would occur.

          bmeeksB 1 Reply Last reply Reply Quote 0
          • bmeeksB
            bmeeks @bhjitsense
            last edited by bmeeks

            @bhjitsense said in SG-3100 on 2.4.4 Rebooting every day at almost same time.:

            @bmeeks Yes, they are sig 10 errors. I recreated this one by disabling pfBlocker. I'm currently remote, but I imagine if I restart the OpenVPN server, the same thing would occur.

            Do the Signal 10 errors reference the Suricata process or something else? As @stephenw10 mentioned in his post, I did make some changes recently to attempt mitigation of the Suricata Signal 10 errors. Those are actually coming from unaligned memory access operations.

            And for clarity to help me understand, you mean that having Suricata and pfBlocker running and then stopping (disabling) pfBlocker will trigger a Suricata Signal 10 error? And same thing with OpenVPN Server?

            B 1 Reply Last reply Reply Quote 0
            • B
              bhjitsense @bmeeks
              last edited by bhjitsense

              @bmeeks Yes, the exact error is pid 58394 (suricata), uid 0, exited on signal 10 (core dumped)

              That's correct, disabling pfBlocker (as an example) causes this error. Same with OVPN server.

              bmeeksB 2 Replies Last reply Reply Quote 0
              • bmeeksB
                bmeeks @bhjitsense
                last edited by bmeeks

                @bhjitsense said in SG-3100 on 2.4.4 Rebooting every day at almost same time.:

                @bmeeks Yes, the exact error is pid 58394 (suricata), uid 0, exited on signal 10 (core dumped)

                That's correct, disabling pfBlocker (as an example) causes this error. Same with OVPN server.

                Okay, thanks! That gives me a possible hint at the problem area. You also said you were using Legacy Mode blocking. This points to the issue being within the custom blocking module I wrote for Suricata. Let me examine that code in greater detail to see where the unaligned access might be triggered. One thing the custom blocking module does is monitor all firewall interface IP addresses for changes. Toggling something like OpenVPN Server (and perhaps pfBlocker) will cause the interfaces to cycle and trigger this monitoring thread within Suricata.

                1 Reply Last reply Reply Quote 0
                • bmeeksB
                  bmeeks @bhjitsense
                  last edited by

                  @bhjitsense said in SG-3100 on 2.4.4 Rebooting every day at almost same time.:

                  @bmeeks Yes, the exact error is pid 58394 (suricata), uid 0, exited on signal 10 (core dumped)

                  That's correct, disabling pfBlocker (as an example) causes this error. Same with OVPN server.

                  Can you please give me one more piece of information -- this time from the suricata.log for the interface. Toggle either OpenVPN Server or pfBlocker to trigger the Suricata Signal 10 error. Then immediately go to the LOGS VIEW tab and select the suricata.log file for the interface and post the last few lines of that file. I'm expecting to see some lines noting that Suricata has detected an IP address change on an interface. You are free to obfuscate the IP addresses if you wish, but I want to know if some of those logging lines are present and what they say. That will help me narrow down precisely which function is the likely culprit.

                  1 Reply Last reply Reply Quote 0
                  • B
                    bhjitsense
                    last edited by

                    @bmeeks

                    These are the last several lines before I triggered a crash.

                    12/2/2019 -- 12:20:11 - <Info> -- Using 1 live device(s).
12/2/2019 -- 12:20:11 - <Info> -- using interface mvneta2
12/2/2019 -- 12:20:11 - <Info> -- Running in 'auto' checksum mode. Detection of interface state will require 1000 packets.
12/2/2019 -- 12:20:11 - <Info> -- Set snaplen to 1518 for 'mvneta2'
12/2/2019 -- 12:20:11 - <Info> -- RunModeIdsPcapAutoFp initialised
12/2/2019 -- 12:20:11 - <Notice> -- all 3 packet processing threads, 4 management threads initialized, engine started.
12/2/2019 -- 12:21:10 - <Info> -- No packets with invalid checksum, assuming checksum offloading is NOT used
                    bmeeksB 1 Reply Last reply Reply Quote 0
                    • bmeeksB
                      bmeeks @bhjitsense
                      last edited by bmeeks

                      @bhjitsense said in SG-3100 on 2.4.4 Rebooting every day at almost same time.:

                      @bmeeks

                      These are the last several lines before I triggered a crash.

                      12/2/2019 -- 12:20:11 - <Info> -- Using 1 live device(s).
12/2/2019 -- 12:20:11 - <Info> -- using interface mvneta2
12/2/2019 -- 12:20:11 - <Info> -- Running in 'auto' checksum mode. Detection of interface state will require 1000 packets.
12/2/2019 -- 12:20:11 - <Info> -- Set snaplen to 1518 for 'mvneta2'
12/2/2019 -- 12:20:11 - <Info> -- RunModeIdsPcapAutoFp initialised
12/2/2019 -- 12:20:11 - <Notice> -- all 3 packet processing threads, 4 management threads initialized, engine started.
12/2/2019 -- 12:21:10 - <Info> -- No packets with invalid checksum, assuming checksum offloading is NOT used

                      Was there anything added to the suricata.log file after you triggered the crash, or is what you posted from after the crash? I was expecting to see one or more lines with info about an interface IP change being detected.

                      If the log was from before the crash, then I need the last few lines of the log after the crash but BEFORE you restart Suricata. Upon a restart Suricata wipes the suricata.log file and starts a new one.

                      And can you verify one more condition for me? With blocking disabled, will it still crash when you toggle the state of pfBlocker or OpenVPN Server?

                      B 1 Reply Last reply Reply Quote 0
                      • B
                        bhjitsense @bmeeks
                        last edited by

                        @bmeeks This was after the crash but before a restart. However, it was only running for a few minutes before I triggered the crash again. For clarity, I just triggered it again. The logs are the same. The last logs were about 30 minutes ago (the exact same as I submitted above), then I triggered the crash. Nothing new was recorded in that log file at the time of the crash.

                        When blocking is disabled, the crashes seem to never happen and I can't seem to trigger it.

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.