Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Suricata core dumping after 2.4.5 upgrade

    Scheduled Pinned Locked Moved IDS/IPS
    27 Posts 6 Posters 1.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      Mobius114
      last edited by Mobius114

      Hey everyone. Long time listener, first time caller.

      After the 2.4.5 update suricata won't start and bombs with the following error:

      pid 91657 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)

      I have removed the interfaces - no go
      I have re-installed the packages - no go
      I have removed, rebooted, re-installed - no go.

      Anyone have any ideas or fixes?

      Running on Netgate SG-1100 appliance.

      bmeeksB 1 Reply Last reply Reply Quote 0
      • bmeeksB
        bmeeks
        last edited by bmeeks

        I have an SG-1100 for testing and will be looking into this issue. The SG-1100 uses a non-Intel CPU and that can bring on a host of potential problems when running C code that was originally developed and widely used on Intel platforms for a long time.

        Snort, Suricata and a few other FreeBSD ports packages have or still have issues with running on non-Intel CPUs (meaning arm and aarch64 processors). Previously identified issues in Snort and Suricata were caused by the way upstream coders had taken some "liberty" with memory pointer casting. I don't know if this new issue with the Suricata binary on aarch64 hardware is more of the same or something totally different.

        1 Reply Last reply Reply Quote 0
        • bmeeksB
          bmeeks @Mobius114
          last edited by bmeeks

          @Mobius114 said in Suricata core dumping after 2.4.5 upgrade:

          Hey everyone. Long time listener, first time caller.

          After the 2.4.5 update suricata won't start and bombs with the following error:

          pid 91657 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)

          I have removed the interfaces - no go
          I have re-installed the packages - no go
          I have removed, rebooted, re-installed - no go.

          Anyone have any ideas or fixes?

          Running on Netgate SG-1100 appliance.

          There is nothing you can do on your end to impact this. It is a problem with the actual compiled code for the package and is due to the SG-1100 appliance having an ARM processor instead of the Intel x86-x64 platform. Signal 4 is the FreeBSD signal for ILLEGAL INSTRUCTION. It indicates an issue with the actual machine code of the Suricata binary executable. It is being looked into, but the fix may not happen quickly.

          M 1 Reply Last reply Reply Quote 2
          • M
            Mobius114 @bmeeks
            last edited by

            @bmeeks Appreciate the update and insights into the issue. I look forward to the fix when it's available.

            It had been working quite well until the upgrade last night and even ran for about 20 minutes before it crapped out completely.

            Cheers

            bmeeksB 1 Reply Last reply Reply Quote 0
            • bmeeksB
              bmeeks @Mobius114
              last edited by bmeeks

              @Mobius114 said in Suricata core dumping after 2.4.5 upgrade:

              @bmeeks Appreciate the update and insights into the issue. I look forward to the fix when it's available.

              It had been working quite well until the upgrade last night and even ran for about 20 minutes before it crapped out completely.

              Cheers

              Oh, so it starts and runs for a while and then crashes? My initial info led me to believe it would not even start up.

              Unfortunately, running for a while and then randomly crashing makes finding the issue way harder. Much easier to find something that fails immediately and the same way every time.

              My SG-1100 is new in the box and I have not actually set it up for testing yet. My main firewall is an SG-5100. Those are Intel-based platforms.

              M 1 Reply Last reply Reply Quote 0
              • M
                Mobius114 @bmeeks
                last edited by

                @bmeeks ah I wasn't clear on that.

                It ran initially after the upgrade and I got a few alerts before it stopped.

                Since that time it has never started since. The core dump error is almost instantaneous.

                bmeeksB 1 Reply Last reply Reply Quote 0
                • bmeeksB
                  bmeeks @Mobius114
                  last edited by

                  @Mobius114 said in Suricata core dumping after 2.4.5 upgrade:

                  @bmeeks ah I wasn't clear on that.

                  It ran initially after the upgrade and I got a few alerts before it stopped.

                  Since that time it has never started since. The core dump error is almost instantaneous.

                  OK. That sounds more like what I had previously heard. Well, as I said, the good news in that is that something immediate and repeatable will hopefully be easier to identify.

                  1 Reply Last reply Reply Quote 1
                  • H
                    h3llo
                    last edited by

                    Same boat for me. Suricata will run for a few seconds (enough to show a green status on the dashboard / interface list) then it dumps.

                    The suricata.log is blank. Here's from the last run:

                    1/4/2020 -- 20:21:11 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
                    1/4/2020 -- 20:21:11 - <Info> -- CPUs/cores online: 2
                    1/4/2020 -- 20:21:11 - <Info> -- HTTP memcap: 67108864
                    1/4/2020 -- 20:21:11 - <Notice> -- using flow hash instead of active packets
                    

                    If you restart you get the "[ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata_mvneta0.409033796.pid' exists but appears stale." issue. If you rm the file via the command prompt it will restart momentarily then dump again.

                    Additionally, I now have a crash notification on the dashboard. "pfSense has detected a crash report or programming bug. Click here for more information." Detail page shows the following:

                    Crash report begins.  Anonymous machine information:
                    
                    arm64
                    11.3-STABLE
                    FreeBSD 11.3-STABLE #233 2c992b2181a(factory-RELENG_2_4_5): Tue Mar 24 15:27:54 EDT 2020     root@buildbot3-nyi.netgate.com:/build/factory-crossbuild-245-aarch64/obj/aarch64/Og10eFss/build/factory-crossbuild-245-aarch64/sources/FreeBSD-src/sys/pfSense
                    
                    Crash report details:
                    
                    PHP Errors:
                    [01-Apr-2020 19:36:48 America/Los_Angeles] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 137897032 bytes) in /usr/local/www/csrf/csrf-magic.php on line 149
                    
                    No FreeBSD crash data found.
                    

                    I haven't changed any memory allocation with pfsense or Suricata and never have seen this error pop. The dashboard still lists memory usage in the mid-20%.

                    Only other thing I can find in terms of logs is the Status -> System Log -> System -> General and it reads as follows:

                    Apr 1 20:10:30 	kernel 		pid 1445 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)
                    

                    I run an extremely basic setup with suricata as really the only service running other than some basic LAN isolation firewall settings.

                    Let me know if there is any other information I might be able to add.

                    1 Reply Last reply Reply Quote 0
                    • H
                      h3llo
                      last edited by

                      I should add that Suricata and the SG-1100 ran problem free for six months after some initial, minor adjustments with the flow/stream settings and the usual tweaking of the rules.

                      bmeeksB 1 Reply Last reply Reply Quote 1
                      • bmeeksB
                        bmeeks @h3llo
                        last edited by bmeeks

                        @h3llo said in Suricata core dumping after 2.4.5 upgrade:

                        I should add that Suricata and the SG-1100 ran problem free for six months after some initial, minor adjustments with the flow/stream settings and the usual tweaking of the rules.

                        Did you not read my posts up above where I described this problem and the fact it is a problem within the binary machine code itself? Here are the relevant lines from one of those posts:

                        There is nothing you can do on your end to impact this. It is a problem with the actual compiled code for the package and is due to the SG-1100 appliance having an ARM processor instead of the Intel x86-x64 platform. Signal 4 is the FreeBSD signal for ILLEGAL INSTRUCTION. It indicates an issue with the actual machine code of the Suricata binary executable. It is being looked into, but the fix may not happen quickly.

                        The SG-1100 has a non-Intel architecture CPU. It is an aarch64 instead of an x86-x64. Fixing this one may not happen quickly at all, so if you want an IDS/IPS package on an SG-1100 you should consider switching over to Snort -- at least for the foreseeable future. I know what the problem is, but not where in the megabytes of C and Rust source code it is located. And I'm talking about the Suricata binary source code which comes from upstream - code I did not write.

                        1 Reply Last reply Reply Quote 0
                        • H
                          h3llo
                          last edited by

                          Yes sir I did read the posts above. I just added some additional details regarding the specific errors thrown in the off chance that it might be helpful. If it is not, ignore away.

                          Appreciate the feedback particularly the expectation management in terms of a fix. While I'm bummed that what once worked does not now, I will take the opportunity to learn a bit about Snort and give it a go.

                          bmeeksB 1 Reply Last reply Reply Quote 0
                          • bmeeksB
                            bmeeks @h3llo
                            last edited by

                            @h3llo said in Suricata core dumping after 2.4.5 upgrade:

                            Yes sir I did read the posts above. I just added some additional details regarding the specific errors thrown in the off chance that it might be helpful. If it is not, ignore away.

                            Appreciate the feedback particularly the expectation management in terms of a fix. While I'm bummed that what once worked does not now, I will take the opportunity to learn a bit about Snort and give it a go.

                            Blame this problem on two things. One is the Suricata upstream team deciding to use Rust in the binary and to make it mandatory starting with version 5.x. The other is the folks who created the llvm compiler used to create armv6/v7 and aarch64 binary from C source code. That compiler makes some binary instruction choices that are "how shall I say?" -- not optimal ... ☺ .

                            1 Reply Last reply Reply Quote 0
                            • B
                              bgalakazam
                              last edited by

                              I just bought a SG-1100 and have this same issue. Sad to find out that it may not be fixed soon. I'll look into Snort in the meantime.

                              1 Reply Last reply Reply Quote 0
                              • B
                                blaytrail
                                last edited by

                                I just posted the same issue after the 2.4.5 Release. I've been working on this for 2 days now. I see I'm not the only one having this issue. See my post below.

                                Suricata will not start using pfsense. I currently have the SG-1100 appliance.

                                Added the Suricata package with no issues.
                                Configured Global Setting using the ETOpen Emerging Threats rules only. No Issues
                                Performed the Emerging Threats Open Rules update with no issues.
                                Enabled emerging rules. No Issues
                                The SG-1100 appliance only has 2 CPU's with 1GB of Ram. It should still run the Suricata application.
                                When i go to start the service on the WAN interface, it runs for 3 seconds then stops.
                                Log View message is below
                                5/4/2020 -- 18:34:18 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
                                5/4/2020 -- 18:34:18 - <Info> -- CPUs/cores online: 2
                                5/4/2020 -- 18:34:18 - <Info> -- HTTP memcap: 67108864
                                5/4/2020 -- 18:34:18 - <Notice> -- using flow hash instead of active packets

                                bmeeksB 1 Reply Last reply Reply Quote 0
                                • bmeeksB
                                  bmeeks @blaytrail
                                  last edited by bmeeks

                                  @blaytrail said in Suricata core dumping after 2.4.5 upgrade:

                                  I just posted the same issue after the 2.4.5 Release. I've been working on this for 2 days now. I see I'm not the only one having this issue. See my post below.

                                  Suricata will not start using pfsense. I currently have the SG-1100 appliance.

                                  Added the Suricata package with no issues.
                                  Configured Global Setting using the ETOpen Emerging Threats rules only. No Issues
                                  Performed the Emerging Threats Open Rules update with no issues.
                                  Enabled emerging rules. No Issues
                                  The SG-1100 appliance only has 2 CPU's with 1GB of Ram. It should still run the Suricata application.
                                  When i go to start the service on the WAN interface, it runs for 3 seconds then stops.
                                  Log View message is below
                                  5/4/2020 -- 18:34:18 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
                                  5/4/2020 -- 18:34:18 - <Info> -- CPUs/cores online: 2
                                  5/4/2020 -- 18:34:18 - <Info> -- HTTP memcap: 67108864
                                  5/4/2020 -- 18:34:18 - <Notice> -- using flow hash instead of active packets

                                  Look in the pfSense system log and I bet you will see the Signal 4 core dump message disscussed in the posts above. Suricata 5.x on SG-1100 hardware is a no-go for now. You will need to either switch over to Snort or just wait for the fix.

                                  There is absolutely no setting or configuration change a user can make that will change this problem. It is an issue with the compiled binary itself.

                                  1 Reply Last reply Reply Quote 0
                                  • B
                                    blaytrail
                                    last edited by

                                    Thanks for the update. You are correct. I just looked in the system logs and found the entry below.

                                    Apr 5 20:09:32 kernel pid 38129 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)

                                    I may use AlienVault with the built in Suricata software and create a span port off the netgate device.

                                    Can you create a span port on the Netgate device using the OPT port?

                                    bmeeksB 1 Reply Last reply Reply Quote 0
                                    • bmeeksB
                                      bmeeks @blaytrail
                                      last edited by bmeeks

                                      @blaytrail said in Suricata core dumping after 2.4.5 upgrade:

                                      Thanks for the update. You are correct. I just looked in the system logs and found the entry below.

                                      Apr 5 20:09:32 kernel pid 38129 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)

                                      I may use AlienVault with the built in Suricata software and create a span port off the netgate device.

                                      Can you create a span port on the Netgate device using the OPT port?

                                      Not to my knowledge, but I'm no expert on the Marvel switch chip inside the SG-1100. You might want to ask that question in the Netgate Hardware forum on here.

                                      Signal 4 means ILLEGAL INSTRUCTION.

                                      1 Reply Last reply Reply Quote 0
                                      • B
                                        blaytrail
                                        last edited by

                                        Thanks. I'm looking forward to learning more about IDS\IPS. Surricata seems pretty simple to setup until i ran into the issue. If you don't mind me asking are you doing to switch to another IDS\IPS solution?

                                        bmeeksB 1 Reply Last reply Reply Quote 0
                                        • bmeeksB
                                          bmeeks @blaytrail
                                          last edited by bmeeks

                                          @blaytrail said in Suricata core dumping after 2.4.5 upgrade:

                                          Thanks. I'm looking forward to learning more about IDS\IPS. Surricata seems pretty simple to setup until i ran into the issue. If you don't mind me asking are you doing to switch to another IDS\IPS solution?

                                          I am the developer/maintainer for Suricata and Snort on pfSense (volunteer, not paid or affiliated with Netgate). I started with Snort years ago and have never changed. Not because it is better or Suricata is worse, just lazy and didn't want to change out my production firewall.

                                          So for now I'm running Snort on a Netgate SG-5100 appliance. Prior to that I ran it on custom-built hardware (mostly Supermicro 1U servers).

                                          1 Reply Last reply Reply Quote 0
                                          • B
                                            blaytrail
                                            last edited by

                                            Cool! It must be fun to be the developer/maintainer for the applications. I really enjoy working with pfSense. I just setup the VPN and it works perfectly. I guess i can move up to the SG-5100. I want to set getting more familiar with IDS/IPS. I take it I will need to use something else beside the SG-1100 for IDS/IPS.

                                            bmeeksB 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.