Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    Suricata core dumping after 2.4.5 upgrade

    IDS/IPS
    6
    27
    390
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      Mobius114 last edited by Mobius114

      Hey everyone. Long time listener, first time caller.

      After the 2.4.5 update suricata won't start and bombs with the following error:

      pid 91657 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)

      I have removed the interfaces - no go
      I have re-installed the packages - no go
      I have removed, rebooted, re-installed - no go.

      Anyone have any ideas or fixes?

      Running on Netgate SG-1100 appliance.

      bmeeks 1 Reply Last reply Reply Quote 0
      • bmeeks
        bmeeks last edited by bmeeks

        I have an SG-1100 for testing and will be looking into this issue. The SG-1100 uses a non-Intel CPU and that can bring on a host of potential problems when running C code that was originally developed and widely used on Intel platforms for a long time.

        Snort, Suricata and a few other FreeBSD ports packages have or still have issues with running on non-Intel CPUs (meaning arm and aarch64 processors). Previously identified issues in Snort and Suricata were caused by the way upstream coders had taken some "liberty" with memory pointer casting. I don't know if this new issue with the Suricata binary on aarch64 hardware is more of the same or something totally different.

        1 Reply Last reply Reply Quote 0
        • bmeeks
          bmeeks @Mobius114 last edited by bmeeks

          @Mobius114 said in Suricata core dumping after 2.4.5 upgrade:

          Hey everyone. Long time listener, first time caller.

          After the 2.4.5 update suricata won't start and bombs with the following error:

          pid 91657 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)

          I have removed the interfaces - no go
          I have re-installed the packages - no go
          I have removed, rebooted, re-installed - no go.

          Anyone have any ideas or fixes?

          Running on Netgate SG-1100 appliance.

          There is nothing you can do on your end to impact this. It is a problem with the actual compiled code for the package and is due to the SG-1100 appliance having an ARM processor instead of the Intel x86-x64 platform. Signal 4 is the FreeBSD signal for ILLEGAL INSTRUCTION. It indicates an issue with the actual machine code of the Suricata binary executable. It is being looked into, but the fix may not happen quickly.

          M 1 Reply Last reply Reply Quote 2
          • M
            Mobius114 @bmeeks last edited by

            @bmeeks Appreciate the update and insights into the issue. I look forward to the fix when it's available.

            It had been working quite well until the upgrade last night and even ran for about 20 minutes before it crapped out completely.

            Cheers

            bmeeks 1 Reply Last reply Reply Quote 0
            • bmeeks
              bmeeks @Mobius114 last edited by bmeeks

              @Mobius114 said in Suricata core dumping after 2.4.5 upgrade:

              @bmeeks Appreciate the update and insights into the issue. I look forward to the fix when it's available.

              It had been working quite well until the upgrade last night and even ran for about 20 minutes before it crapped out completely.

              Cheers

              Oh, so it starts and runs for a while and then crashes? My initial info led me to believe it would not even start up.

              Unfortunately, running for a while and then randomly crashing makes finding the issue way harder. Much easier to find something that fails immediately and the same way every time.

              My SG-1100 is new in the box and I have not actually set it up for testing yet. My main firewall is an SG-5100. Those are Intel-based platforms.

              M 1 Reply Last reply Reply Quote 0
              • M
                Mobius114 @bmeeks last edited by

                @bmeeks ah I wasn't clear on that.

                It ran initially after the upgrade and I got a few alerts before it stopped.

                Since that time it has never started since. The core dump error is almost instantaneous.

                bmeeks 1 Reply Last reply Reply Quote 0
                • bmeeks
                  bmeeks @Mobius114 last edited by

                  @Mobius114 said in Suricata core dumping after 2.4.5 upgrade:

                  @bmeeks ah I wasn't clear on that.

                  It ran initially after the upgrade and I got a few alerts before it stopped.

                  Since that time it has never started since. The core dump error is almost instantaneous.

                  OK. That sounds more like what I had previously heard. Well, as I said, the good news in that is that something immediate and repeatable will hopefully be easier to identify.

                  1 Reply Last reply Reply Quote 1
                  • H
                    h3llo last edited by

                    Same boat for me. Suricata will run for a few seconds (enough to show a green status on the dashboard / interface list) then it dumps.

                    The suricata.log is blank. Here's from the last run:

                    1/4/2020 -- 20:21:11 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
1/4/2020 -- 20:21:11 - <Info> -- CPUs/cores online: 2
1/4/2020 -- 20:21:11 - <Info> -- HTTP memcap: 67108864
1/4/2020 -- 20:21:11 - <Notice> -- using flow hash instead of active packets

                    If you restart you get the "[ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata_mvneta0.409033796.pid' exists but appears stale." issue. If you rm the file via the command prompt it will restart momentarily then dump again.

                    Additionally, I now have a crash notification on the dashboard. "pfSense has detected a crash report or programming bug. Click here for more information." Detail page shows the following:

                    Crash report begins.  Anonymous machine information:

arm64
11.3-STABLE
FreeBSD 11.3-STABLE #233 2c992b2181a(factory-RELENG_2_4_5): Tue Mar 24 15:27:54 EDT 2020     root@buildbot3-nyi.netgate.com:/build/factory-crossbuild-245-aarch64/obj/aarch64/Og10eFss/build/factory-crossbuild-245-aarch64/sources/FreeBSD-src/sys/pfSense

Crash report details:

PHP Errors:
[01-Apr-2020 19:36:48 America/Los_Angeles] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 137897032 bytes) in /usr/local/www/csrf/csrf-magic.php on line 149

No FreeBSD crash data found.

                    I haven't changed any memory allocation with pfsense or Suricata and never have seen this error pop. The dashboard still lists memory usage in the mid-20%.

                    Only other thing I can find in terms of logs is the Status -> System Log -> System -> General and it reads as follows:

                    Apr 1 20:10:30 	kernel 		pid 1445 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)

                    I run an extremely basic setup with suricata as really the only service running other than some basic LAN isolation firewall settings.

                    Let me know if there is any other information I might be able to add.

                    1 Reply Last reply Reply Quote 0
                    • H
                      h3llo last edited by

                      I should add that Suricata and the SG-1100 ran problem free for six months after some initial, minor adjustments with the flow/stream settings and the usual tweaking of the rules.

                      bmeeks 1 Reply Last reply Reply Quote 1
                      • bmeeks
                        bmeeks @h3llo last edited by bmeeks

                        @h3llo said in Suricata core dumping after 2.4.5 upgrade:

                        I should add that Suricata and the SG-1100 ran problem free for six months after some initial, minor adjustments with the flow/stream settings and the usual tweaking of the rules.

                        Did you not read my posts up above where I described this problem and the fact it is a problem within the binary machine code itself? Here are the relevant lines from one of those posts:

                        There is nothing you can do on your end to impact this. It is a problem with the actual compiled code for the package and is due to the SG-1100 appliance having an ARM processor instead of the Intel x86-x64 platform. Signal 4 is the FreeBSD signal for ILLEGAL INSTRUCTION. It indicates an issue with the actual machine code of the Suricata binary executable. It is being looked into, but the fix may not happen quickly.

                        The SG-1100 has a non-Intel architecture CPU. It is an aarch64 instead of an x86-x64. Fixing this one may not happen quickly at all, so if you want an IDS/IPS package on an SG-1100 you should consider switching over to Snort -- at least for the foreseeable future. I know what the problem is, but not where in the megabytes of C and Rust source code it is located. And I'm talking about the Suricata binary source code which comes from upstream - code I did not write.

                        1 Reply Last reply Reply Quote 0
                        • H
                          h3llo last edited by

                          Yes sir I did read the posts above. I just added some additional details regarding the specific errors thrown in the off chance that it might be helpful. If it is not, ignore away.

                          Appreciate the feedback particularly the expectation management in terms of a fix. While I'm bummed that what once worked does not now, I will take the opportunity to learn a bit about Snort and give it a go.

                          bmeeks 1 Reply Last reply Reply Quote 0
                          • bmeeks
                            bmeeks @h3llo last edited by

                            @h3llo said in Suricata core dumping after 2.4.5 upgrade:

                            Yes sir I did read the posts above. I just added some additional details regarding the specific errors thrown in the off chance that it might be helpful. If it is not, ignore away.

                            Appreciate the feedback particularly the expectation management in terms of a fix. While I'm bummed that what once worked does not now, I will take the opportunity to learn a bit about Snort and give it a go.

                            Blame this problem on two things. One is the Suricata upstream team deciding to use Rust in the binary and to make it mandatory starting with version 5.x. The other is the folks who created the llvm compiler used to create armv6/v7 and aarch64 binary from C source code. That compiler makes some binary instruction choices that are "how shall I say?" -- not optimal ... ☺ .

                            1 Reply Last reply Reply Quote 0
                            • B
                              bgalakazam last edited by

                              I just bought a SG-1100 and have this same issue. Sad to find out that it may not be fixed soon. I'll look into Snort in the meantime.

                              1 Reply Last reply Reply Quote 0
                              • B
                                blaytrail last edited by

                                I just posted the same issue after the 2.4.5 Release. I've been working on this for 2 days now. I see I'm not the only one having this issue. See my post below.

                                Suricata will not start using pfsense. I currently have the SG-1100 appliance.

                                Added the Suricata package with no issues.
                                Configured Global Setting using the ETOpen Emerging Threats rules only. No Issues
                                Performed the Emerging Threats Open Rules update with no issues.
                                Enabled emerging rules. No Issues
                                The SG-1100 appliance only has 2 CPU's with 1GB of Ram. It should still run the Suricata application.
                                When i go to start the service on the WAN interface, it runs for 3 seconds then stops.
                                Log View message is below
                                5/4/2020 -- 18:34:18 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
                                5/4/2020 -- 18:34:18 - <Info> -- CPUs/cores online: 2
                                5/4/2020 -- 18:34:18 - <Info> -- HTTP memcap: 67108864
                                5/4/2020 -- 18:34:18 - <Notice> -- using flow hash instead of active packets

                                bmeeks 1 Reply Last reply Reply Quote 0
                                • bmeeks
                                  bmeeks @blaytrail last edited by bmeeks

                                  @blaytrail said in Suricata core dumping after 2.4.5 upgrade:

                                  I just posted the same issue after the 2.4.5 Release. I've been working on this for 2 days now. I see I'm not the only one having this issue. See my post below.

                                  Suricata will not start using pfsense. I currently have the SG-1100 appliance.

                                  Added the Suricata package with no issues.
                                  Configured Global Setting using the ETOpen Emerging Threats rules only. No Issues
                                  Performed the Emerging Threats Open Rules update with no issues.
                                  Enabled emerging rules. No Issues
                                  The SG-1100 appliance only has 2 CPU's with 1GB of Ram. It should still run the Suricata application.
                                  When i go to start the service on the WAN interface, it runs for 3 seconds then stops.
                                  Log View message is below
                                  5/4/2020 -- 18:34:18 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
                                  5/4/2020 -- 18:34:18 - <Info> -- CPUs/cores online: 2
                                  5/4/2020 -- 18:34:18 - <Info> -- HTTP memcap: 67108864
                                  5/4/2020 -- 18:34:18 - <Notice> -- using flow hash instead of active packets

                                  Look in the pfSense system log and I bet you will see the Signal 4 core dump message disscussed in the posts above. Suricata 5.x on SG-1100 hardware is a no-go for now. You will need to either switch over to Snort or just wait for the fix.

                                  There is absolutely no setting or configuration change a user can make that will change this problem. It is an issue with the compiled binary itself.

                                  1 Reply Last reply Reply Quote 0
                                  • B
                                    blaytrail last edited by

                                    Thanks for the update. You are correct. I just looked in the system logs and found the entry below.

                                    Apr 5 20:09:32 kernel pid 38129 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)

                                    I may use AlienVault with the built in Suricata software and create a span port off the netgate device.

                                    Can you create a span port on the Netgate device using the OPT port?

                                    bmeeks 1 Reply Last reply Reply Quote 0
                                    • bmeeks
                                      bmeeks @blaytrail last edited by bmeeks

                                      @blaytrail said in Suricata core dumping after 2.4.5 upgrade:

                                      Thanks for the update. You are correct. I just looked in the system logs and found the entry below.

                                      Apr 5 20:09:32 kernel pid 38129 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)

                                      I may use AlienVault with the built in Suricata software and create a span port off the netgate device.

                                      Can you create a span port on the Netgate device using the OPT port?

                                      Not to my knowledge, but I'm no expert on the Marvel switch chip inside the SG-1100. You might want to ask that question in the Netgate Hardware forum on here.

                                      Signal 4 means ILLEGAL INSTRUCTION.

                                      1 Reply Last reply Reply Quote 0
                                      • B
                                        blaytrail last edited by

                                        Thanks. I'm looking forward to learning more about IDS\IPS. Surricata seems pretty simple to setup until i ran into the issue. If you don't mind me asking are you doing to switch to another IDS\IPS solution?

                                        bmeeks 1 Reply Last reply Reply Quote 0
                                        • bmeeks
                                          bmeeks @blaytrail last edited by bmeeks

                                          @blaytrail said in Suricata core dumping after 2.4.5 upgrade:

                                          Thanks. I'm looking forward to learning more about IDS\IPS. Surricata seems pretty simple to setup until i ran into the issue. If you don't mind me asking are you doing to switch to another IDS\IPS solution?

                                          I am the developer/maintainer for Suricata and Snort on pfSense (volunteer, not paid or affiliated with Netgate). I started with Snort years ago and have never changed. Not because it is better or Suricata is worse, just lazy and didn't want to change out my production firewall.

                                          So for now I'm running Snort on a Netgate SG-5100 appliance. Prior to that I ran it on custom-built hardware (mostly Supermicro 1U servers).

                                          1 Reply Last reply Reply Quote 0
                                          • B
                                            blaytrail last edited by

                                            Cool! It must be fun to be the developer/maintainer for the applications. I really enjoy working with pfSense. I just setup the VPN and it works perfectly. I guess i can move up to the SG-5100. I want to set getting more familiar with IDS/IPS. I take it I will need to use something else beside the SG-1100 for IDS/IPS.

                                            bmeeks 1 Reply Last reply Reply Quote 0
                                            • bmeeks
                                              bmeeks @blaytrail last edited by

                                              @blaytrail said in Suricata core dumping after 2.4.5 upgrade:

                                              Cool! It must be fun to be the developer/maintainer for the applications. I really enjoy working with pfSense. I just setup the VPN and it works perfectly. I guess i can move up to the SG-5100. I want to set getting more familiar with IDS/IPS. I take it I will need to use something else beside the SG-1100 for IDS/IPS.

                                              The SG-1100 is a good starter box for IDS/IPS, but to be honest the amount of RAM it has can limit it when it comes to a full-blown IDS/IPS setup. You would need to be a bit choosy about which rules, and how many in total, you enabled to control RAM usage.

                                              The SG-5100 is more capable both in terms of CPU and RAM. Of course it costs quite a bit more. But I looked around and discovered that getting a chassis, motherboard, CPU, RAM and the other required bits totaled up to be at least as much as the SG-5100 (or so close it was really a wash).

                                              1 Reply Last reply Reply Quote 0
                                              • B
                                                blaytrail last edited by

                                                I will order the SG-5100 appliance when i get paid next week. I don't mind spending the money for educational purposes. Thanks so much for your time this morning.

                                                I have the instructions for setting up Suricata. Is there a good place to get basic instructions for setting up Snort?

                                                bmeeks 1 Reply Last reply Reply Quote 0
                                                • bmeeks
                                                  bmeeks @blaytrail last edited by bmeeks

                                                  @blaytrail said in Suricata core dumping after 2.4.5 upgrade:

                                                  I will order the SG-5100 appliance when i get paid next week. I don't mind spending the money for educational purposes. Thanks so much for your time this morning.

                                                  I have the instructions for setting up Suricata. Is there a good place to get basic instructions for setting up Snort?

                                                  There is a link in the offical Netgate documentation. Both packages work essentially the exact same way in terms of set up on pfSense. In fact, a large percentage of the PHP source is copy/paste with "Snort" changed to "Suricata". The Snort package existed first.

                                                  You should be able to run Snort currently on the SG-1100. I don't think it has the same compiler issue (but I'm not 100% positive).

                                                  Here is the link: https://docs.netgate.com/pfsense/en/latest/ids-ips/setup-snort-package.html.

                                                  1 Reply Last reply Reply Quote 0
                                                  • B
                                                    blaytrail last edited by

                                                    Excellent!

                                                    I just realize there is a separate snort package install on the SG-1100. I will get this setup this morning.

                                                    1 Reply Last reply Reply Quote 0
                                                    • S
                                                      sholekamp last edited by

                                                      I am glad to know this is an existing bug. My SG-1000 upgrade to 2.4.5 failed and I had to restore the factory image from USB key. I just assumed my backup configuration was somehow corrupt and causing an issue.

                                                      Even after installing the factory image and Suricata package with minimal configuration Suricata would die after about 10 seconds. Only when I ran Suricata from the command line did I see it print "Illegal Instruction" to stderr when it crashed - it was not captured to the log file.

                                                      I installed Snort and it works just fine, but I miss Suricata and hope the upstream issue is corrected soon.

                                                      bmeeks 1 Reply Last reply Reply Quote 0
                                                      • bmeeks
                                                        bmeeks @sholekamp last edited by bmeeks

                                                        @sholekamp said in Suricata core dumping after 2.4.5 upgrade:

                                                        I am glad to know this is an existing bug. My SG-1000 upgrade to 2.4.5 failed and I had to restore the factory image from USB key. I just assumed my backup configuration was somehow corrupt and causing an issue.

                                                        Even after installing the factory image and Suricata package with minimal configuration Suricata would die after about 10 seconds. Only when I ran Suricata from the command line did I see it print "Illegal Instruction" to stderr when it crashed - it was not captured to the log file.

                                                        I installed Snort and it works just fine, but I miss Suricata and hope the upstream issue is corrected soon.

                                                        The Signal 4 code (which is the ILLEGAL INSTRUCTION error) is printed to the system log of pfSense by the operating system. The reason it is not printed to the Suricata log is that Suricata is the process executing the illegal instruction. As a result, the operating system immediately terminates the process and there is no "suricata" process to write anything to its log.

                                                        S 1 Reply Last reply Reply Quote 0
                                                        • S
                                                          sholekamp @bmeeks last edited by sholekamp

                                                          @bmeeks
                                                          That all makes sense. I missed the entries in the system log:
                                                          Apr 9 15:31:09 pfSense kernel: pid 51904 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)

                                                          Seeing "Illegal Instruction" printed to the terminal is what led me to this thread. In the end my SG-100 is back up and running and the problem has been identified. I will keep an eye out for a Suricata update. Thank you for the support.

                                                          *** UPDATE ***
                                                          For anyone that stumbles upon this thread, the issue has been corrected with pfSense 2.4.5-p1 released June 9, 2020. See the release notes: https://docs.netgate.com/pfsense/en/latest/releases/2-4-5-p1-new-features-and-changes.html

                                                          Great work!

                                                          1 Reply Last reply Reply Quote 0
                                                          • First post
                                                            Last post

                                                          Products

                                                          • Platform Overview
                                                          • TNSR
                                                          • pfSense
                                                          • Appliances

                                                          Services

                                                          • Training
                                                          • Professional Services

                                                          Support

                                                          • Subscription Plans
                                                          • Contact Support
                                                          • Product Lifecycle
                                                          • Documentation

                                                          News

                                                          • Media Coverage
                                                          • Press
                                                          • Events

                                                          Resources

                                                          • Blog
                                                          • FAQ
                                                          • Find a Partner
                                                          • Resource Library
                                                          • Security Information

                                                          Company

                                                          • About Us
                                                          • Careers
                                                          • Partners
                                                          • Contact Us
                                                          • Legal
                                                          Our Mission

                                                          We provide leading-edge network security at a fair price - regardless of organizational size or network sophistication. We believe that an open-source security model offers disruptive pricing along with the agility required to quickly address emerging threats.

                                                          Subscribe to our Newsletter

                                                          Product information, software announcements, and special offers. See our newsletter archive to sign up for future newsletters and to read past announcements.

                                                          © 2021 Rubicon Communications, LLC | Privacy Policy