Suricata core dumping after 2.4.5 upgrade
-
Hey everyone. Long time listener, first time caller.
After the 2.4.5 update suricata won't start and bombs with the following error:
pid 91657 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)
I have removed the interfaces - no go
I have re-installed the packages - no go
I have removed, rebooted, re-installed - no go.Anyone have any ideas or fixes?
Running on Netgate SG-1100 appliance.
-
I have an SG-1100 for testing and will be looking into this issue. The SG-1100 uses a non-Intel CPU and that can bring on a host of potential problems when running C code that was originally developed and widely used on Intel platforms for a long time.
Snort, Suricata and a few other FreeBSD ports packages have or still have issues with running on non-Intel CPUs (meaning arm and aarch64 processors). Previously identified issues in Snort and Suricata were caused by the way upstream coders had taken some "liberty" with memory pointer casting. I don't know if this new issue with the Suricata binary on aarch64 hardware is more of the same or something totally different.
-
@Mobius114 said in Suricata core dumping after 2.4.5 upgrade:
Hey everyone. Long time listener, first time caller.
After the 2.4.5 update suricata won't start and bombs with the following error:
pid 91657 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)
I have removed the interfaces - no go
I have re-installed the packages - no go
I have removed, rebooted, re-installed - no go.Anyone have any ideas or fixes?
Running on Netgate SG-1100 appliance.
There is nothing you can do on your end to impact this. It is a problem with the actual compiled code for the package and is due to the SG-1100 appliance having an ARM processor instead of the Intel x86-x64 platform. Signal 4 is the FreeBSD signal for ILLEGAL INSTRUCTION. It indicates an issue with the actual machine code of the Suricata binary executable. It is being looked into, but the fix may not happen quickly.
-
@bmeeks Appreciate the update and insights into the issue. I look forward to the fix when it's available.
It had been working quite well until the upgrade last night and even ran for about 20 minutes before it crapped out completely.
Cheers
-
@Mobius114 said in Suricata core dumping after 2.4.5 upgrade:
@bmeeks Appreciate the update and insights into the issue. I look forward to the fix when it's available.
It had been working quite well until the upgrade last night and even ran for about 20 minutes before it crapped out completely.
Cheers
Oh, so it starts and runs for a while and then crashes? My initial info led me to believe it would not even start up.
Unfortunately, running for a while and then randomly crashing makes finding the issue way harder. Much easier to find something that fails immediately and the same way every time.
My SG-1100 is new in the box and I have not actually set it up for testing yet. My main firewall is an SG-5100. Those are Intel-based platforms.
-
@bmeeks ah I wasn't clear on that.
It ran initially after the upgrade and I got a few alerts before it stopped.
Since that time it has never started since. The core dump error is almost instantaneous.
-
@Mobius114 said in Suricata core dumping after 2.4.5 upgrade:
@bmeeks ah I wasn't clear on that.
It ran initially after the upgrade and I got a few alerts before it stopped.
Since that time it has never started since. The core dump error is almost instantaneous.
OK. That sounds more like what I had previously heard. Well, as I said, the good news in that is that something immediate and repeatable will hopefully be easier to identify.
-
Same boat for me. Suricata will run for a few seconds (enough to show a green status on the dashboard / interface list) then it dumps.
The suricata.log is blank. Here's from the last run:
1/4/2020 -- 20:21:11 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode 1/4/2020 -- 20:21:11 - <Info> -- CPUs/cores online: 2 1/4/2020 -- 20:21:11 - <Info> -- HTTP memcap: 67108864 1/4/2020 -- 20:21:11 - <Notice> -- using flow hash instead of active packetsIf you restart you get the "[ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata_mvneta0.409033796.pid' exists but appears stale." issue. If you rm the file via the command prompt it will restart momentarily then dump again.
Additionally, I now have a crash notification on the dashboard. "pfSense has detected a crash report or programming bug. Click here for more information." Detail page shows the following:
Crash report begins. Anonymous machine information: arm64 11.3-STABLE FreeBSD 11.3-STABLE #233 2c992b2181a(factory-RELENG_2_4_5): Tue Mar 24 15:27:54 EDT 2020 root@buildbot3-nyi.netgate.com:/build/factory-crossbuild-245-aarch64/obj/aarch64/Og10eFss/build/factory-crossbuild-245-aarch64/sources/FreeBSD-src/sys/pfSense Crash report details: PHP Errors: [01-Apr-2020 19:36:48 America/Los_Angeles] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 137897032 bytes) in /usr/local/www/csrf/csrf-magic.php on line 149 No FreeBSD crash data found.I haven't changed any memory allocation with pfsense or Suricata and never have seen this error pop. The dashboard still lists memory usage in the mid-20%.
Only other thing I can find in terms of logs is the Status -> System Log -> System -> General and it reads as follows:
Apr 1 20:10:30 kernel pid 1445 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)I run an extremely basic setup with suricata as really the only service running other than some basic LAN isolation firewall settings.
Let me know if there is any other information I might be able to add.
-
I should add that Suricata and the SG-1100 ran problem free for six months after some initial, minor adjustments with the flow/stream settings and the usual tweaking of the rules.
-
@h3llo said in Suricata core dumping after 2.4.5 upgrade:
I should add that Suricata and the SG-1100 ran problem free for six months after some initial, minor adjustments with the flow/stream settings and the usual tweaking of the rules.
Did you not read my posts up above where I described this problem and the fact it is a problem within the binary machine code itself? Here are the relevant lines from one of those posts:
There is nothing you can do on your end to impact this. It is a problem with the actual compiled code for the package and is due to the SG-1100 appliance having an ARM processor instead of the Intel x86-x64 platform. Signal 4 is the FreeBSD signal for ILLEGAL INSTRUCTION. It indicates an issue with the actual machine code of the Suricata binary executable. It is being looked into, but the fix may not happen quickly.
The SG-1100 has a non-Intel architecture CPU. It is an aarch64 instead of an x86-x64. Fixing this one may not happen quickly at all, so if you want an IDS/IPS package on an SG-1100 you should consider switching over to Snort -- at least for the foreseeable future. I know what the problem is, but not where in the megabytes of C and Rust source code it is located. And I'm talking about the Suricata binary source code which comes from upstream - code I did not write.
-
Yes sir I did read the posts above. I just added some additional details regarding the specific errors thrown in the off chance that it might be helpful. If it is not, ignore away.
Appreciate the feedback particularly the expectation management in terms of a fix. While I'm bummed that what once worked does not now, I will take the opportunity to learn a bit about Snort and give it a go.
-
@h3llo said in Suricata core dumping after 2.4.5 upgrade:
Yes sir I did read the posts above. I just added some additional details regarding the specific errors thrown in the off chance that it might be helpful. If it is not, ignore away.
Appreciate the feedback particularly the expectation management in terms of a fix. While I'm bummed that what once worked does not now, I will take the opportunity to learn a bit about Snort and give it a go.
Blame this problem on two things. One is the Suricata upstream team deciding to use Rust in the binary and to make it mandatory starting with version 5.x. The other is the folks who created the llvm compiler used to create armv6/v7 and aarch64 binary from C source code. That compiler makes some binary instruction choices that are "how shall I say?" -- not optimal ...
.
-
I just bought a SG-1100 and have this same issue. Sad to find out that it may not be fixed soon. I'll look into Snort in the meantime.
-
I just posted the same issue after the 2.4.5 Release. I've been working on this for 2 days now. I see I'm not the only one having this issue. See my post below.
Suricata will not start using pfsense. I currently have the SG-1100 appliance.
Added the Suricata package with no issues.
Configured Global Setting using the ETOpen Emerging Threats rules only. No Issues
Performed the Emerging Threats Open Rules update with no issues.
Enabled emerging rules. No Issues
The SG-1100 appliance only has 2 CPU's with 1GB of Ram. It should still run the Suricata application.
When i go to start the service on the WAN interface, it runs for 3 seconds then stops.
Log View message is below
5/4/2020 -- 18:34:18 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
5/4/2020 -- 18:34:18 - <Info> -- CPUs/cores online: 2
5/4/2020 -- 18:34:18 - <Info> -- HTTP memcap: 67108864
5/4/2020 -- 18:34:18 - <Notice> -- using flow hash instead of active packets
-
@blaytrail said in Suricata core dumping after 2.4.5 upgrade:
I just posted the same issue after the 2.4.5 Release. I've been working on this for 2 days now. I see I'm not the only one having this issue. See my post below.
Suricata will not start using pfsense. I currently have the SG-1100 appliance.
Added the Suricata package with no issues.
Configured Global Setting using the ETOpen Emerging Threats rules only. No Issues
Performed the Emerging Threats Open Rules update with no issues.
Enabled emerging rules. No Issues
The SG-1100 appliance only has 2 CPU's with 1GB of Ram. It should still run the Suricata application.
When i go to start the service on the WAN interface, it runs for 3 seconds then stops.
Log View message is below
5/4/2020 -- 18:34:18 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
5/4/2020 -- 18:34:18 - <Info> -- CPUs/cores online: 2
5/4/2020 -- 18:34:18 - <Info> -- HTTP memcap: 67108864
5/4/2020 -- 18:34:18 - <Notice> -- using flow hash instead of active packetsLook in the pfSense system log and I bet you will see the Signal 4 core dump message disscussed in the posts above. Suricata 5.x on SG-1100 hardware is a no-go for now. You will need to either switch over to Snort or just wait for the fix.
There is absolutely no setting or configuration change a user can make that will change this problem. It is an issue with the compiled binary itself.
-
Thanks for the update. You are correct. I just looked in the system logs and found the entry below.
Apr 5 20:09:32 kernel pid 38129 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)
I may use AlienVault with the built in Suricata software and create a span port off the netgate device.
Can you create a span port on the Netgate device using the OPT port?
-
@blaytrail said in Suricata core dumping after 2.4.5 upgrade:
Thanks for the update. You are correct. I just looked in the system logs and found the entry below.
Apr 5 20:09:32 kernel pid 38129 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)
I may use AlienVault with the built in Suricata software and create a span port off the netgate device.
Can you create a span port on the Netgate device using the OPT port?
Not to my knowledge, but I'm no expert on the Marvel switch chip inside the SG-1100. You might want to ask that question in the Netgate Hardware forum on here.
Signal 4 means ILLEGAL INSTRUCTION.
-
Thanks. I'm looking forward to learning more about IDS\IPS. Surricata seems pretty simple to setup until i ran into the issue. If you don't mind me asking are you doing to switch to another IDS\IPS solution?
-
@blaytrail said in Suricata core dumping after 2.4.5 upgrade:
Thanks. I'm looking forward to learning more about IDS\IPS. Surricata seems pretty simple to setup until i ran into the issue. If you don't mind me asking are you doing to switch to another IDS\IPS solution?
I am the developer/maintainer for Suricata and Snort on pfSense (volunteer, not paid or affiliated with Netgate). I started with Snort years ago and have never changed. Not because it is better or Suricata is worse, just lazy and didn't want to change out my production firewall.
So for now I'm running Snort on a Netgate SG-5100 appliance. Prior to that I ran it on custom-built hardware (mostly Supermicro 1U servers).
-
Cool! It must be fun to be the developer/maintainer for the applications. I really enjoy working with pfSense. I just setup the VPN and it works perfectly. I guess i can move up to the SG-5100. I want to set getting more familiar with IDS/IPS. I take it I will need to use something else beside the SG-1100 for IDS/IPS.
-
@blaytrail said in Suricata core dumping after 2.4.5 upgrade:
Cool! It must be fun to be the developer/maintainer for the applications. I really enjoy working with pfSense. I just setup the VPN and it works perfectly. I guess i can move up to the SG-5100. I want to set getting more familiar with IDS/IPS. I take it I will need to use something else beside the SG-1100 for IDS/IPS.
The SG-1100 is a good starter box for IDS/IPS, but to be honest the amount of RAM it has can limit it when it comes to a full-blown IDS/IPS setup. You would need to be a bit choosy about which rules, and how many in total, you enabled to control RAM usage.
The SG-5100 is more capable both in terms of CPU and RAM. Of course it costs quite a bit more. But I looked around and discovered that getting a chassis, motherboard, CPU, RAM and the other required bits totaled up to be at least as much as the SG-5100 (or so close it was really a wash).
-
I will order the SG-5100 appliance when i get paid next week. I don't mind spending the money for educational purposes. Thanks so much for your time this morning.
I have the instructions for setting up Suricata. Is there a good place to get basic instructions for setting up Snort?
-
@blaytrail said in Suricata core dumping after 2.4.5 upgrade:
I will order the SG-5100 appliance when i get paid next week. I don't mind spending the money for educational purposes. Thanks so much for your time this morning.
I have the instructions for setting up Suricata. Is there a good place to get basic instructions for setting up Snort?
There is a link in the offical Netgate documentation. Both packages work essentially the exact same way in terms of set up on pfSense. In fact, a large percentage of the PHP source is copy/paste with "Snort" changed to "Suricata". The Snort package existed first.
You should be able to run Snort currently on the SG-1100. I don't think it has the same compiler issue (but I'm not 100% positive).
Here is the link: https://docs.netgate.com/pfsense/en/latest/ids-ips/setup-snort-package.html.
-
Excellent!
I just realize there is a separate snort package install on the SG-1100. I will get this setup this morning.
-
I am glad to know this is an existing bug. My SG-1000 upgrade to 2.4.5 failed and I had to restore the factory image from USB key. I just assumed my backup configuration was somehow corrupt and causing an issue.
Even after installing the factory image and Suricata package with minimal configuration Suricata would die after about 10 seconds. Only when I ran Suricata from the command line did I see it print "Illegal Instruction" to stderr when it crashed - it was not captured to the log file.
I installed Snort and it works just fine, but I miss Suricata and hope the upstream issue is corrected soon.
-
@sholekamp said in Suricata core dumping after 2.4.5 upgrade:
I am glad to know this is an existing bug. My SG-1000 upgrade to 2.4.5 failed and I had to restore the factory image from USB key. I just assumed my backup configuration was somehow corrupt and causing an issue.
Even after installing the factory image and Suricata package with minimal configuration Suricata would die after about 10 seconds. Only when I ran Suricata from the command line did I see it print "Illegal Instruction" to stderr when it crashed - it was not captured to the log file.
I installed Snort and it works just fine, but I miss Suricata and hope the upstream issue is corrected soon.
The Signal 4 code (which is the ILLEGAL INSTRUCTION error) is printed to the system log of pfSense by the operating system. The reason it is not printed to the Suricata log is that Suricata is the process executing the illegal instruction. As a result, the operating system immediately terminates the process and there is no "suricata" process to write anything to its log.
-
@bmeeks
That all makes sense. I missed the entries in the system log:
Apr 9 15:31:09 pfSense kernel: pid 51904 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)Seeing "Illegal Instruction" printed to the terminal is what led me to this thread. In the end my SG-100 is back up and running and the problem has been identified. I will keep an eye out for a Suricata update. Thank you for the support.
