Suricata core dumping after 2.4.5 upgrade

Mobius114

Hey everyone. Long time listener, first time caller.

After the 2.4.5 update suricata won't start and bombs with the following error:

pid 91657 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)

I have removed the interfaces - no go
I have re-installed the packages - no go
I have removed, rebooted, re-installed - no go.

Anyone have any ideas or fixes?

Running on Netgate SG-1100 appliance.

bmeeks

I have an SG-1100 for testing and will be looking into this issue. The SG-1100 uses a non-Intel CPU and that can bring on a host of potential problems when running C code that was originally developed and widely used on Intel platforms for a long time.

Snort, Suricata and a few other FreeBSD ports packages have or still have issues with running on non-Intel CPUs (meaning arm and aarch64 processors). Previously identified issues in Snort and Suricata were caused by the way upstream coders had taken some "liberty" with memory pointer casting. I don't know if this new issue with the Suricata binary on aarch64 hardware is more of the same or something totally different.

bmeeks

@Mobius114 said in Suricata core dumping after 2.4.5 upgrade:

Hey everyone. Long time listener, first time caller.

After the 2.4.5 update suricata won't start and bombs with the following error:

pid 91657 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)

I have removed the interfaces - no go
I have re-installed the packages - no go
I have removed, rebooted, re-installed - no go.

Anyone have any ideas or fixes?

Running on Netgate SG-1100 appliance.

There is nothing you can do on your end to impact this. It is a problem with the actual compiled code for the package and is due to the SG-1100 appliance having an ARM processor instead of the Intel x86-x64 platform. Signal 4 is the FreeBSD signal for ILLEGAL INSTRUCTION. It indicates an issue with the actual machine code of the Suricata binary executable. It is being looked into, but the fix may not happen quickly.

Mobius114

@bmeeks Appreciate the update and insights into the issue. I look forward to the fix when it's available.

It had been working quite well until the upgrade last night and even ran for about 20 minutes before it crapped out completely.

Cheers

bmeeks

@Mobius114 said in Suricata core dumping after 2.4.5 upgrade:

@bmeeks Appreciate the update and insights into the issue. I look forward to the fix when it's available.

It had been working quite well until the upgrade last night and even ran for about 20 minutes before it crapped out completely.

Cheers

Oh, so it starts and runs for a while and then crashes? My initial info led me to believe it would not even start up.

Unfortunately, running for a while and then randomly crashing makes finding the issue way harder. Much easier to find something that fails immediately and the same way every time.

My SG-1100 is new in the box and I have not actually set it up for testing yet. My main firewall is an SG-5100. Those are Intel-based platforms.

Mobius114

@bmeeks ah I wasn't clear on that.

It ran initially after the upgrade and I got a few alerts before it stopped.

Since that time it has never started since. The core dump error is almost instantaneous.

bmeeks

@Mobius114 said in Suricata core dumping after 2.4.5 upgrade:

@bmeeks ah I wasn't clear on that.

It ran initially after the upgrade and I got a few alerts before it stopped.

Since that time it has never started since. The core dump error is almost instantaneous.

OK. That sounds more like what I had previously heard. Well, as I said, the good news in that is that something immediate and repeatable will hopefully be easier to identify.

h3llo

Same boat for me. Suricata will run for a few seconds (enough to show a green status on the dashboard / interface list) then it dumps.

The suricata.log is blank. Here's from the last run:

1/4/2020 -- 20:21:11 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
1/4/2020 -- 20:21:11 - <Info> -- CPUs/cores online: 2
1/4/2020 -- 20:21:11 - <Info> -- HTTP memcap: 67108864
1/4/2020 -- 20:21:11 - <Notice> -- using flow hash instead of active packets

If you restart you get the "[ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata_mvneta0.409033796.pid' exists but appears stale." issue. If you rm the file via the command prompt it will restart momentarily then dump again.

Additionally, I now have a crash notification on the dashboard. "pfSense has detected a crash report or programming bug. Click here for more information." Detail page shows the following:

Crash report begins.  Anonymous machine information:

arm64
11.3-STABLE
FreeBSD 11.3-STABLE #233 2c992b2181a(factory-RELENG_2_4_5): Tue Mar 24 15:27:54 EDT 2020     root@buildbot3-nyi.netgate.com:/build/factory-crossbuild-245-aarch64/obj/aarch64/Og10eFss/build/factory-crossbuild-245-aarch64/sources/FreeBSD-src/sys/pfSense

Crash report details:

PHP Errors:
[01-Apr-2020 19:36:48 America/Los_Angeles] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 137897032 bytes) in /usr/local/www/csrf/csrf-magic.php on line 149

No FreeBSD crash data found.

I haven't changed any memory allocation with pfsense or Suricata and never have seen this error pop. The dashboard still lists memory usage in the mid-20%.

Only other thing I can find in terms of logs is the Status -> System Log -> System -> General and it reads as follows:

Apr 1 20:10:30 	kernel 		pid 1445 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)

I run an extremely basic setup with suricata as really the only service running other than some basic LAN isolation firewall settings.

Let me know if there is any other information I might be able to add.

h3llo

I should add that Suricata and the SG-1100 ran problem free for six months after some initial, minor adjustments with the flow/stream settings and the usual tweaking of the rules.

bmeeks

@h3llo said in Suricata core dumping after 2.4.5 upgrade:

I should add that Suricata and the SG-1100 ran problem free for six months after some initial, minor adjustments with the flow/stream settings and the usual tweaking of the rules.

Did you not read my posts up above where I described this problem and the fact it is a problem within the binary machine code itself? Here are the relevant lines from one of those posts:

There is nothing you can do on your end to impact this. It is a problem with the actual compiled code for the package and is due to the SG-1100 appliance having an ARM processor instead of the Intel x86-x64 platform. Signal 4 is the FreeBSD signal for ILLEGAL INSTRUCTION. It indicates an issue with the actual machine code of the Suricata binary executable. It is being looked into, but the fix may not happen quickly.

The SG-1100 has a non-Intel architecture CPU. It is an aarch64 instead of an x86-x64. Fixing this one may not happen quickly at all, so if you want an IDS/IPS package on an SG-1100 you should consider switching over to Snort -- at least for the foreseeable future. I know what the problem is, but not where in the megabytes of C and Rust source code it is located. And I'm talking about the Suricata binary source code which comes from upstream - code I did not write.

h3llo

Yes sir I did read the posts above. I just added some additional details regarding the specific errors thrown in the off chance that it might be helpful. If it is not, ignore away.

Appreciate the feedback particularly the expectation management in terms of a fix. While I'm bummed that what once worked does not now, I will take the opportunity to learn a bit about Snort and give it a go.

bmeeks

@h3llo said in Suricata core dumping after 2.4.5 upgrade:

Yes sir I did read the posts above. I just added some additional details regarding the specific errors thrown in the off chance that it might be helpful. If it is not, ignore away.

Appreciate the feedback particularly the expectation management in terms of a fix. While I'm bummed that what once worked does not now, I will take the opportunity to learn a bit about Snort and give it a go.

Blame this problem on two things. One is the Suricata upstream team deciding to use Rust in the binary and to make it mandatory starting with version 5.x. The other is the folks who created the llvm compiler used to create armv6/v7 and aarch64 binary from C source code. That compiler makes some binary instruction choices that are "how shall I say?" -- not optimal ... .

bgalakazam

I just bought a SG-1100 and have this same issue. Sad to find out that it may not be fixed soon. I'll look into Snort in the meantime.

blaytrail

I just posted the same issue after the 2.4.5 Release. I've been working on this for 2 days now. I see I'm not the only one having this issue. See my post below.

Suricata will not start using pfsense. I currently have the SG-1100 appliance.

Added the Suricata package with no issues.
Configured Global Setting using the ETOpen Emerging Threats rules only. No Issues
Performed the Emerging Threats Open Rules update with no issues.
Enabled emerging rules. No Issues
The SG-1100 appliance only has 2 CPU's with 1GB of Ram. It should still run the Suricata application.
When i go to start the service on the WAN interface, it runs for 3 seconds then stops.
Log View message is below
5/4/2020 -- 18:34:18 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
5/4/2020 -- 18:34:18 - <Info> -- CPUs/cores online: 2
5/4/2020 -- 18:34:18 - <Info> -- HTTP memcap: 67108864
5/4/2020 -- 18:34:18 - <Notice> -- using flow hash instead of active packets

bmeeks

@blaytrail said in Suricata core dumping after 2.4.5 upgrade:

I just posted the same issue after the 2.4.5 Release. I've been working on this for 2 days now. I see I'm not the only one having this issue. See my post below.

Suricata will not start using pfsense. I currently have the SG-1100 appliance.

Added the Suricata package with no issues.
Configured Global Setting using the ETOpen Emerging Threats rules only. No Issues
Performed the Emerging Threats Open Rules update with no issues.
Enabled emerging rules. No Issues
The SG-1100 appliance only has 2 CPU's with 1GB of Ram. It should still run the Suricata application.
When i go to start the service on the WAN interface, it runs for 3 seconds then stops.
Log View message is below
5/4/2020 -- 18:34:18 - <Notice> -- This is Suricata version 5.0.2 RELEASE running in SYSTEM mode
5/4/2020 -- 18:34:18 - <Info> -- CPUs/cores online: 2
5/4/2020 -- 18:34:18 - <Info> -- HTTP memcap: 67108864
5/4/2020 -- 18:34:18 - <Notice> -- using flow hash instead of active packets

Look in the pfSense system log and I bet you will see the Signal 4 core dump message disscussed in the posts above. Suricata 5.x on SG-1100 hardware is a no-go for now. You will need to either switch over to Snort or just wait for the fix.

There is absolutely no setting or configuration change a user can make that will change this problem. It is an issue with the compiled binary itself.

blaytrail

Thanks for the update. You are correct. I just looked in the system logs and found the entry below.

Apr 5 20:09:32 kernel pid 38129 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)

I may use AlienVault with the built in Suricata software and create a span port off the netgate device.

Can you create a span port on the Netgate device using the OPT port?

bmeeks

@blaytrail said in Suricata core dumping after 2.4.5 upgrade:

Thanks for the update. You are correct. I just looked in the system logs and found the entry below.

Apr 5 20:09:32 kernel pid 38129 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)

I may use AlienVault with the built in Suricata software and create a span port off the netgate device.

Can you create a span port on the Netgate device using the OPT port?

Not to my knowledge, but I'm no expert on the Marvel switch chip inside the SG-1100. You might want to ask that question in the Netgate Hardware forum on here.

Signal 4 means ILLEGAL INSTRUCTION.

blaytrail

Thanks. I'm looking forward to learning more about IDS\IPS. Surricata seems pretty simple to setup until i ran into the issue. If you don't mind me asking are you doing to switch to another IDS\IPS solution?

bmeeks

@blaytrail said in Suricata core dumping after 2.4.5 upgrade:

Thanks. I'm looking forward to learning more about IDS\IPS. Surricata seems pretty simple to setup until i ran into the issue. If you don't mind me asking are you doing to switch to another IDS\IPS solution?

I am the developer/maintainer for Suricata and Snort on pfSense (volunteer, not paid or affiliated with Netgate). I started with Snort years ago and have never changed. Not because it is better or Suricata is worse, just lazy and didn't want to change out my production firewall.

So for now I'm running Snort on a Netgate SG-5100 appliance. Prior to that I ran it on custom-built hardware (mostly Supermicro 1U servers).

blaytrail

Cool! It must be fun to be the developer/maintainer for the applications. I really enjoy working with pfSense. I just setup the VPN and it works perfectly. I guess i can move up to the SG-5100. I want to set getting more familiar with IDS/IPS. I take it I will need to use something else beside the SG-1100 for IDS/IPS.