Suricata 4.1.5 keeps crashing on SG1100-2.4.4-RELEASE-p3
-
Hi, I have found my Suricata constantly crashing on my SG1100 for months now and have become fed up finally, as crashes will happen multiple times per day, even within 1-2 hours.
I have it setup in Legacy Mode on only the WAN interface and cannot seem to figure it out.
The way I have to deal with it is 2 options: If I suspect Suricata is not working (which means randomly logging into the device Frequently) I then select to "Reboot" the device (killing all connections)
OR, log-in through SSH with Admin credentials, enter the 'Shell', Change to directory " /var/run " and then type "rm sur" & tab to remove the Suricata PID file.
I am not as fluent in the logs but I was unable to discern any obvious issue. I've attached a screenshot of my "Global Settings".
Does no one know how to figure this out or what is causing this?
Thank you for any assistance or direction.
-J
-
You are probably getting a Signal 10 Bus Error message in your system log at the time of the crash (that's my guess). This is a problem on ARM-based hardware and the root cause is the choice made by the llvm-6 compiler for which machine code instructions to use when translating certain sections of C source code into binary.
Look in your pfSense system log and see if you find any Signal 10 errors. Post back with what you find (or don't find).
-
Sorry, this time it happened to take a few days to occur again. The best I was able to discern after monitoring was that it said it ran out of swap space??
Here are the logs:
Nov 11 22:32:30 php-cgi [Suricata] Enabling any flowbit-required rules for: WAN...
Nov 11 22:33:55 kernel pid 90250 (suricata), uid 0, was killed: out of swap space
Nov 11 22:33:55 kernel mvneta0.4090: promiscuous mode disabled
Nov 11 22:33:57 php-cgi [Suricata] Building new sid-msg.map file for WAN...
Nov 11 22:33:59 php-cgi [Suricata] The Rules update has finished.
Nov 12 03:01:00 root rc.update_bogons.sh is starting up.
Nov 12 03:01:00 root rc.update_bogons.sh is sleeping for 64429
Nov 12 04:30:07 php-cgi [Suricata] Emerging Threats Open rules are up to date...
Nov 12 04:30:08 php-cgi [Suricata] Snort VRT rules are up to date...
Nov 12 04:30:08 php-cgi [Suricata] Hide Deprecated Rules is enabled. Removing obsoleted rules categories.
Nov 12 04:30:08 php-cgi [Suricata] Removed 0 obsoleted rules category files.
Nov 12 04:30:08 php-cgi [Suricata] The Rules update has finished.
Nov 12 06:00:01 php-cgi [Suricata] Checking for updated MaxMind GeoLite2 IP database file...
Nov 12 06:00:01 php-cgi [Suricata] GeoLite2-Country IP database is up-to-date.
Nov 12 06:00:01 php-cgi [Suricata] GeoLite2-Country database update check finished. -
You don't have enough RAM in the SG-1100 to run the rules you have enabled. And in particular, during rule updates, Suricata needs to keep two copies of your enabled rules in memory as it swaps out one for the other. You are probably at the ragged edge of running out of memory, and if a given rules update from the Emerging Threats or Snort rules teams enables a few extra rules that were formerly disabled, or adds some completely new rules, it might push your box over the edge during the update process when it needs to keep two copies of the rule set in memory as it swaps from the old to the new set.
FreeBSD will attempt to cycle out dormant sections of RAM data to disk (the swap space) during tight memory situations. If you don't have enough swap space, then things come crashing down. Also note that using swap is super duper SLOW! It will greatly impact the responsiveness of your box.
So you have two solutions:
-
Trim down your enabled rules by not using as many categories and/or by choosing just one set of vendor rules to use: either Emerging Threats or Snort, but not both.
-
Buy another Netgate appliance with more RAM. You really need an SG-5100 to run an IDS/IPS with the most common rules enabled and have some extra RAM left over.
To be perfectly truthful, even though the CPU in the SG-1100 appliance can run an IDS/IPS package, there really is not enough RAM in that box to do it well.
-
-
WOW thank you for the through explanation. Disabling Snort now.
Out of the 3 options (Emerging, Snort Registered, Snort GPL), is there a recommendation for which to choose over the other; this is for home use for me.
Is there any good market place to resell the SG-1100?
-
@petrt3522 said in Suricata 4.1.5 keeps crashing on SG1100-2.4.4-RELEASE-p3:
WOW thank you for the through explanation. Disabling Snort now.
Out of the 3 options, is there a recommendation for which to choose over the other; this is for home use for me.
Is there any good market place to resell the SG-1100?
The Emerging Threats rules are optimized for Suricata. The Snort rules will have some rules within various categories that will fail to load in Suricata because those Snort rules will contain some keywords that Suricata does not accept. I would generally elect to use the Emerging Threats rules on Suricata for that reason. In terms of which rules to choose, you really need to do some research on what the various categories really protect. Several are only useful when you have public-facing servers. The vast majority of home users do not.
You will likely need to cut back on the number of enabled categories you have "checked" on the CATEGORIES tab in order to fit the enabled rules to the RAM available in the SG-1100. You can still run the IDS/IPS on the SG-1100 and be secure, but you just have to be very selective about the number of rules you enable. It's this balancing act of reducing enabled rules to fit in the available RAM that make me say running an IDS/IPS on the SG-1100 is not ideal.
Sorry, but I don't have a suggestion for where to sell your SG-1100 outside of maybe something like eBay.
-
Thanks, sounds good, I'll cut back more categories for starters.
-
Also forgot to mention that if you have a Snort Subscriber Rules subscription (either paid or free), then you do not need to use the Snort GPLv2 rules. The rules in there are already within the Snort Subscriber rule set. So you would just be duplicating rules if you use the Snort GPLv2 Community Rules and the Snort Subscriber rules. The GPLv2 rules are just public free versions of some of the Snort Subscriber rules.