Suricata crashes on rule reload
-
Hello all,
I'm using pfsense hardware (sg-2100) and suricata 7.0.2_1.
FYI On the same device I've also enabled pfblockerNG and OpenVPN.I've been running Suricata for a couple of weeks with its default rules in alert mode, to finetune the system.
Yesterday, I decided to also enable the ET rule categories, and i've noticed a severe bug/malfunction.
Every time the rules have to reload due to a change, my Suricata crashes/exits, and with it my DNS resolver stops too.
This creates total network shutdown, and is obviously not ideal.I can see the following in the system logs:
Mar 7 01:38:37 php-cgi 99740 [Suricata] The Rules update has finished.
Mar 7 01:40:26 kernel pid 14883 (suricata), jid 0, uid 0, was killed: failed to reclaim memory
Mar 7 01:40:26 kernel pid 81150 (unbound), jid 0, uid 59, was killed: failed to reclaim memory
Mar 7 01:40:26 kernel mvneta1: promiscuous mode disabled
Mar 7 01:43:37 dhcpleases 79433 Could not deliver signal HUP to process 81150: No such process.
Mar 7 01:44:43 dhcpleases 79433 Could not deliver signal HUP to process 81150: No such process.
Mar 7 01:45:59 dhcpleases 79433 Could not deliver signal HUP to process 81150: No such process.I am wondering if Suricata is too taxing for my system, I really hope it isn't!
-
@b3rt said in Suricata crashes on rule reload:
ar 7 01:40:26 kernel pid 14883 (suricata), jid 0, uid 0, was killed: failed to reclaim memory
Mar 7 01:40:26 kernel pid 81150 (unbound), jid 0, uid 59, was killed: failed to reclaim memoryIMHO, that indicates the machine is running out of memory - check if there is also a use of swap. Could be that the loaded rules are too big for that sg-2100 ...
(but maybe @bmeeks ca tell you more) -
Yes, the SG-2100 really does not have enough free RAM to run Suricata very well unless you use a severely limited rule set. During the rules update/reload scenario, RAM usage can nearly double as Suricata will have two copies of the rule set in memory for a time during the swap over. I would set 4 GB of RAM as the minimum in my view, with 8 GB being best.
When you say "I decided to also enable the ET rule categories", I assume you mean all of them. That's going to probably exhaust RAM quickly. You need to be very selective about which ET categories you enable. Many of them are totally not applicable for home networks (and even for many business networks).
Managing an IPS/IDS is not just "enable everything and sit back". It requires careful analysis of the vulnerabilities in your network and then selectively enabling only those rules that specifically protect against those vulnerabilities. Most new security admins don't get this. They want to treat IDS/IPS the same as an anti-virus product. But the two are in no way similar.
-
@bmeeks thanks for the reply!
I had 11 ET rule categories disabled, but I did indeed enable all the others at once.
It's sad to hear my router probably won't handle suricate then, don't want my network to have the possibility to go down everytime I change a rule set setting, even though i was already implementing the sid management way of editing the rules.The first weeks i could filter out many general rules, and i thought i could repeat this process for the ET ruleset.
@bmeeks said in Suricata crashes on rule reload:
Managing an IPS/IDS is not just "enable everything and sit back". It requires careful analysis of the vulnerabilities in your network and then selectively enabling only those rules that specifically protect against those vulnerabilities. Most new security admins don't get this. They want to treat IDS/IPS the same as an anti-virus product. But the two are in no way similar.
Not that new in IT security, but hands-on management of an IDS/IPS, true. You say to "only enable the rules that protect against vulnerabilities in your network", meaning potential vulnerabilities right? eg if i have client devices, then my emerging-coinminer rules should be enabled and configured.
-
@b3rt said in Suricata crashes on rule reload:
meaning potential vulnerabilities right? eg if i have client devices, then my emerging-coinminer rules should be enabled and configured.
Yes, but for a home network you would usually not need these categories:
- ET-Policy
- ET-Info
- anything to do with mail servers, web servers or DNS servers (as you don't run those servers in a typical home network).
- and others in a similar vein
I created the Suricata package on pfSense, and I have maintained and updated the Snort package on pfSense for years. I worked in cybersecurity managing firewalls and IPS for many years for a Fortune 500 corp. But in my home network, I don't run either package. Not worth the hassle for the very limited benefit. There is just me and my wife on the network. I keep all my machines updated with the latest security hotfixes and updates. We both know not to click willy-nilly on email attachments- especially any from a sender we did not request something from. I have not had a single security issue in the history of my home network (and it goes back a lot of years).
I use a small bank of VMware virtual machines to run the Suricata and Snort packages for testing when I do software updates to them, but I quit running an IDS/IPS on my home network probably 5 years ago. And even back then I was only running it on my WAN just to collect alerts for testing.
So, what am I saying? I'm saying that for the majority of home network users there is zero point in running an IDS/IPS. To start with, 90% or more of traffic today is encrypted (HTTPS, SMTPS, DoT, DoH, TLS, etc.). The IDS/IPS can't see into those payloads at all. You could hide anything in the packet payload of encrypted traffic and the IDS/IPS will be blind. But with a lot of the rules, you can certainly get a ton of false-positive nuisance blocks and alerts that will keep you constantly trying to figure out why something is suddenly not working.
-
Since I have a NAS in my home network, and I am planning to host a site on it, and run a honeypot (not necessarily in that order), I am steadily increasing my network security :). It is true that the blindness of an IDS / IPS in regards to encryption puts a serious marker on their usefulness.
To familiarise myself with actual malicious alerting in suricata, i thought honeypot x suricata on my FW would make sense.
I do like how, in contrast to pfblockerng, it gives me much more information as to why it alerts (or blocks), eg. an emerging-dns 'contacting a .to domain'.@bmeeks said in Suricata crashes on rule reload:
Yes, but for a home network you would usually not need these categories:
- ET-Policy
- ET-Info
Luckily those were already disabled! I've added an attachment in spoiler of how i currently configured the ET rules.
Didn't find an easier way to export the currently active rules
-
@b3rt: go learn about the features on the SID MGMT tab in the package. So much easier to manage rules and categories that way.
You will find a Sticky Post on the topic at the top of this forum. The feature is available for both Snort and Suricata, and it works the same way in both packages.
-
@b3rt said in Suricata crashes on rule reload:
even though i was already implementing the sid management way of editing the rules.Thx, already on it :).
-
@b3rt said in Suricata crashes on rule reload:
host a site on it, and run a honeypot
Two very bad ideas in my opinion.
First, do you have a static IP from your ISP? If not, hosting a public website is more of a challenge requiring a Dynamic DNS provider configuration. Additionally, almost every ISP I've heard about prohibits hosting public services on your residential connection in their terms of service. You may find yourself in hot water at some point with your ISP if you host a website. If you want to host a website, pay the small fee for a genuine hosting service using a VPS at a data center. They at least likely have the infrastructure to protect from most DDoS attacks. Your ISP and your residential connection can never hope to cope with a DDoS attack. And if one is launched against your public site on your residential IP because somebody gets mad with you, then your ISP is certainly going to terminate your connection (and likely permanently).
Why would you want a honeypot on your home network? I can't fathom that at all. It's like riding through the most crime-ridden neighborhood in your town in an expensive convertible sports car wearing diamond jewelry and a Rolex watch in plain view. You are going to attract the attention of someone who would have otherwise ignored you.
-
@bmeeks The honeypot would be located in a different subnet (a dmz, as you may). Primarily I want to collect attack patterns logs, analyze them on a grander scale in my ELK stack.
@bmeeks said in Suricata crashes on rule reload:
First, do you have a static IP from your ISP?
My public IP has changed once or twice since i've started using my pfsense about a year ago, so not static.
Afaik my ISP should allow it, with port forwarding and ddyns it shouldn't be a problem. I'll definitely read through EULA again before it do it though, thanks for the tip! -
@b3rt said in Suricata crashes on rule reload:
@bmeeks The honeypot would be located in a different subnet (a dmz, as you may). Primarily I want to collect attack patterns logs, analyze them on a grander scale in my ELK stack.
@bmeeks said in Suricata crashes on rule reload:
First, do you have a static IP from your ISP?
My public IP has changed once or twice since i've started using my pfsense about a year ago, so not static.
Afaik my ISP should allow it, with port forwarding and ddyns it shouldn't be a problem. I'll definitely read through EULA again before it do it though, thanks for the tip!You might also want to check if your IP from your ISP is in CGNAT space. That is becoming much more common these days than it used to be. If in CGNAT space, then outside direct access is not possible unless your ISP creates the necessary port forwards in their routing infrastructure. I would say the odds of them agreeing to do that are slim and none
. -
@bmeeks Fortunately, this is not the case for my ISP :). I already thought it wasn't a problem since my openvpn setup just works like <public_ip:port> and port forwarding isn't really a problem (although this could also be due to PCP afaik).
-
Just as an addendum, i've noticed my device(the SG-2100) wouldn't support inline blocking mode, so i'd have to use legacy mode.
I'll need to rethink my strategy for network defense because this HW doesn't seem up for it. -
@b3rt said in Suricata crashes on rule reload:
Just as an addendum, i've noticed my device(the SG-2100) wouldn't support inline blocking mode, so i'd have to use legacy mode.
I'll need to rethink my strategy for network defense because this HW doesn't seem up for it.Yes, the Inline IPS Mode is limited to only NICs that support native netmap operation. While netmap can run in an emulated driver mode, that mode is quite inefficient and can handicap IPS mode performance, especially on marginal hardware.
The PHP code checks the NIC in use against a list of drivers known to support native netmap operation. If the NIC is not in that list, an error is printed and the change to Inline IPS Mode is not allowed. On FreeBSD, these are the approved drivers:
cxgbe,em,iflib(providingvmx,igbandem),ixgbe,ixl,re, andvtnet -
@bmeeks Right! The sg-2100 has the mvnat variant.
I did however improve the stability of the system by making some changes to my auto backup config. Turning of 'creating a backup after each config change', has enabled me to update suricate rules without it crashing. I'll test this evening to see up to which point!
