pfSense Suricata Crashes on Malformed Block List Entry
-
@micah said in pfSense Suricata Crashes on Malformed Block List Entry:
@bmeeks Sorry that I haven't had a chance to reply. I tested, and it fixed everything! Thank you very much!
You're welcome. Thank you for following up with the confirmation.
I never did figure out how the blank lines got in the file, so I just fixed the code reading in the log file to detect and skip blank lines.
One theory is maybe it happens during log rotation, but that's just a guess.
-
@bmeeks that would make sense. It does take a few hours to appear again after I purge the block list.
-
Hi @bmeeks,
Just a heads up. I just logged in to check my pfSense today and it appears to still be crashing. That's so odd. It seems like it worked just fine for a few days after your patch.
-
I'm seeing the same thing. Seems to happen after a rules updatte.
Dec 20 00:39:27 kernel pid 95704 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)
Dec 20 00:36:05 kernel pid 91608 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)
Dec 20 00:30:50 php-cgi 11190 [Suricata] The Rules update has finished.I've also seen a couple entries in the Suricata log about running out of memory which has never happened. I'm running a Netgate 3100 with a pretty small ruleset and have plenty of memory.
-
@micah said in pfSense Suricata Crashes on Malformed Block List Entry:
Hi @bmeeks,
Just a heads up. I just logged in to check my pfSense today and it appears to still be crashing. That's so odd. It seems like it worked just fine for a few days after your patch.
Crashing with the exact same PHP error message or something else?
-
@bitslammer said in pfSense Suricata Crashes on Malformed Block List Entry:
I'm seeing the same thing. Seems to happen after a rules updatte.
Dec 20 00:39:27 kernel pid 95704 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)
Dec 20 00:36:05 kernel pid 91608 (suricata), jid 0, uid 0: exited on signal 11 (core dumped)
Dec 20 00:30:50 php-cgi 11190 [Suricata] The Rules update has finished.I've also seen a couple entries in the Suricata log about running out of memory which has never happened. I'm running a Netgate 3100 with a pretty small ruleset and have plenty of memory.
I found two more bugs yesterday in the custom Legacy Blocking Module we use in Suricata on pfSense. Found those while looking into the reported Hyperscan fatal error exit bug being discussed in another very long thread in this sub-forum.
The additional bugs I found yesterday can most definitely lead to the Signal 11 segfault errors you are experiencing. They come into play when accessing the Pass List. They also appear to have been impacting the Hyperscan library functionality, too.
I am waiting on final confirmation from the Hyperscan bug testers running a test build I sent them that my fixes yesterday actually worked. They are letting their machines run for some time to be sure there is no latent crash. If the fixes have indeed solved the issue, then I will submit a pull request to the Netgate developer team with a package update.
-
@bmeeks Not sure. I will try and do more digging the next time. It's intermittent.
-
Was able to grab more logs with the most recent crash. I did not see any errors in the system or PHP logs.
[108473 - Suricata-Main] 2023-12-22 00:31:02 Notice: detect: rule reload complete
[108473 - Suricata-Main] 2023-12-23 00:30:21 Notice: detect: rule reload starting
[108473 - Suricata-Main] 2023-12-23 00:30:21 Info: conf-yaml-loader: Configuration node 'filetype' redefined.
[108473 - Suricata-Main] 2023-12-23 00:30:21 Error: detect-within: previous keyword has a fast_pattern:only; set. Can't have relative keywords around a fast_pattern only content
[108473 - Suricata-Main] 2023-12-23 00:30:21 Error: detect: error parsing signature "alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"MALWARE-OTHER Win.Trojan.Zeus Spam 2013 dated zip/exe HTTP Response - potential malware download"; flow:to_client,established; content:"-2013.zip|0D 0A|"; fast_pattern:only; content:"-2013.zip|0D 0A|"; http_header; content:"-"; within:1; distance:-14; http_header; file_data; content:"-2013.exe"; content:"-"; within:1; distance:-14; metadata:impact_flag red, policy balanced-ips drop, policy max-detect-ips drop, policy security-ips drop, ruleset community, service http; reference:url,www.virustotal.com/en/file/2eff3ee6ac7f5bf85e4ebcbe51974d0708cef666581ef1385c628233614b22c0/analysis/; classtype:trojan-activity; sid:26470; rev:2;)" from file /usr/local/etc/suricata/suricata_3659_mvneta1/rules/suricata.rules at line 163
[108473 - Suricata-Main] 2023-12-23 00:30:27 Info: detect: 2 rule files processed. 16950 rules successfully loaded, 1 rules failed
[108473 - Suricata-Main] 2023-12-23 00:30:27 Info: threshold-config: Threshold config parsed: 0 rule(s) found
[108473 - Suricata-Main] 2023-12-23 00:30:28 Info: detect: 16950 signatures processed. 131 are IP-only rules, 2837 are inspecting packet payload, 13856 inspect application layer, 7 are decoder event only
[108473 - Suricata-Main] 2023-12-23 00:30:28 Warning: detect-flowbits: flowbit 'et.http.PK' is checked but not set. Checked in 2019835 and 1 other sigs
[108473 - Suricata-Main] 2023-12-23 00:30:36 Error: mpm-ac: Error allocating memory -
@bitslammer said in pfSense Suricata Crashes on Malformed Block List Entry:
[108473 - Suricata-Main] 2023-12-23 00:30:36 Error: mpm-ac: Error allocating memory
This line tells you the problem. You do not have enough free RAM in the box to run the number of rules you have enabled with Suricata 7.x and its increased memory requirements for TCP stream memcap and reassembly memcap.
It's time to replace your hardware or else abandon attempting to run Suricata on it. That is 32-bit ARM hardware, so Hyperscan won't work at all.
You can try rather drastically reducing your enabled rules to see if that helps. I see you have 16,950 rules enabled. With Suricata 7.x and its increased memory requirements, that's pushing the ragged edge - especially if you have any other packages running as well.
-
@bmeeks I'm guessing the rules that I enabled have grown over time so I'll try to trim them. Oddly this doesn't happen every time which you'd kind of expect. It's a Netgate 3100 so it looks like I need to look at some other options since this isn't upgradeable. Thanks for the quick reply adn happy holidays.
-
@bitslammer said in pfSense Suricata Crashes on Malformed Block List Entry:
@bmeeks I'm guessing the rules that I enabled have grown over time so I'll try to trim them. Oddly this doesn't happen every time which you'd kind of expect. It's a Netgate 3100 so it looks like I need to look at some other options since this isn't upgradeable. Thanks for the quick reply adn happy holidays.
Several memory parameters have new increased minimums in Suricata 7.x. You are probably seeing the impact of those on the SG-3100. Same issue exists for SG-1100 users, too. 4GB is the new minimum, and even that might get cramped with lots of rules (more than 15,000).
If I were spec'ing a box today for someone who wanted to run IDS/IPS (and most folks want to run pfBlockerNG with DNSBL, too), then I would set 8 GB as a new minimum RAM requirement. The new default ZFS install will also chew up much more RAM than the old UFS setup.
Edit: also looking once again at your log snippet post, I see it seemed to be updating the rules as I see a "rule reload" message. RAM usage will increase during rule swaps, especially if "live rule swap" is enabled.
