Suricata 7.0.0 being killed by kernel in 23.09.

Gertjan

@jorgek said in Suricata 7.0.0 being killed by kernel in 23.09.:

Nov 15 16:01:28 kernel pid 28473 (suricata), jid 0, uid 0, was killed: failed to reclaim memory

That's close, but not the same, as the famous Out Of Memory (OOM) killer : the OS that kills a process just to keep available memory above 'zero' as that would lock up the system for sure.

Bad memory management is the next best : the process fails memory management (a bug probably) and the process gets 'dumped' by the OS.

"7.0.0" looks like a whole new release ID, full with 'new solutions and new features'.
Also news bugs ^^

bmeeks

@jorgek said in Suricata 7.0.0 being killed by kernel in 23.09.:

The only message logged to system logs is:
Nov 15 16:01:28 kernel pid 28473 (suricata), jid 0, uid 0, was killed: failed to reclaim memory

Hmm... that might be a variation of the same basic bug being talked about here: https://forum.netgate.com/topic/184112/important-snort-and-suricata-package-announcement-probable-bug-in-legacy-blocking-module.

Changes were made to a section of code associated with killing open firewall states for IP addresses that Suricata wants to insert a block for (or Snort, as they share the same basic custom blocking plugin on pfSense).

The changes were made by a Netgate kernel developer to adapt to recent updates in the FreeBSD kernel. I'm working with that developer now to hopefully identify the root cause. The bug is difficult to pin down because it so far has eluded easy and repeatable reproduction.

Maltz

@bmeeks I have found that I can run pfBlocker's DNSBL or Suricata seemingly fine, but if I try to do both, Suricata dies in minutes after it's launched on my Netgate 2100. Perhaps running DNSBL will aid reproducibility?

jorgek

@Maltz @bmeeks I run pfBlocker NG in python mode with Suricat on my 2100 box. The new version of suricata and disabling some lan rules in config (such as stream events that was just flooding the Suricata alerts and blocking a bunch of legitimate IPs (stream services, google, apple, spotify) for example), it seems that helped to keep Suricata running longer without being killed by kernel. I am still monitoring the behavior and fine tuning the "balanced" rules. I have ET open, snort, abuse.ch settings in my config.

jorgek

@bmeeks The latest version of pfsense 23.09.1 and the latest of Suricata, it seems has stabilized memory consumption/memory resources usage.
It's been running for more than 48 hours without being killed by the kernel with that system logs message.
I am keeping monitoring the system, but I can tell that is behaving better.

Maltz

@jorgek Unfortunately, pfSense 23.09.1 + Suricata 7.0.2_2 has not solved the issue for me. The kernel continues to kill Suricata with the "failed to reclaim memory" error for all matching algorithms other than AC-BS.

jorgek

@Maltz damn. It happened again :( I don't know what else what to do to make suricata running from configuration/tuning perspective. It's being annoying since the upgrade.

jorgek

not sure if the memory issue it could be linked when pfBlockerNG cron runs:
Dec 13 09:31:46 kernel pid 52543 (suricata), jid 0, uid 0, was killed: failed to reclaim memory
Dec 13 09:31:35 php 56350 [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload

Dec 12 20:31:07 php 20653 [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
Dec 12 20:31:00 kernel pid 41134 (suricata), jid 0, uid 0, was killed: failed to reclaim memory

Maltz

@jorgek Have you tried switching the Pattern Matcher Algorithm to AC-BS? That workaround has worked for me. It's supposedly not as efficient, but I'm still getting the same speeds (~700Mbps both ways) as I was before.

jorgek

@Maltz hmmm, I will try that and monitor.