Suricata eats all swap

ballistic

Hi,

I have a dedicated server (2 in fact, in HA) as PFsense. I have Suricata running on 8 interfaces with lots of rules. No problem for the Xeon E-2236 and 16GB RAM.

When starting all Suricata instances, memory usage ends up at about 55% and over time drops down to 18%

The problem is, over time, depending on the amount of traffic going through the firewall, the SWAP gets fully consumed. This is 24hrs after start;

When shutting down the Suricata instances, the swap is freed up again.

Not a single CLI tool shows me any SWAP statistics;
PID USERNAME THR PRI NICE SIZE RES SWAP STATE C TIME WCPU COMMAND
72722 root 11 20 0 3277M 2674M 0B nanslp 2 7:49 1.06% suricata
71637 root 11 52 20 2073M 1487M 0B nanslp 2 7:55 0.71% suricata
76097 root 11 52 20 2101M 1484M 0B nanslp 0 7:54 0.81% suricata
95829 root 11 52 20 2129M 1031M 0B nanslp 4 9:41 0.99% suricata
89729 root 11 52 20 2093M 673M 0B nanslp 2 8:18 0.84% suricata
81819 root 10 52 20 685M 412M 0B nanslp 1 13:45 1.86% suricata
21840 root 10 52 20 588M 378M 0B nanslp 4 7:43 0.92% suricata
1514 root 11 52 20 585M 216M 0B nanslp 2 40:00 10.91% suricata

How would I continue troubleshooting this?

bmeeks

Suricata memory consumption can nearly double during rules updates because a copy of both the "old rules" and the "new rules" are kept in memory simultaneously for a bit as the swapout happens. This is true when the "Live Rule Swap" option is enabled on the GLOBAL SETTINGS tab.

However, even if Suricata consumed some swap space during the rules update procedure, I would expect that swap to be released when the rules swapout terminates.

Have your scoured both the pfSense system log and the suricata.log file for the affected Suricata interfaces to see if any pertinent error or warning messages are present?

When does the swap usage manifest itself? Does it happen immediately upon startup of the Suricata processes, or does it creep up later over time? You can stop and then restart all interfaces simulataneously by using the shell script from the CLI here:

/usr/local/etc/rc.d/suricata.sh stop
/usr/local/etc/rc.d/suricata.sh start

Monitor the swap usage while the script above executes to see when swap gets used.

ballistic

@bmeeks Thank you.
Live rule swap is disabled.

Swap is not used directly after Suricata start. 15min after start now and no increase of swap at all.
I will monitor closely the next 24hrs to see when it starts.

Attached some pfsense.suricata.logging.txt

bmeeks

Your firewall seems to be continually synchronizing the settings via XMLRPC Sync. Notice that one cycle barely finishes before another starts. Each of those sync operations is causing a rebuild of the rules on the sync target. That can consume a lot of RAM.

For example:

2024-07-31 17:11:29.189078+02:00	php	39488	[suricata] XMLRPC sync is starting.
...
2024-07-31 17:11:29.711288+02:00	php	39488	[suricata] XMLRPC sync completed.

and then, 26 seconds later:

2024-07-31 17:11:55.043569+02:00	php	68750	[suricata] XMLRPC sync is starting.

Also saw this error in the logs near the top --

2024-07-31 13:36:39.679094+02:00	kernel	-	swp_pager_getswapspace(24): failed
2024-07-31 13:36:39.679014+02:00	kernel	-	swap_pager: out of swap space

ballistic

@bmeeks That is probably because I stopped/started the Suricata instances one-by-one. Using the script and starting them all at the same time will probably overload the CPU.

Swap has reached 100% today so that is probably the reason for the Out of Swap error.

An hour later and everything still stable at 44% RAM and 19% SWAP

bmeeks

Suricata should never use any swap space. Using any amount of swap is highly inefficient and is generally to be avoided.

While I doubt it will have any bearing on the issue you are seeing, the latest version of the Suricata package is 7.0.6. You still have 7.0.4 installed according to the suricata.log messages. Upgrading certainly would not hurt.

You don't say, but I assume you are also running the most recent version of pfSense. If not, you certainly should upgrade it as well. For pfSense CE, the current version is 2.7.2. And for pfSense Plus, the current version is 24.03. If you are not yet running the current pfSense version, upgrade that first and only afterward should you upgrade the Suricata package.

ballistic

@bmeeks Upgrading did not help.

What did help was disable the Extra Rules I had configured. 48 hours with no increased swap sofar.

Using https://urlhaus.abuse.ch/downloads/urlhaus_suricata.tar.gz as extra ruleset will eat all swap.