Suricata v7.0.7_5 abruptly stops

slu

@anishkgt
maybe @bmeeks can help you with this error.

SteveITS

@anishkgt Typically the PID file gets left behind when a process crashes unexpectedly. Typically one can just delete it if it's not held open. What do you see when you try?

Have you tried restarting the router and then deleting it?

anishkgt

@SteveITS This is what i get when deleting the file after the service has stopped.

[24.11-RELEASE][user@pfSense.home.arpa]/var/run: rm -f suricata_mvneta158898.pid
rm: suricata_mvneta158898.pid: Operation not permitted

After disabling Suricata i was able to delete the file. I've started the service again. Lets see how it goes.

bmeeks

@anishkgt said in Suricata v7.0.7_5 abruptly stops:

I know i can use the service watchdog to restart it

No and NO!!!

Never use Service Watchdog with either of the two IDS/IPS packages (Suricata and Snort). That package does not understand how the IDS/IPS packages operate and will needlessly issue duplicate restart commands to the binary resulting in duplicate "zombie" instances of the binary running and causing major issues.

The PID file issue is a consequence of the Suricata process abruptly dying. Normally the binary clears that file when it shuts down in an orderly manner. But if the binary executable crashes suddenly, it will leave an orphaned PID lock file in the /var/run/ directory. You can safely delete that file and then Suricata will restart. There will be a separate PID file for each configured and running Suricata instance.

anishkgt

@bmeeks So today i have just noticed that suricata has stopped. How can i troubleshoot this ?

bmeeks

@anishkgt said in Suricata v7.0.7_5 abruptly stops:

@bmeeks So today i have just noticed that suricata has stopped. How can i troubleshoot this ?

The same way you should troubleshoot any issue with software -- check all the applicable log files to see if anything related to the crash has been logged.

On pfSense that means the system log (under STATUS > SYSTEM LOG). Next, check the suricata.log file under the LOGS VIEW tab of the Suricata GUI. Do not attempt to restart Suricata until you have checked these logs! Each time Suricata is restarted it will clear out the suricata.log file and start with a fresh copy.

Post anything related you find in either log file back here in this thread.

anishkgt

@bmeeks I can see the following error in the system log

Dec 4 22:43:37	kernel		mvneta1: promiscuous mode disabled
Dec 4 22:43:37	kernel		pid 51709 (php-fpm), jid 0, uid 0, was killed: failed to reclaim memory
Dec 4 22:43:37	kernel		pid 26454 (suricata), jid 0, uid 0, was killed: failed to reclaim memory

What does that mean ? Running low on memory ?

bmeeks

@anishkgt said in Suricata v7.0.7_5 abruptly stops:

@bmeeks I can see the following error in the system log

Dec 4 22:43:37	kernel		mvneta1: promiscuous mode disabled
Dec 4 22:43:37	kernel		pid 51709 (php-fpm), jid 0, uid 0, was killed: failed to reclaim memory
Dec 4 22:43:37	kernel		pid 26454 (suricata), jid 0, uid 0, was killed: failed to reclaim memory

What does that mean ? Running low on memory ?

Yes, your firewall is running out of free RAM. You are seeing the actions of the OOM (out-of-memory) Killer in action within FreeBSD. When a critical process requests an allocation of memory, the kernel's memory management subsystem looks for enough available free blocks to satisfy the request. If none are available, the OOM Killer is activated and it looks for the user process currently consuming the most RAM and attempts to kill that process to free up memory that can be allocated to fulfill the request from the critical process. The idea behind this logic is to make sure the kernel and its critical processes can always have RAM to run even when that means unilaterally killing some user process.

You get in this situation by having too much stuff running on the machine. Suricata can consume a lot of RAM. So can some features enabled in packages such as pfBlockerNG and ntopng.

anishkgt

@bmeeks considering the appliance is low on RAM (Netgate 2100). What would be an ideal configuration to run PfblockerNG and Suricata.
Does Netgate allow replacing the current model with another ?

SteveITS

@anishkgt What lists are you using in pfBlocker? We have never had a problem running those together at clients, but, we don't use, for instance, the UT1 list which is gigantic, and we don't just enable all the Suricata rules either.

I don't think Netgate has an exchange program. And I think the 6100 is the first model with 8 GB of RAM but I didn't look into that.

anishkgt

@SteveITS Forgot mention. I've been test running the nextDNS service which seems to be doing what pfblockerNG does. So, as of now I have stopped pfblockerNG and i was only running Suricata.

This is what i have installed in the package manager

aws-wizard
bandwidthd
ipsec-profile-wizard
ntopng
pfBlockerNG
Service_Watchdog
System_Patches
WireGuard

Of which ntopng and pfBlockerNG is nor running. I don't see much CPU (49%) and memory (13%) usage.

bmeeks

@anishkgt said in Suricata v7.0.7_5 abruptly stops:

@SteveITS Forgot mention. I've been test running the nextDNS service which seems to be doing what pfblockerNG does. So, as of now I have stopped pfblockerNG and i was only running Suricata.

This is what i have installed in the package manager

aws-wizard

bandwidthd

ipsec-profile-wizard

ntopng

pfBlockerNG

Service_Watchdog

System_Patches

WireGuard

Of which ntopng and pfBlockerNG is nor running. I don't see much CPU (49%) and memory (13%) usage.

What is Service Watchdog monitoring? It should NEVER be configued to monitor and restart the IDS/IPS packages (Suricata or Snort). It does not understand how to properly monitor those package binaries for correct operation nor does it know how to correct restart individual failed instances.

Your problem is most likely a result of insufficient free RAM. Large rulesets combined with certain configuration options in Suricata can lead to huge RAM usage. This usage increases even more during rule updates.

anishkgt

@bmeeks The Service_watchdog monitors the KEA_DHCP Server and the DNS Resolver. So would somehthing aroung 8GB be worth it. Since Netgate does not offer exchange, I might as well go with Protectli. I was hoping netgate would allow that way i could contribute to pfsense development.

SteveITS

@anishkgt I think you should find out what is using RAM to find out the cause, and what your setup requires. Check Diagnostics/Activity or run "top" at a command line.

To clarify my comment above, many of our clients have 2100s and we've never had a memory problem with Suricata.

Do you have a 2100 Max? A SSD is recommended to run ntopng because of the disk writing.
https://www.netgate.com/supported-pfsense-plus-packages

anishkgt

@SteveITS

This is how it looks. I have the 2100 Max model. Carn't really recall if it is SSD.

SteveITS

@anishkgt The Max has an SSD.

That shows 2791 MB free. I guess you'll have to watch it for when the out of memory crash happens.

anishkgt

@SteveITS I don't get any notification that i could really monitor the Suricata service. How can i monitor ? i mean how can it checked against the time the service stops.

bmeeks

@anishkgt said in Suricata v7.0.7_5 abruptly stops:

@SteveITS I don't get any notification that i could really monitor the Suricata service. How can i monitor ? i mean how can it checked against the time the service stops.

Look for any other events in the pfSense system log that occur around the same time as the out-of-memory process killer log entry.

The entry that says:

Dec 4 22:43:37	kernel		pid 51709 (php-fpm), jid 0, uid 0, was killed: failed to reclaim memory
Dec 4 22:43:37	kernel		pid 26454 (suricata), jid 0, uid 0, was killed: failed to reclaim memory

Post the messages a few minutes either side of the time where the above message is logged. That may offer a clue as to what other processes were trying to do something that triggered the out-of-memory condition.

anishkgt

@bmeeks Here is what i can see in the system logs. I have not started the suricata service ever since it had stopped.

Dec 14 00:01:42	kernel		mvneta1: promiscuous mode disabled
Dec 14 00:01:42	kernel		pid 15084 (unbound), jid 0, uid 59, was killed: failed to reclaim memory
Dec 14 00:01:42	kernel		pid 91364 (suricata), jid 0, uid 0, was killed: failed to reclaim memory
Dec 13 20:56:28	kernel		mvneta1: promiscuous mode enabled
Dec 4 22:43:37	kernel		mvneta1: promiscuous mode disabled
Dec 4 22:43:37	kernel		pid 51709 (php-fpm), jid 0, uid 0, was killed: failed to reclaim memory
Dec 4 22:43:37	kernel		pid 26454 (suricata), jid 0, uid 0, was killed: failed to reclaim memory
Dec 4 18:15:21	kernel		mvneta1: promiscuous mode enabled
Dec 4 00:01:48	kernel		mvneta1: promiscuous mode disabled
Dec 4 00:01:48	kernel		pid 12763 (php-fpm), jid 0, uid 0, was killed: failed to reclaim memory
Dec 4 00:01:48	kernel		pid 4569 (suricata), jid 0, uid 0, was killed: failed to reclaim memory
Dec 3 20:54:12	kernel		mvneta1: promiscuous mode enabled
Dec 3 11:02:53	kernel		mvneta1: promiscuous mode disabled
Dec 3 11:02:53	kernel		pid 31700 (php-fpm), jid 0, uid 0, was killed: failed to reclaim memory
Dec 3 11:02:51	kernel		pid 73039 (suricata), jid 0, uid 0, was killed: failed to reclaim memory
Dec 3 09:33:53	kernel		mvneta1: promiscuous mode enabled
Dec 3 00:01:50	kernel		mvneta1: promiscuous mode disabled
Dec 3 00:01:50	kernel		pid 90934 (php-fpm), jid 0, uid 0, was killed: failed to reclaim memory
Dec 3 00:01:50	kernel		pid 75984 (suricata), jid 0, uid 0, was killed: failed to reclaim memory

I can see the DNS Resolver which i assume is the "unbound" was also killed. I remeber earlier it being an issue when suricata stopped. What could be root cause.

bmeeks

@anishkgt said in Suricata v7.0.7_5 abruptly stops:

@bmeeks Here is what i can see in the system logs. I have not started the suricata service ever since it had stopped.

Dec 14 00:01:42	kernel		mvneta1: promiscuous mode disabled
Dec 14 00:01:42	kernel		pid 15084 (unbound), jid 0, uid 59, was killed: failed to reclaim memory
Dec 14 00:01:42	kernel		pid 91364 (suricata), jid 0, uid 0, was killed: failed to reclaim memory
Dec 13 20:56:28	kernel		mvneta1: promiscuous mode enabled
Dec 4 22:43:37	kernel		mvneta1: promiscuous mode disabled
Dec 4 22:43:37	kernel		pid 51709 (php-fpm), jid 0, uid 0, was killed: failed to reclaim memory
Dec 4 22:43:37	kernel		pid 26454 (suricata), jid 0, uid 0, was killed: failed to reclaim memory
Dec 4 18:15:21	kernel		mvneta1: promiscuous mode enabled
Dec 4 00:01:48	kernel		mvneta1: promiscuous mode disabled
Dec 4 00:01:48	kernel		pid 12763 (php-fpm), jid 0, uid 0, was killed: failed to reclaim memory
Dec 4 00:01:48	kernel		pid 4569 (suricata), jid 0, uid 0, was killed: failed to reclaim memory
Dec 3 20:54:12	kernel		mvneta1: promiscuous mode enabled
Dec 3 11:02:53	kernel		mvneta1: promiscuous mode disabled
Dec 3 11:02:53	kernel		pid 31700 (php-fpm), jid 0, uid 0, was killed: failed to reclaim memory
Dec 3 11:02:51	kernel		pid 73039 (suricata), jid 0, uid 0, was killed: failed to reclaim memory
Dec 3 09:33:53	kernel		mvneta1: promiscuous mode enabled
Dec 3 00:01:50	kernel		mvneta1: promiscuous mode disabled
Dec 3 00:01:50	kernel		pid 90934 (php-fpm), jid 0, uid 0, was killed: failed to reclaim memory
Dec 3 00:01:50	kernel		pid 75984 (suricata), jid 0, uid 0, was killed: failed to reclaim memory

I can see the DNS Resolver which i assume is the "unbound" was also killed. I remeber earlier it being an issue when suricata stopped. What could be root cause.

Your firewall kernel is running out of free memory space for critical procesess. This line is the clue:

Dec 4 00:01:48	kernel		pid 4569 (suricata), jid 0, uid 0, was killed: failed to reclaim memory

and these --

Dec 14 00:01:42	kernel		pid 15084 (unbound), jid 0, uid 59, was killed: failed to reclaim memory
Dec 3 00:01:50	kernel		pid 90934 (php-fpm), jid 0, uid 0, was killed: failed to reclaim memory

Suricata and the other processes listed are being arbitrarily killed by the FreeBSD kernel's OOM (out-of-memory) reaper. I explained in an earlier post in this thread what that means and how it works: https://forum.netgate.com/topic/195456/suricata-v7-0-7_5-abruptly-stops/9?_=1734182772335.

You do not have enough RAM in an SG-2100 to run all the packages and options you have enabled. You will need to drastically trim your Suricata ruleset and/or curtail use of some pfBlockerNG DNSBL lists.

When Suricata is updating its rules the memory usage can nearly double. The same sort of RAM consumption increase likely happens when pfBlockerNG updates its DNSBL lists and the unbound Python module is running. It is likely that during one of these times is when the OOM reaper engages and kills one or more of your user space processes.

One last comment, I don't know if you are running pfSense in ZFS mode, but if you are that can exacerbate memory problems on a box with limited RAM running optional packages due to the ZFS ARC (Adaptive Replacement Cache). ZFS likes to use free RAM for caching.