Suricata won't stop
-
Suricata still rock stable.
Regarding the config; It's probably my insane amount of enabled rules that is causing the issue. There are only 5500 lines in the config but some lines are miles long, even Notepad++chokes on it. Specially this one;
<customrules>YWxlcnQgdGNwICRIT01FX05FVCBhbnkgLT4gWzY3L... +another few miles of txt
And that times 3 (3 suricata enabled interfaces) I will look into more efficient rule enabling. Thanks!
-
@ballistic said in Suricata won't stop:
Suricata still rock stable.
Regarding the config; It's probably my insane amount of enabled rules that is causing the issue. There are only 5500 lines in the config but some lines are miles long, even Notepad++chokes on it. Specially this one;
<customrules>YWxlcnQgdGNwICRIT01FX05FVCBhbnkgLT4gWzY3L... +another few miles of txt
And that times 3 (3 suricata enabled interfaces) I will look into more efficient rule enabling. Thanks!
Custom Rules and various lists such as SID management conf files are stored in the
config.xmlas Base64-encoded ASCII text within the applicable element tag. Those can get long if you have tons and tons of those kinds of entries. As originally envisoned and coded, the idea was users would have maybe only a dozen or couple of dozen custom rules max. Nothing inherently wrong with having more, but it will result in a much bigger XML element entry in theconfig.xmlfile. -
Thanks for the explanation.
The problems I got after enabling a 4th Suricata interface was something like this:
https://forum.netgate.com/topic/156679/pfsense-fatal-error-allowed-memory-exhausted-cause
Only able to restore after manual config file replacement and reboot.This was on 2.4 with Suricata 2.0.5 or something. I haven't dared to try it on 2.5/2.0.6 yet but perhaps I should open a seperate topic for that as it's out of scope for this topic's subject.
Spoiler alert: I have 56 thousand custom rules per interface. It's urlhaus blocklist which I haven't updated in a while it seems. If i update now, it would up it to 84k rules :) -
@ballistic said in Suricata won't stop:
Thanks for the explanation.
The problems I got after enabling a 4th Suricata interface was something like this:
https://forum.netgate.com/topic/156679/pfsense-fatal-error-allowed-memory-exhausted-cause
Only able to restore after manual config file replacement and reboot.This was on 2.4 with Suricata 2.0.5 or something. I haven't dared to try it on 2.5/2.0.6 yet but perhaps I should open a seperate topic for that as it's out of scope for this topic's subject.
Spoiler alert: I have 56 thousand custom rules per interface. It's urlhaus blocklist which I haven't updated in a while it seems. If i update now, it would up it to 84k rules :)That is an absurd number of rules to be honest. Are they just IP addresses? If from URLHaus, then I suspect they are actually simple lists of IP addresses/networks to block translated into Suricata/Snort rules syntax. Something like that would be more efficient when used as an IP list in pfBlockerNG-devel or else simply loading a URL table alias in pfSense. Because almost 100% of email traffic today is TLS (and thus encrypted), I can't imagine those rules examining any actual content (meaning data in the packet payloads). That is unless you are doing full MITM proxying of email traffic.
-
It's this list;
https://urlhaus.abuse.ch/downloads/suricata-ids/ -
@ballistic said in Suricata won't stop:
It's this list;
https://urlhaus.abuse.ch/downloads/suricata-ids/Did you know that there is now an option under the GLOBAL SETTINGS tab to add your own additional rules URLs? User @viktor_g here on the forums added that new feature a few months ago.
So you could simply copy this URL on that tab as an "additional rules" entry and then Suricata will download the list each time it updates the rules. You would not need to copy all of that text into the Custom Rules dialog, and your
config.xmlwould be considerably smaller as well (since it would no longer need all that Base64-encoded text). -
I could hug you sir! :)
Correct URL in this case is https://urlhaus.abuse.ch/downloads/urlhaus_suricata.tar.gz
-
@ballistic said in Suricata won't stop:
I could hug you sir! :)
Correct URL in this case is https://urlhaus.abuse.ch/downloads/urlhaus_suricata.tar.gz
Yes, I was about to post that I've never seen rules lists like that without also having some kind of zip or tar archive available as well...
.Oh, and after examining some of those rules, I see that they are doing content matching on the URI (so the only unencrypted part of https traffic). So not looking at the actual content as I originally said, but looking at the URI instead of just the IP. So these rules would not work in a typical alias.
-
Ok it took a few days but it stopped again after enabling updates.
GUI currently states stopped.
Process is actually running:
root 59462 110.9 21.3 1939960 1774988 - SNs Thu00 590:16.49 /usr/local/bin/suricata -i vtnet0.100 -D -c /usr/local/etc/suricata/suricata_24829_vtnet0.100/suricata.yaml --pidfile /var/run/suricata_vtnet0.10024829.pidPID file is not there;
[2.5.2-RELEASE][admin@thuis]/root: ls -al /var/run |grep suri
-rw-r--r-- 1 root wheel 6 Feb 11 00:03 suricata_vtnet0.10133180.pid
-rw-r--r-- 1 root wheel 6 Feb 11 00:03 suricata_vtnet0.20053803.pidI have now enabled "Live Rule Swap on Update" See how that goes.
-
Will you please post the entire contents of this file:
/usr/local/etc/rc.d/suricata.sh? I want to see how the interfaces are named in there, because I see you are running VLANs.And just to clarify, is blocking currently disabled?
But if enabled, which type are you using: Legacy Mode or Inline IPS Mode?
-
I have 2 pretty much identical configurated machines. 1 one bare metal (Xeon E-2236, 16GB) which does not experience any kinds of these issues. The one we are have been talking about is a VM on a Proxmox node (i5-8259U, 16GB. 8GB for PFsense)
Because everything was stable, I re-enabled blocking Legacy and it was still stable. Now after enabling the Updates, the problem came back within a day. (1d update interval)#!/bin/sh ######## # This file was automatically generated # by the pfSense service handler. ######## Start of main suricata.sh rc_start() { ### Make sure libraries path cache is up2date /etc/rc.d/ldconfig start ### Lock out other start signals until we are done /usr/bin/touch /var/run/suricata_pkg_starting.lck ## Start suricata on SECUREWIFI (vtnet0.101) ## if [ ! -f /var/run/suricata_vtnet0.10133180.pid ]; then pid=`/bin/pgrep -fn "suricata -i vtnet0.101 -D -c /usr/local/etc/suricata/suricata_33180_vtnet0.101/suricata.yaml "` else pid=`/bin/pgrep -F /var/run/suricata_vtnet0.10133180.pid` fi if [ -z $pid ]; then /bin/cp /dev/null /var/log/suricata/suricata_vtnet0.10133180/suricata.log /usr/bin/logger -p daemon.info -i -t SuricataStartup "Suricata START for SECUREWIFI(33180_vtnet0.101)..." /usr/local/bin/suricata -i vtnet0.101 -D -c /usr/local/etc/suricata/suricata_33180_vtnet0.101/suricata.yaml --pidfile /var/run/suricata_vtnet0.10133180.pid > /dev/null 2>&1 fi sleep 1 ## Start suricata on UNSECUREWIFI (vtnet0.200) ## if [ ! -f /var/run/suricata_vtnet0.20053803.pid ]; then pid=`/bin/pgrep -fn "suricata -i vtnet0.200 -D -c /usr/local/etc/suricata/suricata_53803_vtnet0.200/suricata.yaml "` else pid=`/bin/pgrep -F /var/run/suricata_vtnet0.20053803.pid` fi if [ -z $pid ]; then /bin/cp /dev/null /var/log/suricata/suricata_vtnet0.20053803/suricata.log /usr/bin/logger -p daemon.info -i -t SuricataStartup "Suricata START for UNSECUREWIFI(53803_vtnet0.200)..." /usr/local/bin/suricata -i vtnet0.200 -D -c /usr/local/etc/suricata/suricata_53803_vtnet0.200/suricata.yaml --pidfile /var/run/suricata_vtnet0.20053803.pid > /dev/null 2>&1 fi sleep 1 ## Start suricata on WIRED (vtnet0.100) ## if [ ! -f /var/run/suricata_vtnet0.10024829.pid ]; then pid=`/bin/pgrep -fn "suricata -i vtnet0.100 -D -c /usr/local/etc/suricata/suricata_24829_vtnet0.100/suricata.yaml "` else pid=`/bin/pgrep -F /var/run/suricata_vtnet0.10024829.pid` fi if [ -z $pid ]; then /bin/cp /dev/null /var/log/suricata/suricata_vtnet0.10024829/suricata.log /usr/bin/logger -p daemon.info -i -t SuricataStartup "Suricata START for WIRED(24829_vtnet0.100)..." /usr/local/bin/suricata -i vtnet0.100 -D -c /usr/local/etc/suricata/suricata_24829_vtnet0.100/suricata.yaml --pidfile /var/run/suricata_vtnet0.10024829.pid > /dev/null 2>&1 fi sleep 1 ### Remove the lock since we have started all interfaces if [ -f /var/run/suricata_pkg_starting.lck ]; then /bin/rm /var/run/suricata_pkg_starting.lck fi } rc_stop() { if [ -f /var/run/suricata_vtnet0.10133180.pid ]; then pid=`/bin/pgrep -F /var/run/suricata_vtnet0.10133180.pid` /usr/bin/logger -p daemon.info -i -t SuricataStartup "Suricata STOP for SECUREWIFI(33180_vtnet0.101)..." /bin/pkill -TERM -F /var/run/suricata_vtnet0.10133180.pid time=0 timeout=30 while /bin/kill -TERM $pid 2>/dev/null; do sleep 1 time=$((time+1)) if [ $time -gt $timeout ]; then break fi done if [ -f /var/run/suricata_vtnet0.10133180.pid ]; then /bin/rm /var/run/suricata_vtnet0.10133180.pid fi else pid=`/bin/pgrep -fn "suricata -i vtnet0.101 -D -c /usr/local/etc/suricata/suricata_33180_vtnet0.101/suricata.yaml "` if [ ! -z $pid ]; then /usr/bin/logger -p daemon.info -i -t SuricataStartup "Suricata STOP for SECUREWIFI(33180_vtnet0.101)..." /bin/pkill -TERM -fn "suricata -i vtnet0.101 " time=0 timeout=30 while /bin/kill -TERM $pid 2>/dev/null; do sleep 1 time=$((time+1)) if [ $time -gt $timeout ]; then break fi done fi fi sleep 1 if [ -f /var/run/suricata_vtnet0.20053803.pid ]; then pid=`/bin/pgrep -F /var/run/suricata_vtnet0.20053803.pid` /usr/bin/logger -p daemon.info -i -t SuricataStartup "Suricata STOP for UNSECUREWIFI(53803_vtnet0.200)..." /bin/pkill -TERM -F /var/run/suricata_vtnet0.20053803.pid time=0 timeout=30 while /bin/kill -TERM $pid 2>/dev/null; do sleep 1 time=$((time+1)) if [ $time -gt $timeout ]; then break fi done if [ -f /var/run/suricata_vtnet0.20053803.pid ]; then /bin/rm /var/run/suricata_vtnet0.20053803.pid fi else pid=`/bin/pgrep -fn "suricata -i vtnet0.200 -D -c /usr/local/etc/suricata/suricata_53803_vtnet0.200/suricata.yaml "` if [ ! -z $pid ]; then /usr/bin/logger -p daemon.info -i -t SuricataStartup "Suricata STOP for UNSECUREWIFI(53803_vtnet0.200)..." /bin/pkill -TERM -fn "suricata -i vtnet0.200 " time=0 timeout=30 while /bin/kill -TERM $pid 2>/dev/null; do sleep 1 time=$((time+1)) if [ $time -gt $timeout ]; then break fi done fi fi sleep 1 if [ -f /var/run/suricata_vtnet0.10024829.pid ]; then pid=`/bin/pgrep -F /var/run/suricata_vtnet0.10024829.pid` /usr/bin/logger -p daemon.info -i -t SuricataStartup "Suricata STOP for WIRED(24829_vtnet0.100)..." /bin/pkill -TERM -F /var/run/suricata_vtnet0.10024829.pid time=0 timeout=30 while /bin/kill -TERM $pid 2>/dev/null; do sleep 1 time=$((time+1)) if [ $time -gt $timeout ]; then break fi done if [ -f /var/run/suricata_vtnet0.10024829.pid ]; then /bin/rm /var/run/suricata_vtnet0.10024829.pid fi else pid=`/bin/pgrep -fn "suricata -i vtnet0.100 -D -c /usr/local/etc/suricata/suricata_24829_vtnet0.100/suricata.yaml "` if [ ! -z $pid ]; then /usr/bin/logger -p daemon.info -i -t SuricataStartup "Suricata STOP for WIRED(24829_vtnet0.100)..." /bin/pkill -TERM -fn "suricata -i vtnet0.100 " time=0 timeout=30 while /bin/kill -TERM $pid 2>/dev/null; do sleep 1 time=$((time+1)) if [ $time -gt $timeout ]; then break fi done fi fi sleep 1 } case $1 in start) if [ ! -f /var/run/suricata_pkg_starting.lck ]; then rc_start else /usr/bin/logger -p daemon.info -i -t SuricataStartup "Ignoring additional START command since Suricata is already starting..." fi ;; stop) rc_stop ;; restart) rc_stop sleep 5 rc_start ;; esac -
@ballistic said in Suricata won't stop:
I have 2 pretty much identical configurated machines. 1 one bare metal (Xeon E-2236, 16GB) which does not experience any kinds of these issues. The one we are have been talking about is a VM on a Proxmox node (i5-8259U, 16GB. 8GB for PFsense)
Oh, that certainly changes the possible causes. In my mind this strongly points the finger at something Proxmox related. If you search here on the Netgate forum you will find a fair number of posts related to various issues with running pfSense on Proxmox. Has this just started and otherwise the Proxmox instances have been good, or is the Proxmox install new?
The shell script you posted looks fine and contains what I expected to see. The script works by searching for the running Suricata process (one section in the file for each configured interface) and sending it a TERM command. It searches two ways. First it looks for the PID file, and if found kills the process by PID using the process ID read from the file in
/var/run. If it can't find the PID file, then it searches usingpkillwith some of the command line arguments passed when that Suricata instance was started in an attempt to be sure any matching running process is stopped. In either case, it then waits up to 30 seconds in a loop for the signalled Suricata process to stop and remove its PID file from/var/run. After that 30 seconds, the PID file is forcibly removed if it still exists. This is necessary because if that PID file exists, then Suricata will not start up. It will log an error about a "stale PID file". -
Ok disregard last about it not happening on bare metal. It did last night.
In GUI, Suricata is down for the interface, but process is running.
No PID file in /var/runThe proxmox Pfsense node has remained stable so the Live-reload rules might have helped. I will also enable it on the bare metal machine.
