Suricata - interface show the service as stopped after sometime.

bmeeks

@tchadrack said in Suricata - interface show the service as stopped after sometime.:

Thankyou. I believe more people could be facing the same issue. So anyone here could help us to know what is happening?

Obs: I tried to update Suricata before, but it didnt help

Complete start up log of suricata:
[2.7.2-RELEASE][admin@*****.*******.lan]/root: ps aux | grep surica
root    66156   0.0  2.0  572652  123832  -  SNs  18:46       3:40.11 /usr/local/bin/suricata -i re0 -D -c /usr/local/etc/suricata/suricata_58258_re
root    99211   0.0  2.4  586480  143904  -  SNs  18:46       4:20.86 /usr/local/bin/suricata -i pppoe0 -D -c /usr/local/etc/suricata/suricata_30338
 

And immediately after this did you check in /var/run for any files with suricata in their name? If the binary starts, it should be creating the PID file in that directory. If that is failing, perhaps there is a permissions or even disk space issue ???

Realtek NICs are not the best for use on FreeBSD, but even still I don't see how that could be related to not creating a PID file.

Try this --

Kill all the running Suricata processes using this command from a shell prompt:

kill <PID>

Once all running Suricata processes are terminated, return to the GUI and go to the INTERFACES tab in Suricata and click the icon to start each interface. Report back on what happens then.

tchadrack

@bmeeks said in Suricata - interface show the service as stopped after sometime.:

And immediately after this did you check in /var/run for any files with suricata in their name? If the binary starts, it should be creating the PID file in that directory. If that is failing, perhaps there is a permissions or even disk space issue ???

I didnt, because the down status on web interface only happens after some time and not imediatelly. Sometimes, they can be up for more than one day, even more.

@bmeeks said in Suricata - interface show the service as stopped after sometime.:

Kill all the running Suricata processes using this command from a shell prompt: kill <PID>
Once all running Suricata processes are terminated, return to the GUI and go to the INTERFACES tab in Suricata and click the icon to start each interface. Report back on what happens

I tried to do it already. I've killed all suricata processes and started again from web interface.

What happens is that they start and work normally, showing themselves as up in the web interface. But after some time, when I open and look into web interface, they are showing as down.

If I go on the shell i can see they are up, but without the pid file on /var/run

 0) Logout (SSH only)                  9) pfTop
 1) Assign Interfaces                 10) Filter Logs
 2) Set interface(s) IP address       11) Restart webConfigurator
 3) Reset webConfigurator password    12) PHP shell + pfSense tools
 4) Reset to factory defaults         13) Update from console
 5) Reboot system                     14) Disable Secure Shell (sshd)
 6) Halt system                       15) Restore recent configuration
 7) Ping host                         16) Restart PHP-FPM
 8) Shell

Enter an option: 8

[2.7.2-RELEASE][admin@****************]/root: cd /var/run
[2.7.2-RELEASE][admin@****************]/var/run: ls | grep surica
[2.7.2-RELEASE][admin@****************]/var/run: ps aux | grep suric
root    66156   0.0  1.2  581868   74928  -  SNs  18:46       5:08.75 /usr/local/bin/suricata -i re0 -D
root    99211   0.0  1.6  586480   95276  -  SNs  18:46       6:00.33 /usr/local/bin/suricata -i pppoe0
root    49197   0.0  0.0   12752    2136  0  S+   16:49       0:00.00 grep suric
[2.7.2-RELEASE][admin@****************]/var/run: kill -9 66156
[2.7.2-RELEASE][admin@****************]/var/run: kill -9 99211
[2.7.2-RELEASE][admin@****************]/var/run: ps aux | grep suric
root    42936   0.0  0.0   12752    2140  0  S+   16:50       0:00.00 grep suric
(here I started suricata under the web interface)
[2.7.2-RELEASE][admin@****************]/var/run: ls | grep surica
suricata_pppoe030338.pid
suricata_re058258.pid

After killing the processes, restarting them from web interface, the pid files are created inside /var/run

bmeeks

Okay, let's try an experiment.

Go to the GLOBAL SETTINGS tab in Suricata and check the box for Live Rule Swap on Update (if it is not already checked). Save that change.

Next, kill the Suricata processes from the shell prompt as explained in my post above, then restart Suricata on each interface in the GUI. Let's see if that keeps the GUI's status in sync with the actual processes.

Something is deleting the PID files, but I currently do not really know what that is. And my guess is it happens during the periodic rules update when Suricata is normally restarted. Checking the box I noted prevents a physical restart of Suricata and instead tells it to live reload the updated rules.

The outcome of this experiment may give me a clue (or at least help me narrow down where to be looking).

tchadrack

@bmeeks Okay, i've done it. I will return later with more information about what happened. Thank you

bmeeks

@tchadrack said in Suricata - interface show the service as stopped after sometime.:

@bmeeks Okay, i've done it. I will return later with more information about what happened. Thank you

Okay. Also, I did not see it in your process list captures, but make sure you are NOT using the Service Watchdog package to monitor Suricata. That package is not compatible with either Suricata or Snort.

tchadrack

@bmeeks

root    78291   0.0  4.8  765872  291224  -  Ss   17:00       6:31.07 /usr/local/bin/suricata -i pppoe0
root    86453   0.0  5.3  872748  325348  -  Ss   17:00       6:37.23 /usr/local/bin/suricata -i re0 -D

Up to now the web interface is showing the suricata service as up and running (pid files present in /var/run)

Watchdog is running, but not enabled for suricata

bmeeks

@tchadrack said in Suricata - interface show the service as stopped after sometime.:

@bmeeks
root    78291   0.0  4.8  765872  291224  -  Ss   17:00       6:31.07 /usr/local/bin/suricata -i pppoe0
root    86453   0.0  5.3  872748  325348  -  Ss   17:00       6:37.23 /usr/local/bin/suricata -i re0 -D
Up to now the web interface is showing the suricata service as up and running (pid files present in /var/run)

Watchdog is running, but not enabled for suricata

Okay. I think I may know what could be happening. I notice you appear to have stats collection enabled based on the log entries I saw in the suricata.log snippet you posted. I'm suspecting your instance is either taking a long time to shutdown (due to collecting and printing the stats summary) or it is not actually fully shutting down during the rules update. But the PHP function that restarts Suricata after a rule update does unconditionally delete the PID file when stopping the binary to be sure the subsequent restart will succeed. I think I may need to make that code "smarter" so that it does more thorough checks before deleting the PID.

The change I had you make skips restarting the binary on rules updates. It instructs Suricata to reload the rules while still running. That way the PID file is not deleted. The only downside to that is a temporary increase in RAM consumption while the rules swap is happening. That can be a problem only with installs that have limited RAM.

tchadrack

@bmeeks Ok, Thankyou.

bmeeks

@tchadrack said in Suricata - interface show the service as stopped after sometime.:

@bmeeks Ok, Thankyou.

I also recall, quite some time back- maybe around the time Suricata upstream was still in the 2.x or 3.x version branch- that sometimes the binary would not respond to the first "shutdown" command issued. It would sometimes take two cycles of issuing shutdown to the process to get Suricata to honor the request. Perhaps that is happening here again in this particular instance. The one thing you appear to have enabled that is not normally enabled by the majority of users is the stats collection option. That may figure in here.

I will try and replicate this in my test virtual machine environment. I don't normally test with stats collection enabled, so I need to specifically test with that option enabled.

bmeeks

@tchadrack:
I was not able to replicate the exact behavior you are seeing, but I did find some potential edge-case scenarios where the PID file might get deleted prematurely.

I've submitted Pull Requests to address some issues in the package and also cleanup the Suricata binary startup and shutdown scripts. Look for a new 7.0.8_1 Suricata package version to appear in the near future containing these improvements.

The Pull Requests are waiting on the Netgate developer team to review and merge them into the pfSense branches for CE, DEVEL, and Plus. Here is the PR for the current CE Release: https://github.com/pfsense/FreeBSD-ports/pull/1404.

btspce

@bmeeks Just updated to 7.0.8_1. Thanks !

bmeeks

@btspce said in Suricata - interface show the service as stopped after sometime.:

@bmeeks Just updated to 7.0.8_1. Thanks !

I am cautiously optimistic this update helps with your issue. But since I could not directly reproduce the problem in my test environment, I had to proceed with a bit of guess as to what the cause and fix might be.

Please post back here later with an update.

tchadrack

@bmeeks Using the "Live Rule Swap on Update" option checked, resolves the problem. Suricata worked for several days, with no issue.

I updated to the new version of suricata, unchecked "Live Rule Swap on Update" to test and see what would happen.

Unfortunately, the problem continues, so I am still using the Live Rule Swap on Update, that is working very well.

tchadrack

@tchadrack

Thermal Sensors
Zone 1: 29.9 °C
Zone 0: 27.9 °C

Name ..**
User admin@192.168..** (Local Database)
System pfSense
Netgate Device ID: ******************
BIOS Vendor: American Megatrends Inc.
Version: F2
Release Date: Mon Oct 7 2013
Version 2.7.2-RELEASE (amd64)
built on Fri Dec 8 17:55:00 -03 2023
FreeBSD 14.0-CURRENT

The system is on the latest version.
Version information updated at Sat Feb 1 9:35:25 -03 2025
CPU Type Intel(R) Pentium(R) CPU G3220 @ 3.00GHz
2 CPUs: 1 package(s) x 2 core(s)
AES-NI CPU Crypto: No
QAT Crypto: No
Hardware crypto Inactive
Kernel PTI Enabled
MDS Mitigation Inactive
Uptime 1 Day 03 Hours 20 Minutes 04 Seconds
Current date/time
Sat Feb 1 9:44:46 -03 2025
DNS server(s)
8.8.8.8
8.8.4.4
Last config change Sat Feb 1 9:05:02 -03 2025
State table size
0% (222/1000000) Show states
MBUF Usage
5% (18856/371768)
Temperature
27.9°C
Load average
0.47, 0.48, 0.42
CPU usage
4%
Memory usage
53% of 5980 MiB
SWAP usage
17% of 3851 MiB

DISKS:
Mount Used Size Usage
/ 28G 447G
7% of 447G (ufs)

SERVICES:
arpwatch Arpwatch Daemon
bandwidthd BandwidthD bandwidth monitoring daemon
captiveportal Captive Portal: **********
darkstat Darkstat bandwidth monitoring daemon
dpinger Gateway Monitoring Daemon
kea-dhcp4 Kea DHCP Server
ntopng ntopng Network Traffic Monitor
ntpd NTP clock sync
openvpn OpenVPN server: *************
pfb_dnsbl pfBlockerNG DNSBL service
pfb_filter pfBlockerNG firewall filter service
radiusd FreeRADIUS Server
sshd Secure Shell Daemon
suricata Suricata IDS/IPS Daemon
syslogd System Logger Daemon
unbound DNS Resolver
vnstatd Status Traffic Totals data collection daemon

S.M.A.R.T. Status
Drive Ident S.M.A.R.T. Status
ada0 WD-************ PASSED