Suricata SIGHUP every 5 minutes
-
I'm seeing a flood of SIGHUP in the General -> System log exactly every 5 minutes. Not sure what is going on.
Netgate SG-5100
2.4.5-RELEASE (amd64)
built on Tue Mar 24 15:25:53 EDT 2020
FreeBSD 11.3-STABLESuricata package version 5.0.4_2
Blocking Mode = Legacy
Using Snort + ET rulesThe only recent changes I made were enablement of SID Mgmt adding DROP SID list for all ET-Known-Compromised; and upgrading suricata package to latest available.
Excerpt from my system logs:
Jan 31 11:50:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 11:55:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 11:55:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 12:00:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 12:00:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 12:05:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 12:05:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 12:10:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 12:10:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 12:15:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 12:15:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 12:20:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 12:20:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 12:25:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 12:25:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 12:30:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 12:30:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 12:30:27 php-cgi [Suricata] Emerging Threats Open rules are up to date... Jan 31 12:30:28 php-cgi [Suricata] Snort VRT rules are up to date... Jan 31 12:30:28 php-cgi [Suricata] The Rules update has finished. Jan 31 12:35:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 12:35:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 12:40:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 12:40:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 12:45:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 12:45:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 12:50:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 12:50:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 12:55:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 12:55:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 13:00:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 13:00:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 13:05:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 13:05:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 13:10:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 13:10:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 13:15:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 13:15:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 13:20:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 13:20:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 13:21:41 check_reload_status Syncing firewall Jan 31 13:21:41 php-fpm 16860 /suricata/suricata_interfaces_edit.php: Beginning configuration backup to https://acb.netgate.com/save Jan 31 13:21:46 php-fpm 16860 /suricata/suricata_interfaces_edit.php: End of configuration backup to https://acb.netgate.com/save (success). Jan 31 13:25:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 13:25:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 13:30:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 13:30:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ... Jan 31 13:35:00 php-cgi [Suricata] Suricata signalled with SIGHUP for WAN (igb0)... Jan 31 13:35:00 php-cgi [Suricata] Logs Mgmt job rotated 1 file(s) in '/var/log/suricata/suricata_igb061308/' ...Any ideas?
-
That probably means some log file is rapidly filling up and getting rotated. Suricata is sent a SIGHUP message each time a log file is rotated. This is so Suricata will stop writing to the old rotated log and begin writing to the new empty one.
Look on the LOGS MGMT tab and see if you have a "busy" log set to a very low size limit. This would trigger rapid rotating of that log.
The cron task that checks for logs needing rotation runs every 5 minutes, thus the 5-minute interval you are seeing.
-
@bmeeks, If I had to guess it's the EVE.JSON file which i'm ingesting into ELK for dashboards. In the Log Size and Retention Limits configuration the max size is 10MB, but I currently have a file in there "eve.json" that is 1.2GB
-
@terminalhit said in Suricata SIGHUP every 5 minutes:
@bmeeks, If I had to guess it's the EVE.JSON file which i'm ingesting into ELK for dashboards. In the Log Size and Retention Limits configuration the max size is 10MB, but I currently have a file in there "eve.json" that is 1.2GB
The idea is for the log to rotate and get a new name with a UNIX timestamp appended to it. Then a new empty log file is opened for Suricata. The SIGHUP is supposed to tell Suricata to reopen log files. Unfortunately, the Suricata binary can only rotate certain logs natively. So without the GUI attempting to rotate the others, they will grow to impossibly large sizes.
Do you have any eve.json logs that have a UNIX timestamp on the end? If not, the log rotation is not actually working. That would be why it keeps trying each time the cron task runs (every 5 minutes).
You might have a duplicate Suricata zombie process attempting to use the log file. If you can, stop Suricata on the interface for more than 5 minutes. This will allow the cron task to run and hopefully rotate that huge file. Then restart Suricata on the interface. If stopping Suricata for more than 5 minutes does not result in the file rotating, then manually rename it yourself (the big 1.2 GB file) to something else and then restart Suricata.
