Suricata log files are filling the disk.
-
If using the firewall itself as a logkeeper for Suricata it will fill the disk very fast and this will lead to a reboot of the firewall.
While rebooting it will fail to use last configuration and you will need to manually restore to last working configuration after cleaning up in /var/log/suricata/suricata_em063194.
Also if uninstalling the Suricata package it will still hogg disk. You need to manually kill the Suricata process.
This may relate to bug #9188. -
@assar said in Suricata log files are filling the disk.:
If using the firewall itself as a logkeeper for Suricata it will fill the disk very fast and this will lead to a reboot of the firewall.
While rebooting it will fail to use last configuration and you will need to manually restore to last working configuration after cleaning up in /var/log/suricata/suricata_em063194.
Also if uninstalling the Suricata package it will still hogg disk. You need to manually kill the Suricata process.
This may relate to bug #9188.How much free disk space do you have in /var/log and what values have you configured for Log Limits on the LOGS MGMT tab?
Removing the Suricata package will not, by default, clear out existing log files. If you want that to occur, it must be configured on the LOGS MGMT tab.
-
This time (second) i saw Suricata logs eat 40% of a 500GB disk in three hours and were able to mitigate the problem before the firewall got unmanageable.
This is the directory per 11:10 yesterday:[2.4.4-RELEASE][root@firewall.xxxxxxxx.se]/var/log/suricata: ls -l suricata_em063194/ total 188226260 -rw-r--r-- 1 root wheel 152372320173 Feb 25 11:10 alerts.log -rw-r--r-- 1 root wheel 14209253376 Feb 25 11:10 alerts.log.2019_0225_1025 -rw-r--r-- 1 root wheel 7778467840 Feb 25 11:10 alerts.log.2019_0225_1030 -rw-r--r-- 1 root wheel 5460721664 Feb 25 11:10 alerts.log.2019_0225_1035 -rw-r--r-- 1 root wheel 4129554432 Feb 25 11:10 alerts.log.2019_0225_1040 -rw-r--r-- 1 root wheel 3220439040 Feb 25 11:10 alerts.log.2019_0225_1045 -rw-r--r-- 1 root wheel 2317066240 Feb 25 11:10 alerts.log.2019_0225_1050 -rw-r--r-- 1 root wheel 1527250944 Feb 25 11:10 alerts.log.2019_0225_1055 -rw-r--r-- 1 root wheel 947912704 Feb 25 11:10 alerts.log.2019_0225_1100 -rw-r--r-- 1 root wheel 455081984 Feb 25 11:10 alerts.log.2019_0225_1105 -rw-r--r-- 1 root wheel 25198592 Feb 25 11:10 alerts.log.2019_0225_1110 -rw-r--r-- 1 root wheel 252727552 Feb 25 11:10 http.log -rw-r--r-- 1 root wheel 52029 Feb 25 00:30 suricata.logAnd this is one and a half hours later
[2.4.4-RELEASE][root@firewall.xxxxxxxxx.se]/var/log/suricata: ls -l suricata_em063194/ total 262695156 -rw-r--r-- 1 root wheel 152394298743 Feb 25 12:43 alerts.log -rw-r--r-- 1 root wheel 18432786432 Feb 25 12:43 alerts.log.2019_0225_1025 -rw-r--r-- 1 root wheel 12073697280 Feb 25 12:43 alerts.log.2019_0225_1030 -rw-r--r-- 1 root wheel 9830006784 Feb 25 12:43 alerts.log.2019_0225_1035 -rw-r--r-- 1 root wheel 8571977728 Feb 25 12:43 alerts.log.2019_0225_1040 -rw-r--r-- 1 root wheel 7684489216 Feb 25 12:43 alerts.log.2019_0225_1045 -rw-r--r-- 1 root wheel 6776946688 Feb 25 12:43 alerts.log.2019_0225_1050 -rw-r--r-- 1 root wheel 6002442240 Feb 25 12:43 alerts.log.2019_0225_1055 -rw-r--r-- 1 root wheel 5494800384 Feb 25 12:43 alerts.log.2019_0225_1100 -rw-r--r-- 1 root wheel 4957274112 Feb 25 12:43 alerts.log.2019_0225_1105 -rw-r--r-- 1 root wheel 4543348736 Feb 25 12:43 alerts.log.2019_0225_1110 -rw-r--r-- 1 root wheel 4116185088 Feb 25 12:43 alerts.log.2019_0225_1115 -rw-r--r-- 1 root wheel 3785490432 Feb 25 12:43 alerts.log.2019_0225_1120 -rw-r--r-- 1 root wheel 3376807936 Feb 25 12:43 alerts.log.2019_0225_1125 -rw-r--r-- 1 root wheel 3025272832 Feb 25 12:43 alerts.log.2019_0225_1130 -rw-r--r-- 1 root wheel 2754871296 Feb 25 12:43 alerts.log.2019_0225_1135 -rw-r--r-- 1 root wheel 2476081152 Feb 25 12:43 alerts.log.2019_0225_1140 -rw-r--r-- 1 root wheel 2220752896 Feb 25 12:43 alerts.log.2019_0225_1145 -rw-r--r-- 1 root wheel 1936719872 Feb 25 12:43 alerts.log.2019_0225_1150 -rw-r--r-- 1 root wheel 1710882816 Feb 25 12:43 alerts.log.2019_0225_1155 -rw-r--r-- 1 root wheel 1463156736 Feb 25 12:43 alerts.log.2019_0225_1200 -rw-r--r-- 1 root wheel 1263271936 Feb 25 12:43 alerts.log.2019_0225_1205 -rw-r--r-- 1 root wheel 1049559040 Feb 25 12:43 alerts.log.2019_0225_1210 -rw-r--r-- 1 root wheel 868089856 Feb 25 12:43 alerts.log.2019_0225_1215 -rw-r--r-- 1 root wheel 681050112 Feb 25 12:43 alerts.log.2019_0225_1220 -rw-r--r-- 1 root wheel 537919488 Feb 25 12:43 alerts.log.2019_0225_1225 -rw-r--r-- 1 root wheel 351846400 Feb 25 12:43 alerts.log.2019_0225_1230 -rw-r--r-- 1 root wheel 222740480 Feb 25 12:43 alerts.log.2019_0225_1235 -rw-r--r-- 1 root wheel 76840960 Feb 25 12:43 alerts.log.2019_0225_1240 -rw-r--r-- 1 root wheel 253081174 Feb 25 12:43 http.log -rw-r--r-- 1 root wheel 52029 Feb 25 00:30 suricata.log -
Settings were to save logs for 14 days and size were set to 50MB
-
And when I removed the package the whole /var/log/suricata directory were removed. But since the multiple processes were still running the disk kept running out until they were manually killed. Then i got back to 0% disk usage.
Before removal:
Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/ufsid/5609536a6f6d18d7 464921956 272469992 155258208 64% / devfs 1 1 0 100% /dev /dev/md0 3484 120 3088 4% /var/run devfs 1 1 0 100% /var/dhcpd/devAfter removal:
Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/ufsid/5609536a6f6d18d7 464921956 1117684 426610516 0% / devfs 1 1 0 100% /dev /dev/md0 3484 120 3088 4% /var/run devfs 1 1 0 100% /var/dhcpd/dev -
What version of the Suricata package are you running? Is it 4.1.2_3, or something older? The last update of the package fixed a log rotation issue. The size of the
suricata.logfile is also curious to me. With the latest package version, that file is overwritten with each restart of Suricata. So I would not expect it to be 52K in size (I would expect more like only 1/10th of that size). -
@bmeeks
It were quite recently installed and I did peek if there were any updates when this were discovered.
It's not installed at the moment but the last update seems to have been Jan 18 and that's more than a month ago.
Since the Packet Manager is used, it ought to be the 4.1.2_3 and nothing older. -
Contrary to my last answer.
As time flies away, it might have been before Jan 18 so this issue may be fixed.
I'll test again later.
