Suricata log files are filling the disk.



  • If using the firewall itself as a logkeeper for Suricata it will fill the disk very fast and this will lead to a reboot of the firewall.
    While rebooting it will fail to use last configuration and you will need to manually restore to last working configuration after cleaning up in /var/log/suricata/suricata_em063194.
    Also if uninstalling the Suricata package it will still hogg disk. You need to manually kill the Suricata process.
    This may relate to bug #9188.



  • @assar said in Suricata log files are filling the disk.:

    If using the firewall itself as a logkeeper for Suricata it will fill the disk very fast and this will lead to a reboot of the firewall.
    While rebooting it will fail to use last configuration and you will need to manually restore to last working configuration after cleaning up in /var/log/suricata/suricata_em063194.
    Also if uninstalling the Suricata package it will still hogg disk. You need to manually kill the Suricata process.
    This may relate to bug #9188.

    How much free disk space do you have in /var/log and what values have you configured for Log Limits on the LOGS MGMT tab?

    Removing the Suricata package will not, by default, clear out existing log files. If you want that to occur, it must be configured on the LOGS MGMT tab.



  • This time (second) i saw Suricata logs eat 40% of a 500GB disk in three hours and were able to mitigate the problem before the firewall got unmanageable.
    This is the directory per 11:10 yesterday:

    [2.4.4-RELEASE][root@firewall.xxxxxxxx.se]/var/log/suricata: ls -l suricata_em063194/
    total 188226260
    -rw-r--r--  1 root  wheel  152372320173 Feb 25 11:10 alerts.log
    -rw-r--r--  1 root  wheel   14209253376 Feb 25 11:10 alerts.log.2019_0225_1025
    -rw-r--r--  1 root  wheel    7778467840 Feb 25 11:10 alerts.log.2019_0225_1030
    -rw-r--r--  1 root  wheel    5460721664 Feb 25 11:10 alerts.log.2019_0225_1035
    -rw-r--r--  1 root  wheel    4129554432 Feb 25 11:10 alerts.log.2019_0225_1040
    -rw-r--r--  1 root  wheel    3220439040 Feb 25 11:10 alerts.log.2019_0225_1045
    -rw-r--r--  1 root  wheel    2317066240 Feb 25 11:10 alerts.log.2019_0225_1050
    -rw-r--r--  1 root  wheel    1527250944 Feb 25 11:10 alerts.log.2019_0225_1055
    -rw-r--r--  1 root  wheel     947912704 Feb 25 11:10 alerts.log.2019_0225_1100
    -rw-r--r--  1 root  wheel     455081984 Feb 25 11:10 alerts.log.2019_0225_1105
    -rw-r--r--  1 root  wheel      25198592 Feb 25 11:10 alerts.log.2019_0225_1110
    -rw-r--r--  1 root  wheel     252727552 Feb 25 11:10 http.log
    -rw-r--r--  1 root  wheel         52029 Feb 25 00:30 suricata.log
    

    And this is one and a half hours later

    [2.4.4-RELEASE][root@firewall.xxxxxxxxx.se]/var/log/suricata: ls -l suricata_em063194/
    total 262695156
    -rw-r--r--  1 root  wheel  152394298743 Feb 25 12:43 alerts.log
    -rw-r--r--  1 root  wheel   18432786432 Feb 25 12:43 alerts.log.2019_0225_1025
    -rw-r--r--  1 root  wheel   12073697280 Feb 25 12:43 alerts.log.2019_0225_1030
    -rw-r--r--  1 root  wheel    9830006784 Feb 25 12:43 alerts.log.2019_0225_1035
    -rw-r--r--  1 root  wheel    8571977728 Feb 25 12:43 alerts.log.2019_0225_1040
    -rw-r--r--  1 root  wheel    7684489216 Feb 25 12:43 alerts.log.2019_0225_1045
    -rw-r--r--  1 root  wheel    6776946688 Feb 25 12:43 alerts.log.2019_0225_1050
    -rw-r--r--  1 root  wheel    6002442240 Feb 25 12:43 alerts.log.2019_0225_1055
    -rw-r--r--  1 root  wheel    5494800384 Feb 25 12:43 alerts.log.2019_0225_1100
    -rw-r--r--  1 root  wheel    4957274112 Feb 25 12:43 alerts.log.2019_0225_1105
    -rw-r--r--  1 root  wheel    4543348736 Feb 25 12:43 alerts.log.2019_0225_1110
    -rw-r--r--  1 root  wheel    4116185088 Feb 25 12:43 alerts.log.2019_0225_1115
    -rw-r--r--  1 root  wheel    3785490432 Feb 25 12:43 alerts.log.2019_0225_1120
    -rw-r--r--  1 root  wheel    3376807936 Feb 25 12:43 alerts.log.2019_0225_1125
    -rw-r--r--  1 root  wheel    3025272832 Feb 25 12:43 alerts.log.2019_0225_1130
    -rw-r--r--  1 root  wheel    2754871296 Feb 25 12:43 alerts.log.2019_0225_1135
    -rw-r--r--  1 root  wheel    2476081152 Feb 25 12:43 alerts.log.2019_0225_1140
    -rw-r--r--  1 root  wheel    2220752896 Feb 25 12:43 alerts.log.2019_0225_1145
    -rw-r--r--  1 root  wheel    1936719872 Feb 25 12:43 alerts.log.2019_0225_1150
    -rw-r--r--  1 root  wheel    1710882816 Feb 25 12:43 alerts.log.2019_0225_1155
    -rw-r--r--  1 root  wheel    1463156736 Feb 25 12:43 alerts.log.2019_0225_1200
    -rw-r--r--  1 root  wheel    1263271936 Feb 25 12:43 alerts.log.2019_0225_1205
    -rw-r--r--  1 root  wheel    1049559040 Feb 25 12:43 alerts.log.2019_0225_1210
    -rw-r--r--  1 root  wheel     868089856 Feb 25 12:43 alerts.log.2019_0225_1215
    -rw-r--r--  1 root  wheel     681050112 Feb 25 12:43 alerts.log.2019_0225_1220
    -rw-r--r--  1 root  wheel     537919488 Feb 25 12:43 alerts.log.2019_0225_1225
    -rw-r--r--  1 root  wheel     351846400 Feb 25 12:43 alerts.log.2019_0225_1230
    -rw-r--r--  1 root  wheel     222740480 Feb 25 12:43 alerts.log.2019_0225_1235
    -rw-r--r--  1 root  wheel      76840960 Feb 25 12:43 alerts.log.2019_0225_1240
    -rw-r--r--  1 root  wheel     253081174 Feb 25 12:43 http.log
    -rw-r--r--  1 root  wheel         52029 Feb 25 00:30 suricata.log
    


  • Settings were to save logs for 14 days and size were set to 50MB



  • And when I removed the package the whole /var/log/suricata directory were removed. But since the multiple processes were still running the disk kept running out until they were manually killed. Then i got back to 0% disk usage.

    Before removal:

    Filesystem                  1K-blocks      Used     Avail Capacity  Mounted on
    /dev/ufsid/5609536a6f6d18d7 464921956 272469992 155258208    64%    /
    devfs                               1         1         0   100%    /dev
    /dev/md0                         3484       120      3088     4%    /var/run
    devfs                               1         1         0   100%    /var/dhcpd/dev
    

    After removal:

    Filesystem                  1K-blocks    Used     Avail Capacity  Mounted on
    /dev/ufsid/5609536a6f6d18d7 464921956 1117684 426610516     0%    /
    devfs                               1       1         0   100%    /dev
    /dev/md0                         3484     120      3088     4%    /var/run
    devfs                               1       1         0   100%    /var/dhcpd/dev
    


  • What version of the Suricata package are you running? Is it 4.1.2_3, or something older? The last update of the package fixed a log rotation issue. The size of the suricata.log file is also curious to me. With the latest package version, that file is overwritten with each restart of Suricata. So I would not expect it to be 52K in size (I would expect more like only 1/10th of that size).



  • @bmeeks
    It were quite recently installed and I did peek if there were any updates when this were discovered.
    It's not installed at the moment but the last update seems to have been Jan 18 and that's more than a month ago.
    Since the Packet Manager is used, it ought to be the 4.1.2_3 and nothing older.



  • Contrary to my last answer.
    As time flies away, it might have been before Jan 18 so this issue may be fixed.
    I'll test again later.


Log in to reply