Disk usage keeps building

James 0

du -d 1 -h /var/log

16K /var/log/nginx
4.0K /var/log/ntp
2.8G /var/log/suricata
2.8G /var/log

du -d 1 -h /var/log/suricata

2.8G /var/log/suricata/suricata_mvneta0.40908369
2.8G /var/log/suricata

Rico

Quite clearly suricata is eating 2.8G of your space.

-Rico

James 0

So do what do I do with surcata? Do I just live with this logger and start over every year?

Gertjan

@james-0 said in Disk usage keeps building:

So do what do I do with surcata? Do I just live with this logger and start over every year?

Suricata is not that package that you install, and then leave it all by itself : you have to keep (regularly !!) checking it.
What you check : the log file(s). When your done with them, and your disk space is limited, your remove them, or you delete them.

I thought that Suricate could do some log (size) handling by itself : see, for example : https://forum.netgate.com/topic/149695/suricata-error-php-fatal-error-allowed-memory-size-of-536870912-bytes-exhausted-tried-to-allocate-540538808-bytes-in-usr-local-www-suricata-suricata_logs_browser-php-on-line-54?_=1607354015304

Btw : I'm even have my pfSense free disk space being watched - and I'll receive a mail if less then 15 % is left.

Suricata on a SG-1100 : I'm impressed.

SteveITS

@james-0 Suricata had an issue a few years ago (give or take) where the log management tab showed log management was enabled after installation, but it wasn't actually working by default. Try saving the log management settings and see if it prunes its logs.

edit:
https://forum.netgate.com/topic/137652/suricata-suricata-log-not-rotated
https://forum.netgate.com/topic/140951/suricata-log-files-are-filling-the-disk
https://forum.netgate.com/topic/130980/suricata-not-limiting-log-sizes-by-default

James 0

Thank you all for your comments and suggestions. I did a clear on the alerts and blocked which didn't have anything in them anyways. The only log I see is Suricata log which only has 27 lines.

Could I uninstall Suricata to delete the all the logs to reduce the disk usage?

SteveITS

@james-0 said in Disk usage keeps building:

Could I uninstall Suricata to delete the all the logs to reduce the disk usage?

Did you save the log management settings as I suggested? Try checking "Log Directory Size Limit" as well.

Uninstall will work if you check "Remove Suricata Logs On Package Uninstall" on the log management tab.

James 0

I went to Logs Mgmt and clicked on Enable Directory Size Limit and saved. I did that about two hours ago and the disk usage size still has not changed which it is at 63%.

bmeeks

@james-0 said in Disk usage keeps building:

I went to Logs Mgmt and clicked on Enable Directory Size Limit and saved. I did that about two hours ago and the disk usage size still has not changed which it is at 63%.

What Suricata package version are you running? As mentioned by another poster, there were some issues with the automatic log managment several versions back, but I fixed those (or thought I did ... ).

There are sub-directories within /var/log/suricata for each configured interface. In one of those sub-directories is where you will find your large file or files. Post a listing back here of every sub-directory you find underneath /var/log/suricata and the contents of each. That will help me determine if there still may be a log management issue.

The log management process is a cron task that runs every 5 minutes if I recall correctly. It prunes the logs based on settings configured on the LOGS MGMT tab of the GUI. Of course the first thing you must do is enable automatic log file management by clicking the Enable checkbox on the LOG MGMT tab and then save that change. Automatic log management is disabled by default because some folks take offense to the system automatically removing log files without the specific consent of the admin.

James 0

I have installed version 5.0.4 which I just updated yesterday morning.

du -d 1 -h /var/log/suricata/

2.8G /var/log/suricata/suricata_mvneta0.40918369
2.8G /var/log/suricata/

I had an PHP error that did come up at some point. I attached it I hope.

[0_1607436728423_PHP_errors.log](Uploading 0%)

James 0

I guess the PHP error did not upload. This is the content of the error.

[07-Dec-2020 10:06:33 America/New_York] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 2964803200 bytes) in /usr/local/www/suricata/suricata_logs_browser.php on line 54

James 0

I did find a command that would show me more detail of the folders.

ls -lha /var/log/suricata/suricata_mvneta0.40918369

total 5871712
drwxr-xr-x 2 root wheel 512B Dec 7 10:15 .
drwx------ 3 root wheel 512B Dec 7 10:54 ..
-rw-r--r-- 1 root wheel 0B Dec 8 09:25 alerts.log
-rw-r--r-- 1 root wheel 2.8G Dec 7 08:43 alerts.log.2020_1207_1015
-rw-r--r-- 1 root wheel 0B Dec 8 09:25 http.log
-rw-r--r-- 1 root wheel 39M Dec 7 10:14 http.log.2020_1207_1015
-rw-r--r-- 1 root wheel 6.2K Dec 8 08:40 suricata.log

I see that the alerts log is 2.8G but when I go to Services, Suricata and click on Alerts it is empty.

bmeeks

@james-0 said in Disk usage keeps building:

I guess the PHP error did not upload. This is the content of the error.

[07-Dec-2020 10:06:33 America/New_York] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 2964803200 bytes) in /usr/local/www/suricata/suricata_logs_browser.php on line 54

This is an expected error when trying to read huge log files. There is not enough PHP system memory to hold the data. The PHP code tries to read the whole file into memory and then display it.

bmeeks

@james-0 said in Disk usage keeps building:

I did find a command that would show me more detail of the folders.

ls -lha /var/log/suricata/suricata_mvneta0.40918369

total 5871712
drwxr-xr-x 2 root wheel 512B Dec 7 10:15 .
drwx------ 3 root wheel 512B Dec 7 10:54 ..
-rw-r--r-- 1 root wheel 0B Dec 8 09:25 alerts.log
-rw-r--r-- 1 root wheel 2.8G Dec 7 08:43 alerts.log.2020_1207_1015
-rw-r--r-- 1 root wheel 0B Dec 8 09:25 http.log
-rw-r--r-- 1 root wheel 39M Dec 7 10:14 http.log.2020_1207_1015
-rw-r--r-- 1 root wheel 6.2K Dec 8 08:40 suricata.log

I see that the alerts log is 2.8G but when I go to Services, Suricata and click on Alerts it is empty.

This indicates that the Log Rotation process itself worked (notice the active alerts.log file was renamed to alerts.log.2020_1207_1015) and a new empty alerts.log file was created. The next phase of logs management will kick in when the renamed file ages out (in other words, it's last write date goes beyond the retention period you selected on the LOGS MGMT tab for alerts).

If you want the space back immediately, simply delete that large 2.8 GB file.

James 0

Thank you all. Before your updated comment I went to Services/Suricata/logs Mgmt and made sure Remove Suricata Logs On Package Uninstall was check. I then uninstalled Suricata and after, reinstalled it. It looks like all my settings came back and the large file logs were gone which now puts me to 19%.

I will keep an eye on this for a while.

Thank you again for all your help. I am learning a few things.

bmeeks

@james-0 said in Disk usage keeps building:

Thank you all. Before your updated comment I went to Services/Suricata/logs Mgmt and made sure Remove Suricata Logs On Package Uninstall was check. I then uninstalled Suricata and after, reinstalled it. It looks like all my settings came back and the large file logs were gone which now puts me to 19%.

I will keep an eye on this for a while.

Thank you again for all your help. I am learning a few things.

Glad you solved your issue. But please keep an eye on the log usage in that directory and post back here if the usage gets beyond the limits you configured on the LOGS MGMT tab. There are settings for how large the files can get before being rotated, and a setting for retention time that determines how long rotated files are kept on disk before they are deleted.

The other limit available on that tab up at the top sets a limit on the overall logging directory size (including the interface sub-directories contained within). That limit is configurable as a specific value set by the user, or it will automatically default to a percentage (20% or so I think it is) of disk space. However, no log file management of any type happens until the Enable checkbox is clicked on the LOGS MGMT tab and the change saved.