Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds
-
Enable a log directory size limit as well as log sizes to be sure duplicate files don't fill the drive.
Steve
-
Hello,
I am also having this issue. My snort LAN interface logs are filling my disk. I have turned on Auto Log Management and changed the values and the issue persists. I have to go in and manually clean out the logs of the LAN interface. It generates multiple alert logs.
I literally just cleaned out this directory and it has generated all of those within 30 min.
It's rather annoying. Is there a fix for this ?Below is my current Log Mgmt settings:
-
What versions of pfSense and the Snort package are you running? Some issues with log rotation were corrected several versions back in the Snort package. To my knowledge, users are not reporting widespread issues with logs management on either Snort or Suricata in the current package versions.
There have been a few isolated cases that were caused by things the users had done (or more aptly, not done).
How much space do you have allocated for /var on your firewall? Are you by chance using a RAM disk? If so, don't do that with Snort!
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
What versions of pfSense and the Snort package are you running? Some issues with log rotation were corrected several versions back in the Snort package. To my knowledge, users are not reporting widespread issues with logs management on either Snort or Suricata in the current package versions.
There have been a few isolated cases that were caused by things the users had done (or more aptly, not done).
How much space do you have allocated for /var on your firewall? Are you by chance using a RAM disk? If so, don't do that with Snort!
Hello and thank you for the quick response. I am using PFsense 2.4.5 and Snort 3.2.9.10_4. I am not using a RAM disk. Where would i check how space I have allocated for /var? I don't think i have ever set anything...
-
The quickest way is to look on the pfSense Dashboard page down towards the bottom. It will show you the amount of used and allocated space for
/
(which is the root system volume). The/var/log/snort/
directory will be allocated from that space.I have an SG-5100 and currently it shows me using 25% of 6.7 GB. That 25% is all of my installed software plus logs (including Snort logs). I have Snort running on three interfaces.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
The quickest way is to look on the pfSense Dashboard page down towards the bottom. It will show you the amount of used and allocated space for
/
(which is the root system volume). The/var/log/snort/
directory will be allocated from that space.I have an SG-5100 and currently it shows me using 25% of 6.7 GB. That 25% is all of my installed software plus logs (including Snort logs). I have Snort running on three interfaces.
It shows 43GiB
-
If you have that much empty space on the system volume, then it sounds like the Snort log cleanup cron task is not executing, or else permissions are messed up somehow in the logging directory.
Are there any messages in the pfSense system log from Snort? The log cleanup job executes every 5 minutes if I remember correctly. So if your Snort logging path size currently exceeds the configured max value, that cron task should print some data about what it's doing to the system log every 5 minutes. Look and see if anything is being logged or if any error messages are showing up.
Look in the
/etc/crontab
file and see if thesnort_check_cron_misc.inc
file is showing in the list of executable jobs.I am running the same version of pfSense and Snort as you and my logging directory is being kept below the configured limit. I have no log files in the directory older than 14 days (my retention limit setting) and the total size in bytes is below the 1340 MB limit I have configured. So the Logs Management process is working for me.
-
How much traffic is going through Snort? Just noticed the timestamps on those alert logs. You are generating a new file every few seconds! Your limit per log file in the interface says 500 KB. How big are the actual alert log files on the disk? Are they also 500 KB?
You have something seriously wrong if your system is generating 500 KBs/few seconds of alert logs. Those numbers on the end of each log file are Unix timestamps, so you can look at how close together the values are and see that your system is creating new files every few seconds.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
snort_check_cron_misc.inc
ok. i checked that directory and i do not have snort_check_cron_misc.inc file. Idk what happened to it or if there was ever one.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
How much traffic is going through Snort? Just noticed the timestamps on those alert logs. You are generating a new file every few seconds! Your limit per log file in the interface says 500 KB. How big are the actual alert log files on the disk? Are they also 500 KB?
You have something seriously wrong if your system is generating 500 KBs/few seconds of alert logs. Those numbers on the end of each log file are Unix timestamps, so you can look at how close together the values are and see that your system is creating new files every few seconds.
This is the LAN interface logs so anytime someone browses or anything.
I have two Snort interfaces; WAN and LAN.
-
@cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
snort_check_cron_misc.inc
ok. i checked that directory and i do not have snort_check_cron_misc.inc file. Idk what happened to it or if there was ever one.
No, it won't be a file in that directory. It will be one of several entries (lines) shown inside of the
etc/crontab
file. Go to DIAGNOSTICS > EDIT FILE and browse to and open the/etc/crontab
file. Just look at its contents. Don't change anything or you can mess up your firewall. Just look at the lines and see if one contains the snort file I mentioned.That file is installed as a periodic crontask.
Here is the bottom half of my file as an example:
The "*/5" means the file
/usr/local/pkg/snort/snort_check_cron_misc.inc
is executed every 5 minutes. That PHP file contains the code that does the logs management for Snort.But you have another more serious problem like some runaway process or something if your firewall is generating that many alert log files in 30 minutes. I would suggest rebooting your firewall.
On a typical home network it should take at least several weeks to generate enough alert logs to fill up 1 GB of space.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
/etc/crontab
Ok I located it.
Do you have LAN setup on your Snort service?
Even when i have rebooted the service, the LAN generates multiple alert logs. And it doesn't seem like its getting overwritten or cleaned out by the log management.
I do have my WAN with Send Alerts to System Log enabled but LAN. Do you have that turned on ?
-
@cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
/etc/crontab
Ok I located it.
Do you have LAN setup on your Snort service?
Even when i have rebooted the service, the LAN generates multiple alert logs. And it doesn't seem like its getting overwritten or cleaned out by the log management.
I do have my WAN with Send Alerts to System Log enabled but LAN. Do you have that turned on ?
Yes, I have logging enabled for all my Snort interfaces. I run Snort on the WAN using a few IP blacklists solely to generate logging data for me to test new Snort package updates with.
Are you on a home network? If so, there is absolutely no way you should be generating enough logs to fill up a 43 GB disk in 30 minutes. Even if you are not filling up the disk, there is no way in just 30 minutes you should be generating enough logs to cause you any problems whatsoever.
I seriously suggest rebooting your firewall to be sure you don't have some zombie process out there churning out logs.
You can also try removing and then reinstalling the Snort package. You won't lose any settings by doing that. All of your configuration will be preserved.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
@cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
/etc/crontab
Ok I located it.
Do you have LAN setup on your Snort service?
Even when i have rebooted the service, the LAN generates multiple alert logs. And it doesn't seem like its getting overwritten or cleaned out by the log management.
I do have my WAN with Send Alerts to System Log enabled but LAN. Do you have that turned on ?
Yes, I have logging enabled for all my Snort interfaces. I run Snort on the WAN using a few IP blacklists solely to generate logging data for me to test new Snort package updates with.
Are you on a home network? If so, there is absolutely no way you should be generating enough logs to fill up a 43 GB disk in 30 minutes. Even if you are not filling up the disk, there is no way in just 30 minutes you should be generating enough logs to cause you any problems whatsoever.
I seriously suggest rebooting your firewall to be sure you don't have some zombie process out there churning out logs.
You can also try removing and then reinstalling the Snort package. You won't lose any settings by doing that. All of your configuration will be preserved.
Oh I'm sorry. I didn't mean that it fills up the disk that fast but overtime it does fill up the disk without no cleanup. Maybe monthly or two or so my disk will fill up with these logs.
Yes, this is a home network.
Do you have your logs being sent to system log on either or both of your interfaces?
-
No, I don't send any Snort alert logs to the pfSense system log.
Get to a shell prompt on your firewall and execute this command to force the log cleanup task to run:
php -f /usr/local/pkg/snort/snort_check_cron_misc.inc
Then go look in the pfSense system log to see if anything is logged there. If your directory has files that exceed the limits shown on the LOG MGMT tab, then you should see some log messages showing a cleanup in progress.
If you notice any weird messages on the console when you execute the command above, come back and post those.
-
The symptoms you describe indicate the log cleanup job is not running or else is generating some kind of error.
I've never ever had Snort fill up my disk. And like I mentioned at the top of this exchange, the log management has to be working or else there would be lots of folks here complaining. There are over 24,000 Snort and Suricata installs on pfSense.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
The symptoms you describe indicate the log cleanup job is not running or else is generating some kind of error.
I've never ever had Snort fill up my disk. And like I mentioned at the top of this exchange, the log management has to be working or else there would be lots of folks here complaining. There are over 24,000 Snort and Suricata installs on pfSense.
okay so i ran that command and i cant even tell it executed; i didnt see anything in my system log. i ran it from putty as well as the pfsense shell command.
-
Normally you won't see anything. And if the files currently in the log directory are below the limits (total megabytes space consumption and retention period), then nothing happens. Only when the log directory is beyond the limits does something happen.
So give it a few days or weeks or however long it takes to accumulate more than 1 GB of files in the directory (specifically, 1024 MB as that is your currently configured limit) and then try running the command again manually.
You might also have some zombie Snort process hanging on as well. I still suggest you reboot the firewall to be sure.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
Normally you won't see anything. And if the files currently in the log directory are below the limits (total megabytes space consumption and retention period), then nothing happens. Only when the log directory is beyond the limits does something happen.
So give it a few days or weeks or however long it takes to accumulate more than 1 GB of files in the directory (specifically, 1024 MB as that is your currently configured limit) and then try running the command again manually.
You might also have some zombie Snort process hanging on as well. I still suggest you reboot the firewall to be sure.
okay. thanks for all your help. I will reboot and monitor and report back. Thanks again! :D
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
Normally you won't see anything. And if the files currently in the log directory are below the limits (total megabytes space consumption and retention period), then nothing happens. Only when the log directory is beyond the limits does something happen.
So give it a few days or weeks or however long it takes to accumulate more than 1 GB of files in the directory (specifically, 1024 MB as that is your currently configured limit) and then try running the command again manually.
You might also have some zombie Snort process hanging on as well. I still suggest you reboot the firewall to be sure.
Hello,
so since yesterday it looks it has gone up 3GB.
This is the LAN logs. The only thing i have in LAN is Snort OPENAPPI. I saw in reddit where someone was also having this issue and they said the only way around it was to disable the LAN interface in Snort. I really don't understand why the LAN interface is not cleaning itself.
It is just loaded with alert logs.