Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds
-
v2.4.4. For the second time now Snort alerts have hammered the drive and filled it up to capacity. I manage to remove older logs but I have two issues at hand. The first is that when this occurs, the Dashboard disappears. It just shows the menu at the top, then "Status/Dashboard" and nothing else. Bonus, the Snort option also disappears from the menu. I have gone into the console and restarted the webConfigurator and the php-fpm process but it doesn't resolve the issue. The results are consistent across multiple browsers. The only way to get everything back to normal is to reboot the system which isn't idea but it gets the job done, sorta. Read below. Is there another way to work around this issue if it happens again?
The reason I ask is because the Snort Log MGMT settings page is goofy at best. It doesn't seem to obey the Log Directory Size config parameter and often reverts back to some funky value like -1819 or something.
Lastly, usually when I manage to reboot the system it boots up and acts like it has lost all interface configurations. It was hang indefinitely at the point in the attached image. And even if I go through hand resetting everything, the LAN interface rarely comes back on net. It gets limited ARP information and is not pingable from most systems on the network. In all the times this has happened, I really just end up factory resetting the system and quickly rebuilding it.
-
This is Snort 4.0 default, and values might be different from yours...did you changed yours?
-
Enable a log directory size limit as well as log sizes to be sure duplicate files don't fill the drive.
Steve
-
Hello,
I am also having this issue. My snort LAN interface logs are filling my disk. I have turned on Auto Log Management and changed the values and the issue persists. I have to go in and manually clean out the logs of the LAN interface. It generates multiple alert logs.
I literally just cleaned out this directory and it has generated all of those within 30 min.
It's rather annoying. Is there a fix for this ?Below is my current Log Mgmt settings:
-
What versions of pfSense and the Snort package are you running? Some issues with log rotation were corrected several versions back in the Snort package. To my knowledge, users are not reporting widespread issues with logs management on either Snort or Suricata in the current package versions.
There have been a few isolated cases that were caused by things the users had done (or more aptly, not done).
How much space do you have allocated for /var on your firewall? Are you by chance using a RAM disk? If so, don't do that with Snort!
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
What versions of pfSense and the Snort package are you running? Some issues with log rotation were corrected several versions back in the Snort package. To my knowledge, users are not reporting widespread issues with logs management on either Snort or Suricata in the current package versions.
There have been a few isolated cases that were caused by things the users had done (or more aptly, not done).
How much space do you have allocated for /var on your firewall? Are you by chance using a RAM disk? If so, don't do that with Snort!
Hello and thank you for the quick response. I am using PFsense 2.4.5 and Snort 3.2.9.10_4. I am not using a RAM disk. Where would i check how space I have allocated for /var? I don't think i have ever set anything...
-
The quickest way is to look on the pfSense Dashboard page down towards the bottom. It will show you the amount of used and allocated space for
/
(which is the root system volume). The/var/log/snort/
directory will be allocated from that space.I have an SG-5100 and currently it shows me using 25% of 6.7 GB. That 25% is all of my installed software plus logs (including Snort logs). I have Snort running on three interfaces.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
The quickest way is to look on the pfSense Dashboard page down towards the bottom. It will show you the amount of used and allocated space for
/
(which is the root system volume). The/var/log/snort/
directory will be allocated from that space.I have an SG-5100 and currently it shows me using 25% of 6.7 GB. That 25% is all of my installed software plus logs (including Snort logs). I have Snort running on three interfaces.
It shows 43GiB
-
If you have that much empty space on the system volume, then it sounds like the Snort log cleanup cron task is not executing, or else permissions are messed up somehow in the logging directory.
Are there any messages in the pfSense system log from Snort? The log cleanup job executes every 5 minutes if I remember correctly. So if your Snort logging path size currently exceeds the configured max value, that cron task should print some data about what it's doing to the system log every 5 minutes. Look and see if anything is being logged or if any error messages are showing up.
Look in the
/etc/crontab
file and see if thesnort_check_cron_misc.inc
file is showing in the list of executable jobs.I am running the same version of pfSense and Snort as you and my logging directory is being kept below the configured limit. I have no log files in the directory older than 14 days (my retention limit setting) and the total size in bytes is below the 1340 MB limit I have configured. So the Logs Management process is working for me.
-
How much traffic is going through Snort? Just noticed the timestamps on those alert logs. You are generating a new file every few seconds! Your limit per log file in the interface says 500 KB. How big are the actual alert log files on the disk? Are they also 500 KB?
You have something seriously wrong if your system is generating 500 KBs/few seconds of alert logs. Those numbers on the end of each log file are Unix timestamps, so you can look at how close together the values are and see that your system is creating new files every few seconds.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
snort_check_cron_misc.inc
ok. i checked that directory and i do not have snort_check_cron_misc.inc file. Idk what happened to it or if there was ever one.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
How much traffic is going through Snort? Just noticed the timestamps on those alert logs. You are generating a new file every few seconds! Your limit per log file in the interface says 500 KB. How big are the actual alert log files on the disk? Are they also 500 KB?
You have something seriously wrong if your system is generating 500 KBs/few seconds of alert logs. Those numbers on the end of each log file are Unix timestamps, so you can look at how close together the values are and see that your system is creating new files every few seconds.
This is the LAN interface logs so anytime someone browses or anything.
I have two Snort interfaces; WAN and LAN.
-
@cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
snort_check_cron_misc.inc
ok. i checked that directory and i do not have snort_check_cron_misc.inc file. Idk what happened to it or if there was ever one.
No, it won't be a file in that directory. It will be one of several entries (lines) shown inside of the
etc/crontab
file. Go to DIAGNOSTICS > EDIT FILE and browse to and open the/etc/crontab
file. Just look at its contents. Don't change anything or you can mess up your firewall. Just look at the lines and see if one contains the snort file I mentioned.That file is installed as a periodic crontask.
Here is the bottom half of my file as an example:
The "*/5" means the file
/usr/local/pkg/snort/snort_check_cron_misc.inc
is executed every 5 minutes. That PHP file contains the code that does the logs management for Snort.But you have another more serious problem like some runaway process or something if your firewall is generating that many alert log files in 30 minutes. I would suggest rebooting your firewall.
On a typical home network it should take at least several weeks to generate enough alert logs to fill up 1 GB of space.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
/etc/crontab
Ok I located it.
Do you have LAN setup on your Snort service?
Even when i have rebooted the service, the LAN generates multiple alert logs. And it doesn't seem like its getting overwritten or cleaned out by the log management.
I do have my WAN with Send Alerts to System Log enabled but LAN. Do you have that turned on ?
-
@cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
/etc/crontab
Ok I located it.
Do you have LAN setup on your Snort service?
Even when i have rebooted the service, the LAN generates multiple alert logs. And it doesn't seem like its getting overwritten or cleaned out by the log management.
I do have my WAN with Send Alerts to System Log enabled but LAN. Do you have that turned on ?
Yes, I have logging enabled for all my Snort interfaces. I run Snort on the WAN using a few IP blacklists solely to generate logging data for me to test new Snort package updates with.
Are you on a home network? If so, there is absolutely no way you should be generating enough logs to fill up a 43 GB disk in 30 minutes. Even if you are not filling up the disk, there is no way in just 30 minutes you should be generating enough logs to cause you any problems whatsoever.
I seriously suggest rebooting your firewall to be sure you don't have some zombie process out there churning out logs.
You can also try removing and then reinstalling the Snort package. You won't lose any settings by doing that. All of your configuration will be preserved.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
@cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
/etc/crontab
Ok I located it.
Do you have LAN setup on your Snort service?
Even when i have rebooted the service, the LAN generates multiple alert logs. And it doesn't seem like its getting overwritten or cleaned out by the log management.
I do have my WAN with Send Alerts to System Log enabled but LAN. Do you have that turned on ?
Yes, I have logging enabled for all my Snort interfaces. I run Snort on the WAN using a few IP blacklists solely to generate logging data for me to test new Snort package updates with.
Are you on a home network? If so, there is absolutely no way you should be generating enough logs to fill up a 43 GB disk in 30 minutes. Even if you are not filling up the disk, there is no way in just 30 minutes you should be generating enough logs to cause you any problems whatsoever.
I seriously suggest rebooting your firewall to be sure you don't have some zombie process out there churning out logs.
You can also try removing and then reinstalling the Snort package. You won't lose any settings by doing that. All of your configuration will be preserved.
Oh I'm sorry. I didn't mean that it fills up the disk that fast but overtime it does fill up the disk without no cleanup. Maybe monthly or two or so my disk will fill up with these logs.
Yes, this is a home network.
Do you have your logs being sent to system log on either or both of your interfaces?
-
No, I don't send any Snort alert logs to the pfSense system log.
Get to a shell prompt on your firewall and execute this command to force the log cleanup task to run:
php -f /usr/local/pkg/snort/snort_check_cron_misc.inc
Then go look in the pfSense system log to see if anything is logged there. If your directory has files that exceed the limits shown on the LOG MGMT tab, then you should see some log messages showing a cleanup in progress.
If you notice any weird messages on the console when you execute the command above, come back and post those.
-
The symptoms you describe indicate the log cleanup job is not running or else is generating some kind of error.
I've never ever had Snort fill up my disk. And like I mentioned at the top of this exchange, the log management has to be working or else there would be lots of folks here complaining. There are over 24,000 Snort and Suricata installs on pfSense.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
The symptoms you describe indicate the log cleanup job is not running or else is generating some kind of error.
I've never ever had Snort fill up my disk. And like I mentioned at the top of this exchange, the log management has to be working or else there would be lots of folks here complaining. There are over 24,000 Snort and Suricata installs on pfSense.
okay so i ran that command and i cant even tell it executed; i didnt see anything in my system log. i ran it from putty as well as the pfsense shell command.
-
Normally you won't see anything. And if the files currently in the log directory are below the limits (total megabytes space consumption and retention period), then nothing happens. Only when the log directory is beyond the limits does something happen.
So give it a few days or weeks or however long it takes to accumulate more than 1 GB of files in the directory (specifically, 1024 MB as that is your currently configured limit) and then try running the command again manually.
You might also have some zombie Snort process hanging on as well. I still suggest you reboot the firewall to be sure.