Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds
-
The quickest way is to look on the pfSense Dashboard page down towards the bottom. It will show you the amount of used and allocated space for
/
(which is the root system volume). The/var/log/snort/
directory will be allocated from that space.I have an SG-5100 and currently it shows me using 25% of 6.7 GB. That 25% is all of my installed software plus logs (including Snort logs). I have Snort running on three interfaces.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
The quickest way is to look on the pfSense Dashboard page down towards the bottom. It will show you the amount of used and allocated space for
/
(which is the root system volume). The/var/log/snort/
directory will be allocated from that space.I have an SG-5100 and currently it shows me using 25% of 6.7 GB. That 25% is all of my installed software plus logs (including Snort logs). I have Snort running on three interfaces.
It shows 43GiB
-
If you have that much empty space on the system volume, then it sounds like the Snort log cleanup cron task is not executing, or else permissions are messed up somehow in the logging directory.
Are there any messages in the pfSense system log from Snort? The log cleanup job executes every 5 minutes if I remember correctly. So if your Snort logging path size currently exceeds the configured max value, that cron task should print some data about what it's doing to the system log every 5 minutes. Look and see if anything is being logged or if any error messages are showing up.
Look in the
/etc/crontab
file and see if thesnort_check_cron_misc.inc
file is showing in the list of executable jobs.I am running the same version of pfSense and Snort as you and my logging directory is being kept below the configured limit. I have no log files in the directory older than 14 days (my retention limit setting) and the total size in bytes is below the 1340 MB limit I have configured. So the Logs Management process is working for me.
-
How much traffic is going through Snort? Just noticed the timestamps on those alert logs. You are generating a new file every few seconds! Your limit per log file in the interface says 500 KB. How big are the actual alert log files on the disk? Are they also 500 KB?
You have something seriously wrong if your system is generating 500 KBs/few seconds of alert logs. Those numbers on the end of each log file are Unix timestamps, so you can look at how close together the values are and see that your system is creating new files every few seconds.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
snort_check_cron_misc.inc
ok. i checked that directory and i do not have snort_check_cron_misc.inc file. Idk what happened to it or if there was ever one.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
How much traffic is going through Snort? Just noticed the timestamps on those alert logs. You are generating a new file every few seconds! Your limit per log file in the interface says 500 KB. How big are the actual alert log files on the disk? Are they also 500 KB?
You have something seriously wrong if your system is generating 500 KBs/few seconds of alert logs. Those numbers on the end of each log file are Unix timestamps, so you can look at how close together the values are and see that your system is creating new files every few seconds.
This is the LAN interface logs so anytime someone browses or anything.
I have two Snort interfaces; WAN and LAN.
-
@cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
snort_check_cron_misc.inc
ok. i checked that directory and i do not have snort_check_cron_misc.inc file. Idk what happened to it or if there was ever one.
No, it won't be a file in that directory. It will be one of several entries (lines) shown inside of the
etc/crontab
file. Go to DIAGNOSTICS > EDIT FILE and browse to and open the/etc/crontab
file. Just look at its contents. Don't change anything or you can mess up your firewall. Just look at the lines and see if one contains the snort file I mentioned.That file is installed as a periodic crontask.
Here is the bottom half of my file as an example:
The "*/5" means the file
/usr/local/pkg/snort/snort_check_cron_misc.inc
is executed every 5 minutes. That PHP file contains the code that does the logs management for Snort.But you have another more serious problem like some runaway process or something if your firewall is generating that many alert log files in 30 minutes. I would suggest rebooting your firewall.
On a typical home network it should take at least several weeks to generate enough alert logs to fill up 1 GB of space.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
/etc/crontab
Ok I located it.
Do you have LAN setup on your Snort service?
Even when i have rebooted the service, the LAN generates multiple alert logs. And it doesn't seem like its getting overwritten or cleaned out by the log management.
I do have my WAN with Send Alerts to System Log enabled but LAN. Do you have that turned on ?
-
@cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
/etc/crontab
Ok I located it.
Do you have LAN setup on your Snort service?
Even when i have rebooted the service, the LAN generates multiple alert logs. And it doesn't seem like its getting overwritten or cleaned out by the log management.
I do have my WAN with Send Alerts to System Log enabled but LAN. Do you have that turned on ?
Yes, I have logging enabled for all my Snort interfaces. I run Snort on the WAN using a few IP blacklists solely to generate logging data for me to test new Snort package updates with.
Are you on a home network? If so, there is absolutely no way you should be generating enough logs to fill up a 43 GB disk in 30 minutes. Even if you are not filling up the disk, there is no way in just 30 minutes you should be generating enough logs to cause you any problems whatsoever.
I seriously suggest rebooting your firewall to be sure you don't have some zombie process out there churning out logs.
You can also try removing and then reinstalling the Snort package. You won't lose any settings by doing that. All of your configuration will be preserved.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
@cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
/etc/crontab
Ok I located it.
Do you have LAN setup on your Snort service?
Even when i have rebooted the service, the LAN generates multiple alert logs. And it doesn't seem like its getting overwritten or cleaned out by the log management.
I do have my WAN with Send Alerts to System Log enabled but LAN. Do you have that turned on ?
Yes, I have logging enabled for all my Snort interfaces. I run Snort on the WAN using a few IP blacklists solely to generate logging data for me to test new Snort package updates with.
Are you on a home network? If so, there is absolutely no way you should be generating enough logs to fill up a 43 GB disk in 30 minutes. Even if you are not filling up the disk, there is no way in just 30 minutes you should be generating enough logs to cause you any problems whatsoever.
I seriously suggest rebooting your firewall to be sure you don't have some zombie process out there churning out logs.
You can also try removing and then reinstalling the Snort package. You won't lose any settings by doing that. All of your configuration will be preserved.
Oh I'm sorry. I didn't mean that it fills up the disk that fast but overtime it does fill up the disk without no cleanup. Maybe monthly or two or so my disk will fill up with these logs.
Yes, this is a home network.
Do you have your logs being sent to system log on either or both of your interfaces?
-
No, I don't send any Snort alert logs to the pfSense system log.
Get to a shell prompt on your firewall and execute this command to force the log cleanup task to run:
php -f /usr/local/pkg/snort/snort_check_cron_misc.inc
Then go look in the pfSense system log to see if anything is logged there. If your directory has files that exceed the limits shown on the LOG MGMT tab, then you should see some log messages showing a cleanup in progress.
If you notice any weird messages on the console when you execute the command above, come back and post those.
-
The symptoms you describe indicate the log cleanup job is not running or else is generating some kind of error.
I've never ever had Snort fill up my disk. And like I mentioned at the top of this exchange, the log management has to be working or else there would be lots of folks here complaining. There are over 24,000 Snort and Suricata installs on pfSense.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
The symptoms you describe indicate the log cleanup job is not running or else is generating some kind of error.
I've never ever had Snort fill up my disk. And like I mentioned at the top of this exchange, the log management has to be working or else there would be lots of folks here complaining. There are over 24,000 Snort and Suricata installs on pfSense.
okay so i ran that command and i cant even tell it executed; i didnt see anything in my system log. i ran it from putty as well as the pfsense shell command.
-
Normally you won't see anything. And if the files currently in the log directory are below the limits (total megabytes space consumption and retention period), then nothing happens. Only when the log directory is beyond the limits does something happen.
So give it a few days or weeks or however long it takes to accumulate more than 1 GB of files in the directory (specifically, 1024 MB as that is your currently configured limit) and then try running the command again manually.
You might also have some zombie Snort process hanging on as well. I still suggest you reboot the firewall to be sure.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
Normally you won't see anything. And if the files currently in the log directory are below the limits (total megabytes space consumption and retention period), then nothing happens. Only when the log directory is beyond the limits does something happen.
So give it a few days or weeks or however long it takes to accumulate more than 1 GB of files in the directory (specifically, 1024 MB as that is your currently configured limit) and then try running the command again manually.
You might also have some zombie Snort process hanging on as well. I still suggest you reboot the firewall to be sure.
okay. thanks for all your help. I will reboot and monitor and report back. Thanks again! :D
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
Normally you won't see anything. And if the files currently in the log directory are below the limits (total megabytes space consumption and retention period), then nothing happens. Only when the log directory is beyond the limits does something happen.
So give it a few days or weeks or however long it takes to accumulate more than 1 GB of files in the directory (specifically, 1024 MB as that is your currently configured limit) and then try running the command again manually.
You might also have some zombie Snort process hanging on as well. I still suggest you reboot the firewall to be sure.
Hello,
so since yesterday it looks it has gone up 3GB.
This is the LAN logs. The only thing i have in LAN is Snort OPENAPPI. I saw in reddit where someone was also having this issue and they said the only way around it was to disable the LAN interface in Snort. I really don't understand why the LAN interface is not cleaning itself.
It is just loaded with alert logs.
-
The LAN logging sub-directory should be getting cleaned out, but why your firewall is generating that many logs needs to be examined as well.
You say you have OpenAppID enabled on the LAN. Do you have it in blocking mode or just alerting mode? And do you have all of the OpenAppID rules enabled? That's just an extraordinary amount of logging data for a home network.
For now let's go back to the problem of not clearing the directory. What kind of logs are accumulating on your WAN? The directories under
/var/log/snort
will be named using the physical NIC interface name along with a random UUID. So referring to the screenshot you posted, I'm guessing thesnort_igb317847
sub-directory is for your WAN. If so, are the logs in there getting cleaned out properly? Can you post the output of this command for both sub-directories:ls -l /var/log/snort/snort_igb317847'
and
ls -l /var/log/snort/snort_igb044797'
Did your interfaces get renamed perhaps at some point by you changing hardware?
Look again in your system log to see if you see any entries similar to this one. This is from my firewall early this morning. It shows the clean-up cron task executing and removing a pcap log file from the logging sub-directory for my WAN.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
The LAN logging sub-directory should be getting cleaned out, but why your firewall is generating that many logs needs to be examined as well.
You say you have OpenAppID enabled on the LAN. Do you have it in blocking mode or just alerting mode? And do you have all of the OpenAppID rules enabled? That's just an extraordinary amount of logging data for a home network.
For now let's go back to the problem of not clearing the directory. What kind of logs are accumulating on your WAN? The directories under
/var/log/snort
will be named using the physical NIC interface name along with a random UUID. So referring to the screenshot you posted, I'm guessing thesnort_igb317847
sub-directory is for your WAN. If so, are the logs in there getting cleaned out properly? Can you post the output of this command for both sub-directories:ls -l /var/log/snort/snort_igb317847'
and
ls -l /var/log/snort/snort_igb044797'
Did your interfaces get renamed perhaps at some point by you changing hardware?
Look again in your system log to see if you see any entries similar to this one. This is from my firewall early this morning. It shows the clean-up cron task executing and removing a pcap log file from the logging sub-directory for my WAN.
I have all of the OpenAppID rules enabled and its just in alerting mode.
Below is the config:Here is the '/var/log/snort' directory
So idk if WAN is getting cleaned up due to it being forwarded to syslog or what.
This is the result for 'ls -l /var/log/snort/snort_igb317847'
This is the result for 'ls -l /var/log/snort/snort_igb044797'
I can't even get all the logs to show via putty for this interface.
I've never changed any hardware in my system.
I looked in the syslog and it says the cron task ran but the directory size remains...
-
One question from me would be why do you want to enable all those OpenAppID rules for your LAN? If you are using Snort on a home network, what do you care if there is social media and other similar traffic? Why alert and not block if you are concerned about the traffic? You have your firewall very busy doing needless logging (gathering all that alert data and then having to laboriously write it all out to a disk file that you just then come along and later delete). That really makes no sense, but if that's what you want to do, then to each his own.
Back to the question at hand, the only thing I know to do is to create a special debugging version of the cleanup script and let you execute it manually so we can capture more details from each step of the process. Give me a little time to create a debug version and I will attempt to upload it here so you can download and run it.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
One question from me would be why do you want to enable all those OpenAppID rules for your LAN? If you are using Snort on a home network, what do you care if there is social media and other similar traffic? Why alert and not block if you are concerned about the traffic? You have your firewall very busy doing needless logging (gathering all that alert data and then having to laboriously write it all out to a disk file that you just then come along and later delete). That really makes no sense, but if that's what you want to do, then to each his own.
Back to the question at hand, the only thing I know to do is to create a special debugging version of the cleanup script and let you execute it manually so we can capture more details from each step of the process. Give me a little time to create a debug version and I will attempt to upload it here so you can download and run it.
That makes sense. I followed a guide that said to enable them all.
Sometimes it does come in handy when I see that the WAN has blocked something and then I can go to the LAN alerts and see which IP it came from and determine if that is something I should whitelist. I can change it though. How do you have your LAN interface setup ? Which rules do you recommend on the LAN side? I noticed sometimes my Google Homes will not be able to reach the internet sometimes.
What IPS policy did you choose for WAN?