Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds
-
v2.4.4. For the second time now Snort alerts have hammered the drive and filled it up to capacity. I manage to remove older logs but I have two issues at hand. The first is that when this occurs, the Dashboard disappears. It just shows the menu at the top, then "Status/Dashboard" and nothing else. Bonus, the Snort option also disappears from the menu. I have gone into the console and restarted the webConfigurator and the php-fpm process but it doesn't resolve the issue. The results are consistent across multiple browsers. The only way to get everything back to normal is to reboot the system which isn't idea but it gets the job done, sorta. Read below. Is there another way to work around this issue if it happens again?
The reason I ask is because the Snort Log MGMT settings page is goofy at best. It doesn't seem to obey the Log Directory Size config parameter and often reverts back to some funky value like -1819 or something.
Lastly, usually when I manage to reboot the system it boots up and acts like it has lost all interface configurations. It was hang indefinitely at the point in the attached image. And even if I go through hand resetting everything, the LAN interface rarely comes back on net. It gets limited ARP information and is not pingable from most systems on the network. In all the times this has happened, I really just end up factory resetting the system and quickly rebuilding it.
-
This is Snort 4.0 default, and values might be different from yours...did you changed yours?
-
Enable a log directory size limit as well as log sizes to be sure duplicate files don't fill the drive.
Steve
-
Hello,
I am also having this issue. My snort LAN interface logs are filling my disk. I have turned on Auto Log Management and changed the values and the issue persists. I have to go in and manually clean out the logs of the LAN interface. It generates multiple alert logs.
I literally just cleaned out this directory and it has generated all of those within 30 min.
It's rather annoying. Is there a fix for this ?Below is my current Log Mgmt settings:
-
What versions of pfSense and the Snort package are you running? Some issues with log rotation were corrected several versions back in the Snort package. To my knowledge, users are not reporting widespread issues with logs management on either Snort or Suricata in the current package versions.
There have been a few isolated cases that were caused by things the users had done (or more aptly, not done).
How much space do you have allocated for /var on your firewall? Are you by chance using a RAM disk? If so, don't do that with Snort!
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
What versions of pfSense and the Snort package are you running? Some issues with log rotation were corrected several versions back in the Snort package. To my knowledge, users are not reporting widespread issues with logs management on either Snort or Suricata in the current package versions.
There have been a few isolated cases that were caused by things the users had done (or more aptly, not done).
How much space do you have allocated for /var on your firewall? Are you by chance using a RAM disk? If so, don't do that with Snort!
Hello and thank you for the quick response. I am using PFsense 2.4.5 and Snort 3.2.9.10_4. I am not using a RAM disk. Where would i check how space I have allocated for /var? I don't think i have ever set anything...
-
The quickest way is to look on the pfSense Dashboard page down towards the bottom. It will show you the amount of used and allocated space for
/
(which is the root system volume). The/var/log/snort/
directory will be allocated from that space.I have an SG-5100 and currently it shows me using 25% of 6.7 GB. That 25% is all of my installed software plus logs (including Snort logs). I have Snort running on three interfaces.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
The quickest way is to look on the pfSense Dashboard page down towards the bottom. It will show you the amount of used and allocated space for
/
(which is the root system volume). The/var/log/snort/
directory will be allocated from that space.I have an SG-5100 and currently it shows me using 25% of 6.7 GB. That 25% is all of my installed software plus logs (including Snort logs). I have Snort running on three interfaces.
It shows 43GiB
-
If you have that much empty space on the system volume, then it sounds like the Snort log cleanup cron task is not executing, or else permissions are messed up somehow in the logging directory.
Are there any messages in the pfSense system log from Snort? The log cleanup job executes every 5 minutes if I remember correctly. So if your Snort logging path size currently exceeds the configured max value, that cron task should print some data about what it's doing to the system log every 5 minutes. Look and see if anything is being logged or if any error messages are showing up.
Look in the
/etc/crontab
file and see if thesnort_check_cron_misc.inc
file is showing in the list of executable jobs.I am running the same version of pfSense and Snort as you and my logging directory is being kept below the configured limit. I have no log files in the directory older than 14 days (my retention limit setting) and the total size in bytes is below the 1340 MB limit I have configured. So the Logs Management process is working for me.
-
How much traffic is going through Snort? Just noticed the timestamps on those alert logs. You are generating a new file every few seconds! Your limit per log file in the interface says 500 KB. How big are the actual alert log files on the disk? Are they also 500 KB?
You have something seriously wrong if your system is generating 500 KBs/few seconds of alert logs. Those numbers on the end of each log file are Unix timestamps, so you can look at how close together the values are and see that your system is creating new files every few seconds.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
snort_check_cron_misc.inc
ok. i checked that directory and i do not have snort_check_cron_misc.inc file. Idk what happened to it or if there was ever one.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
How much traffic is going through Snort? Just noticed the timestamps on those alert logs. You are generating a new file every few seconds! Your limit per log file in the interface says 500 KB. How big are the actual alert log files on the disk? Are they also 500 KB?
You have something seriously wrong if your system is generating 500 KBs/few seconds of alert logs. Those numbers on the end of each log file are Unix timestamps, so you can look at how close together the values are and see that your system is creating new files every few seconds.
This is the LAN interface logs so anytime someone browses or anything.
I have two Snort interfaces; WAN and LAN.
-
@cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
snort_check_cron_misc.inc
ok. i checked that directory and i do not have snort_check_cron_misc.inc file. Idk what happened to it or if there was ever one.
No, it won't be a file in that directory. It will be one of several entries (lines) shown inside of the
etc/crontab
file. Go to DIAGNOSTICS > EDIT FILE and browse to and open the/etc/crontab
file. Just look at its contents. Don't change anything or you can mess up your firewall. Just look at the lines and see if one contains the snort file I mentioned.That file is installed as a periodic crontask.
Here is the bottom half of my file as an example:
The "*/5" means the file
/usr/local/pkg/snort/snort_check_cron_misc.inc
is executed every 5 minutes. That PHP file contains the code that does the logs management for Snort.But you have another more serious problem like some runaway process or something if your firewall is generating that many alert log files in 30 minutes. I would suggest rebooting your firewall.
On a typical home network it should take at least several weeks to generate enough alert logs to fill up 1 GB of space.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
/etc/crontab
Ok I located it.
Do you have LAN setup on your Snort service?
Even when i have rebooted the service, the LAN generates multiple alert logs. And it doesn't seem like its getting overwritten or cleaned out by the log management.
I do have my WAN with Send Alerts to System Log enabled but LAN. Do you have that turned on ?
-
@cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
/etc/crontab
Ok I located it.
Do you have LAN setup on your Snort service?
Even when i have rebooted the service, the LAN generates multiple alert logs. And it doesn't seem like its getting overwritten or cleaned out by the log management.
I do have my WAN with Send Alerts to System Log enabled but LAN. Do you have that turned on ?
Yes, I have logging enabled for all my Snort interfaces. I run Snort on the WAN using a few IP blacklists solely to generate logging data for me to test new Snort package updates with.
Are you on a home network? If so, there is absolutely no way you should be generating enough logs to fill up a 43 GB disk in 30 minutes. Even if you are not filling up the disk, there is no way in just 30 minutes you should be generating enough logs to cause you any problems whatsoever.
I seriously suggest rebooting your firewall to be sure you don't have some zombie process out there churning out logs.
You can also try removing and then reinstalling the Snort package. You won't lose any settings by doing that. All of your configuration will be preserved.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
@cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
/etc/crontab
Ok I located it.
Do you have LAN setup on your Snort service?
Even when i have rebooted the service, the LAN generates multiple alert logs. And it doesn't seem like its getting overwritten or cleaned out by the log management.
I do have my WAN with Send Alerts to System Log enabled but LAN. Do you have that turned on ?
Yes, I have logging enabled for all my Snort interfaces. I run Snort on the WAN using a few IP blacklists solely to generate logging data for me to test new Snort package updates with.
Are you on a home network? If so, there is absolutely no way you should be generating enough logs to fill up a 43 GB disk in 30 minutes. Even if you are not filling up the disk, there is no way in just 30 minutes you should be generating enough logs to cause you any problems whatsoever.
I seriously suggest rebooting your firewall to be sure you don't have some zombie process out there churning out logs.
You can also try removing and then reinstalling the Snort package. You won't lose any settings by doing that. All of your configuration will be preserved.
Oh I'm sorry. I didn't mean that it fills up the disk that fast but overtime it does fill up the disk without no cleanup. Maybe monthly or two or so my disk will fill up with these logs.
Yes, this is a home network.
Do you have your logs being sent to system log on either or both of your interfaces?
-
No, I don't send any Snort alert logs to the pfSense system log.
Get to a shell prompt on your firewall and execute this command to force the log cleanup task to run:
php -f /usr/local/pkg/snort/snort_check_cron_misc.inc
Then go look in the pfSense system log to see if anything is logged there. If your directory has files that exceed the limits shown on the LOG MGMT tab, then you should see some log messages showing a cleanup in progress.
If you notice any weird messages on the console when you execute the command above, come back and post those.
-
The symptoms you describe indicate the log cleanup job is not running or else is generating some kind of error.
I've never ever had Snort fill up my disk. And like I mentioned at the top of this exchange, the log management has to be working or else there would be lots of folks here complaining. There are over 24,000 Snort and Suricata installs on pfSense.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
The symptoms you describe indicate the log cleanup job is not running or else is generating some kind of error.
I've never ever had Snort fill up my disk. And like I mentioned at the top of this exchange, the log management has to be working or else there would be lots of folks here complaining. There are over 24,000 Snort and Suricata installs on pfSense.
okay so i ran that command and i cant even tell it executed; i didnt see anything in my system log. i ran it from putty as well as the pfsense shell command.
-
Normally you won't see anything. And if the files currently in the log directory are below the limits (total megabytes space consumption and retention period), then nothing happens. Only when the log directory is beyond the limits does something happen.
So give it a few days or weeks or however long it takes to accumulate more than 1 GB of files in the directory (specifically, 1024 MB as that is your currently configured limit) and then try running the command again manually.
You might also have some zombie Snort process hanging on as well. I still suggest you reboot the firewall to be sure.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
Normally you won't see anything. And if the files currently in the log directory are below the limits (total megabytes space consumption and retention period), then nothing happens. Only when the log directory is beyond the limits does something happen.
So give it a few days or weeks or however long it takes to accumulate more than 1 GB of files in the directory (specifically, 1024 MB as that is your currently configured limit) and then try running the command again manually.
You might also have some zombie Snort process hanging on as well. I still suggest you reboot the firewall to be sure.
okay. thanks for all your help. I will reboot and monitor and report back. Thanks again! :D
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
Normally you won't see anything. And if the files currently in the log directory are below the limits (total megabytes space consumption and retention period), then nothing happens. Only when the log directory is beyond the limits does something happen.
So give it a few days or weeks or however long it takes to accumulate more than 1 GB of files in the directory (specifically, 1024 MB as that is your currently configured limit) and then try running the command again manually.
You might also have some zombie Snort process hanging on as well. I still suggest you reboot the firewall to be sure.
Hello,
so since yesterday it looks it has gone up 3GB.
This is the LAN logs. The only thing i have in LAN is Snort OPENAPPI. I saw in reddit where someone was also having this issue and they said the only way around it was to disable the LAN interface in Snort. I really don't understand why the LAN interface is not cleaning itself.
It is just loaded with alert logs.
-
The LAN logging sub-directory should be getting cleaned out, but why your firewall is generating that many logs needs to be examined as well.
You say you have OpenAppID enabled on the LAN. Do you have it in blocking mode or just alerting mode? And do you have all of the OpenAppID rules enabled? That's just an extraordinary amount of logging data for a home network.
For now let's go back to the problem of not clearing the directory. What kind of logs are accumulating on your WAN? The directories under
/var/log/snort
will be named using the physical NIC interface name along with a random UUID. So referring to the screenshot you posted, I'm guessing thesnort_igb317847
sub-directory is for your WAN. If so, are the logs in there getting cleaned out properly? Can you post the output of this command for both sub-directories:ls -l /var/log/snort/snort_igb317847'
and
ls -l /var/log/snort/snort_igb044797'
Did your interfaces get renamed perhaps at some point by you changing hardware?
Look again in your system log to see if you see any entries similar to this one. This is from my firewall early this morning. It shows the clean-up cron task executing and removing a pcap log file from the logging sub-directory for my WAN.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
The LAN logging sub-directory should be getting cleaned out, but why your firewall is generating that many logs needs to be examined as well.
You say you have OpenAppID enabled on the LAN. Do you have it in blocking mode or just alerting mode? And do you have all of the OpenAppID rules enabled? That's just an extraordinary amount of logging data for a home network.
For now let's go back to the problem of not clearing the directory. What kind of logs are accumulating on your WAN? The directories under
/var/log/snort
will be named using the physical NIC interface name along with a random UUID. So referring to the screenshot you posted, I'm guessing thesnort_igb317847
sub-directory is for your WAN. If so, are the logs in there getting cleaned out properly? Can you post the output of this command for both sub-directories:ls -l /var/log/snort/snort_igb317847'
and
ls -l /var/log/snort/snort_igb044797'
Did your interfaces get renamed perhaps at some point by you changing hardware?
Look again in your system log to see if you see any entries similar to this one. This is from my firewall early this morning. It shows the clean-up cron task executing and removing a pcap log file from the logging sub-directory for my WAN.
I have all of the OpenAppID rules enabled and its just in alerting mode.
Below is the config:Here is the '/var/log/snort' directory
So idk if WAN is getting cleaned up due to it being forwarded to syslog or what.
This is the result for 'ls -l /var/log/snort/snort_igb317847'
This is the result for 'ls -l /var/log/snort/snort_igb044797'
I can't even get all the logs to show via putty for this interface.
I've never changed any hardware in my system.
I looked in the syslog and it says the cron task ran but the directory size remains...
-
One question from me would be why do you want to enable all those OpenAppID rules for your LAN? If you are using Snort on a home network, what do you care if there is social media and other similar traffic? Why alert and not block if you are concerned about the traffic? You have your firewall very busy doing needless logging (gathering all that alert data and then having to laboriously write it all out to a disk file that you just then come along and later delete). That really makes no sense, but if that's what you want to do, then to each his own.
Back to the question at hand, the only thing I know to do is to create a special debugging version of the cleanup script and let you execute it manually so we can capture more details from each step of the process. Give me a little time to create a debug version and I will attempt to upload it here so you can download and run it.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
One question from me would be why do you want to enable all those OpenAppID rules for your LAN? If you are using Snort on a home network, what do you care if there is social media and other similar traffic? Why alert and not block if you are concerned about the traffic? You have your firewall very busy doing needless logging (gathering all that alert data and then having to laboriously write it all out to a disk file that you just then come along and later delete). That really makes no sense, but if that's what you want to do, then to each his own.
Back to the question at hand, the only thing I know to do is to create a special debugging version of the cleanup script and let you execute it manually so we can capture more details from each step of the process. Give me a little time to create a debug version and I will attempt to upload it here so you can download and run it.
That makes sense. I followed a guide that said to enable them all.
Sometimes it does come in handy when I see that the WAN has blocked something and then I can go to the LAN alerts and see which IP it came from and determine if that is something I should whitelist. I can change it though. How do you have your LAN interface setup ? Which rules do you recommend on the LAN side? I noticed sometimes my Google Homes will not be able to reach the internet sometimes.
What IPS policy did you choose for WAN?
-
I really have no rules on the WAN. Snort is not needed there. As I mentioned before, the only ones I have are some blacklisted IP rules on my WAN just to generate alert logs so I have something to test with when working on new versions of the Snort package. So yes, I do have two categories of rules on my WAN, but they are not for security. They are to generate data for testing. If I did not need the testing data, I would remove those rules and have nothing on my WAN.
On my LAN I do not use the OpenAppID rules. They are fundamentally useless on a home network. Those rules are designed for the corporate enterprise world where you don't want employees wasting time and the company's network bandwidth engaging in social media chit-chat and streaming music or movies. That's really the only environment where using the OpenAppID rules make any sense. They are not designed for a home network.
If you are an experienced IDS/IPS admin and know how to tune a rule set, then put Snort on your LAN and enable an IPS Policy. I usually recommend folks start with "Connectivity" and only when they are really proficient should they try "Balanced". That is the max. Never use "Security" or the other "Max Detect" policy on a home network! Those are for protecting maybe the CIA or NSA servers and for testing in a lab. They are not for any typical home or enterprise network! Enabling those will drive you crazy with needless blocks due to false positives.
So to answer your question, I have zero rules on my WAN and only the IPS Policy "Balanced" on my LAN. I've tuned the HTTP_INSPECT preprocessor rules to eliminate false positives and disabled a tiny handful of other preprocessor rules and that's it. I get maybe three to six alerts per quarter on my LAN. I've had only 2 alerts on my LAN this month (for April). One on April 10 and the other on April 17. Both were false positives from a portscan rule triggered by my iPhone and my wife's iPhone interracting with Facebook's CDN. So far this entire year, I've had exactly 5 alerts triggered on my LAN. This is how it should be when you have a well-tuned IDS/IPS rule set that matches the actual attack surfaces in your network. You don't need to enable every rule. Many of the rules are to protect traffic that your network would never see. Why would you want web server and web server rules enabled when you don't have a web server or mail server on your network accepting inbound traffic from the Internet?
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
I really have no rules on the WAN. Snort is not needed there. As I mentioned before, the only I have are some blacklisted IP rules on my WAN just to generate alert logs so I have something test with when working on new versions of the Snort package. So yes, I do have two categories of rules on my WAN, but they are not for security. They are to generate data for testing. If I did not need the testing data, I would remove those rules and have nothing on my WAN.
On my LAN I do not use the OpenAppID rules. They are fundamentally useless on a home network. Those rules are designed for the corporate enterprise world where you don't want employees wasting time and the company's network bandwidth engaging in social media chit-chat and streaming music or movies. That's really the only environment where using the OpenAppID rules make any sense. They are not designed for a home network.
If you are an experienced IDS/IPS admin and know how to tune a rule set, then put Snort on your LAN and enable an IPS Policy. I usually recommend folks start with "Connectivity" and only when they are really proficient should they try "Balanced". That is the max. Never use "Security" or the other "Max Detect" policy on a home network! Those are for protecting maybe the CIA or NSA servers and for testing in a lab. They are not for any typical home or enterprise network!
So to answer your question, I have zero rules on my WAN and only the IPS Policy "Balanced" on my LAN. I've tuned the HTTP_INSPECT preprocessor rules to eliminate false positives and disabled a tiny handful of other preprocessor rules and that's it. I get maybe three to six alerts per quarter on my LAN. I've had only 2 alerts on my LAN this month (for April). One on April 10 and the other on April 17. Both were false positives from a portscan rule triggered by my iPhone and my wife's iPhone interracting with Facebook's CDN. So far this entire year, I've had exactly 5 alerts triggered on my LAN. This is how it should be when you have a well-tuned IDS/IPS rule set that matches the actual attack surfaces in your network. You don't need to enable every rule. Many of the rules are to protect traffic that your network would never see. Why would you want web server and web server rules enabled when you don't have a web server or mail server on your network accepting inbound traffic from the Internet?
Hmmm....That's interesting. Most of the guides I've viewed show to enable IPS on the WAN. And the screenshot I sent you was directly from netgate.
So in doing your method under Global settings which rules did you have checked ?
Wouldn't the network be more protected in using the WAN interface?
Sorry for my ignorance in the matter. And I do appreciate all of your help!
-
@cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
Hmmm....That's interesting. Most of the guides I've viewed show to enable IPS on the WAN. And the screenshot I sent you was directly from netgate.
So in doing your method under Global settings which rules did you have checked ?
Wouldn't the network be more protected in using the WAN interface?
Sorry for my ignorance in the matter. And I do appreciate all of your help!
Many of those guides are quite old and really were originally written from an enterprise network point of view where you may not necessarily be using NAT and where public-facing services are more common.
In a home network I will venture out there and say NEVER put Snort or any IDS/IPS on the WAN. No need. It does nothing at all for security because the WAN will already block all unsolicited inbound traffic by default. The WAN will accept no incoming packet unless it is a stateful reply to a previously established session that was initiated by an internal host. Putting the IDS/IPS on your LAN and other internal interfaces (if desired) is all you need. Any traffic coming from and going to internal hosts must traverse the internal interfaces on the firewall. So putting the IDS/IPS there is sufficient.
The rules I have enabled on the GLOBAL SETTINGS tab are just the Snort Subscriber Rules. When you use those, you don't need the Snort GPLv2 Community Rules as every one of those is already in the Snort Subscriber Rules set.
Oh, I forgot. I did enable the ET-Open rules simply to get access to two of their IP blacklists. Those are the two I use only for testing on my WAN to generate that test data. I would not use the ET-Open rules otherwise.
-
Oh, and in looking over the PHP code for that cron task while getting ready to modify it for debug mode, I think I found the issue with the directory cleanup. The code is not looking for the rotated alert logs (the files with alert.<timestamp> in their names), and thus it does not clean those up.
You are seeing the issue because you are logging so much data due to having all those rules enabled. Turn that stuff off and you should see your alert log generation go way down. I will work on a fix and post it a bit later for everyone as a Snort package update.
It's not affecting a large number of folks because most will not be generating nearly as much logging traffic as you are with all those enabled rules alerting. You must have some terribly busy folks on your home LAN visiting social media and streaming sites to generate that much data ... .
-
Okay. I have made some changes so we shall see how this goes. Thanks again. I do still want the new script whenever you can. Thanks
-
@cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
Okay. I have made some changes so we shall see how this goes. Thanks again. I do still want the new script whenever you can. Thanks
I will make an update to the package and post it for the pfSense team to review and merge, so it will be a few days before that is done.
-
The fix for the Logs Management task skipping rotated alert log files has been submitted to both the 2.4.5-RELEASE and 2.5-DEVEL branches of pfSense for the developer team to review and merge.
I also updated the underlying Snort binary to the latest 2.9.16 version released by the Snort team.
Look for the updated Snort package to appear in the near future.
-
The new Snort package version containing the fix for this issue has been posted. It should show up as Snort v3.2.9.11 on pfSense-2.4.5 machines.
-
Just wanted to report that it looks like things are working now. Thanks for the help!
-
@cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
Just wanted to report that it looks like things are working now. Thanks for the help!
Great! Turns out there was an issue in the log purging code after all. I had just not ever hit it because my logging "rate" is low and my system was not generating as many logs. Also, I do not run OpenAppID and thus my personal system was not generating any of those logs at all.
Thanks for reporting the issue.