Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds
-
The LAN logging sub-directory should be getting cleaned out, but why your firewall is generating that many logs needs to be examined as well.
You say you have OpenAppID enabled on the LAN. Do you have it in blocking mode or just alerting mode? And do you have all of the OpenAppID rules enabled? That's just an extraordinary amount of logging data for a home network.
For now let's go back to the problem of not clearing the directory. What kind of logs are accumulating on your WAN? The directories under
/var/log/snort
will be named using the physical NIC interface name along with a random UUID. So referring to the screenshot you posted, I'm guessing thesnort_igb317847
sub-directory is for your WAN. If so, are the logs in there getting cleaned out properly? Can you post the output of this command for both sub-directories:ls -l /var/log/snort/snort_igb317847'
and
ls -l /var/log/snort/snort_igb044797'
Did your interfaces get renamed perhaps at some point by you changing hardware?
Look again in your system log to see if you see any entries similar to this one. This is from my firewall early this morning. It shows the clean-up cron task executing and removing a pcap log file from the logging sub-directory for my WAN.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
The LAN logging sub-directory should be getting cleaned out, but why your firewall is generating that many logs needs to be examined as well.
You say you have OpenAppID enabled on the LAN. Do you have it in blocking mode or just alerting mode? And do you have all of the OpenAppID rules enabled? That's just an extraordinary amount of logging data for a home network.
For now let's go back to the problem of not clearing the directory. What kind of logs are accumulating on your WAN? The directories under
/var/log/snort
will be named using the physical NIC interface name along with a random UUID. So referring to the screenshot you posted, I'm guessing thesnort_igb317847
sub-directory is for your WAN. If so, are the logs in there getting cleaned out properly? Can you post the output of this command for both sub-directories:ls -l /var/log/snort/snort_igb317847'
and
ls -l /var/log/snort/snort_igb044797'
Did your interfaces get renamed perhaps at some point by you changing hardware?
Look again in your system log to see if you see any entries similar to this one. This is from my firewall early this morning. It shows the clean-up cron task executing and removing a pcap log file from the logging sub-directory for my WAN.
I have all of the OpenAppID rules enabled and its just in alerting mode.
Below is the config:Here is the '/var/log/snort' directory
So idk if WAN is getting cleaned up due to it being forwarded to syslog or what.
This is the result for 'ls -l /var/log/snort/snort_igb317847'
This is the result for 'ls -l /var/log/snort/snort_igb044797'
I can't even get all the logs to show via putty for this interface.
I've never changed any hardware in my system.
I looked in the syslog and it says the cron task ran but the directory size remains...
-
One question from me would be why do you want to enable all those OpenAppID rules for your LAN? If you are using Snort on a home network, what do you care if there is social media and other similar traffic? Why alert and not block if you are concerned about the traffic? You have your firewall very busy doing needless logging (gathering all that alert data and then having to laboriously write it all out to a disk file that you just then come along and later delete). That really makes no sense, but if that's what you want to do, then to each his own.
Back to the question at hand, the only thing I know to do is to create a special debugging version of the cleanup script and let you execute it manually so we can capture more details from each step of the process. Give me a little time to create a debug version and I will attempt to upload it here so you can download and run it.
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
One question from me would be why do you want to enable all those OpenAppID rules for your LAN? If you are using Snort on a home network, what do you care if there is social media and other similar traffic? Why alert and not block if you are concerned about the traffic? You have your firewall very busy doing needless logging (gathering all that alert data and then having to laboriously write it all out to a disk file that you just then come along and later delete). That really makes no sense, but if that's what you want to do, then to each his own.
Back to the question at hand, the only thing I know to do is to create a special debugging version of the cleanup script and let you execute it manually so we can capture more details from each step of the process. Give me a little time to create a debug version and I will attempt to upload it here so you can download and run it.
That makes sense. I followed a guide that said to enable them all.
Sometimes it does come in handy when I see that the WAN has blocked something and then I can go to the LAN alerts and see which IP it came from and determine if that is something I should whitelist. I can change it though. How do you have your LAN interface setup ? Which rules do you recommend on the LAN side? I noticed sometimes my Google Homes will not be able to reach the internet sometimes.
What IPS policy did you choose for WAN?
-
I really have no rules on the WAN. Snort is not needed there. As I mentioned before, the only ones I have are some blacklisted IP rules on my WAN just to generate alert logs so I have something to test with when working on new versions of the Snort package. So yes, I do have two categories of rules on my WAN, but they are not for security. They are to generate data for testing. If I did not need the testing data, I would remove those rules and have nothing on my WAN.
On my LAN I do not use the OpenAppID rules. They are fundamentally useless on a home network. Those rules are designed for the corporate enterprise world where you don't want employees wasting time and the company's network bandwidth engaging in social media chit-chat and streaming music or movies. That's really the only environment where using the OpenAppID rules make any sense. They are not designed for a home network.
If you are an experienced IDS/IPS admin and know how to tune a rule set, then put Snort on your LAN and enable an IPS Policy. I usually recommend folks start with "Connectivity" and only when they are really proficient should they try "Balanced". That is the max. Never use "Security" or the other "Max Detect" policy on a home network! Those are for protecting maybe the CIA or NSA servers and for testing in a lab. They are not for any typical home or enterprise network! Enabling those will drive you crazy with needless blocks due to false positives.
So to answer your question, I have zero rules on my WAN and only the IPS Policy "Balanced" on my LAN. I've tuned the HTTP_INSPECT preprocessor rules to eliminate false positives and disabled a tiny handful of other preprocessor rules and that's it. I get maybe three to six alerts per quarter on my LAN. I've had only 2 alerts on my LAN this month (for April). One on April 10 and the other on April 17. Both were false positives from a portscan rule triggered by my iPhone and my wife's iPhone interracting with Facebook's CDN. So far this entire year, I've had exactly 5 alerts triggered on my LAN. This is how it should be when you have a well-tuned IDS/IPS rule set that matches the actual attack surfaces in your network. You don't need to enable every rule. Many of the rules are to protect traffic that your network would never see. Why would you want web server and web server rules enabled when you don't have a web server or mail server on your network accepting inbound traffic from the Internet?
-
@bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
I really have no rules on the WAN. Snort is not needed there. As I mentioned before, the only I have are some blacklisted IP rules on my WAN just to generate alert logs so I have something test with when working on new versions of the Snort package. So yes, I do have two categories of rules on my WAN, but they are not for security. They are to generate data for testing. If I did not need the testing data, I would remove those rules and have nothing on my WAN.
On my LAN I do not use the OpenAppID rules. They are fundamentally useless on a home network. Those rules are designed for the corporate enterprise world where you don't want employees wasting time and the company's network bandwidth engaging in social media chit-chat and streaming music or movies. That's really the only environment where using the OpenAppID rules make any sense. They are not designed for a home network.
If you are an experienced IDS/IPS admin and know how to tune a rule set, then put Snort on your LAN and enable an IPS Policy. I usually recommend folks start with "Connectivity" and only when they are really proficient should they try "Balanced". That is the max. Never use "Security" or the other "Max Detect" policy on a home network! Those are for protecting maybe the CIA or NSA servers and for testing in a lab. They are not for any typical home or enterprise network!
So to answer your question, I have zero rules on my WAN and only the IPS Policy "Balanced" on my LAN. I've tuned the HTTP_INSPECT preprocessor rules to eliminate false positives and disabled a tiny handful of other preprocessor rules and that's it. I get maybe three to six alerts per quarter on my LAN. I've had only 2 alerts on my LAN this month (for April). One on April 10 and the other on April 17. Both were false positives from a portscan rule triggered by my iPhone and my wife's iPhone interracting with Facebook's CDN. So far this entire year, I've had exactly 5 alerts triggered on my LAN. This is how it should be when you have a well-tuned IDS/IPS rule set that matches the actual attack surfaces in your network. You don't need to enable every rule. Many of the rules are to protect traffic that your network would never see. Why would you want web server and web server rules enabled when you don't have a web server or mail server on your network accepting inbound traffic from the Internet?
Hmmm....That's interesting. Most of the guides I've viewed show to enable IPS on the WAN. And the screenshot I sent you was directly from netgate.
So in doing your method under Global settings which rules did you have checked ?
Wouldn't the network be more protected in using the WAN interface?
Sorry for my ignorance in the matter. And I do appreciate all of your help!
-
@cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
Hmmm....That's interesting. Most of the guides I've viewed show to enable IPS on the WAN. And the screenshot I sent you was directly from netgate.
So in doing your method under Global settings which rules did you have checked ?
Wouldn't the network be more protected in using the WAN interface?
Sorry for my ignorance in the matter. And I do appreciate all of your help!
Many of those guides are quite old and really were originally written from an enterprise network point of view where you may not necessarily be using NAT and where public-facing services are more common.
In a home network I will venture out there and say NEVER put Snort or any IDS/IPS on the WAN. No need. It does nothing at all for security because the WAN will already block all unsolicited inbound traffic by default. The WAN will accept no incoming packet unless it is a stateful reply to a previously established session that was initiated by an internal host. Putting the IDS/IPS on your LAN and other internal interfaces (if desired) is all you need. Any traffic coming from and going to internal hosts must traverse the internal interfaces on the firewall. So putting the IDS/IPS there is sufficient.
The rules I have enabled on the GLOBAL SETTINGS tab are just the Snort Subscriber Rules. When you use those, you don't need the Snort GPLv2 Community Rules as every one of those is already in the Snort Subscriber Rules set.
Oh, I forgot. I did enable the ET-Open rules simply to get access to two of their IP blacklists. Those are the two I use only for testing on my WAN to generate that test data. I would not use the ET-Open rules otherwise.
-
Oh, and in looking over the PHP code for that cron task while getting ready to modify it for debug mode, I think I found the issue with the directory cleanup. The code is not looking for the rotated alert logs (the files with alert.<timestamp> in their names), and thus it does not clean those up.
You are seeing the issue because you are logging so much data due to having all those rules enabled. Turn that stuff off and you should see your alert log generation go way down. I will work on a fix and post it a bit later for everyone as a Snort package update.
It's not affecting a large number of folks because most will not be generating nearly as much logging traffic as you are with all those enabled rules alerting. You must have some terribly busy folks on your home LAN visiting social media and streaming sites to generate that much data ... .
-
Okay. I have made some changes so we shall see how this goes. Thanks again. I do still want the new script whenever you can. Thanks
-
@cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
Okay. I have made some changes so we shall see how this goes. Thanks again. I do still want the new script whenever you can. Thanks
I will make an update to the package and post it for the pfSense team to review and merge, so it will be a few days before that is done.
-
The fix for the Logs Management task skipping rotated alert log files has been submitted to both the 2.4.5-RELEASE and 2.5-DEVEL branches of pfSense for the developer team to review and merge.
I also updated the underlying Snort binary to the latest 2.9.16 version released by the Snort team.
Look for the updated Snort package to appear in the near future.
-
The new Snort package version containing the fix for this issue has been posted. It should show up as Snort v3.2.9.11 on pfSense-2.4.5 machines.
-
Just wanted to report that it looks like things are working now. Thanks for the help!
-
@cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:
Just wanted to report that it looks like things are working now. Thanks for the help!
Great! Turns out there was an issue in the log purging code after all. I had just not ever hit it because my logging "rate" is low and my system was not generating as many logs. Also, I do not run OpenAppID and thus my personal system was not generating any of those logs at all.
Thanks for reporting the issue.