Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds

    Scheduled Pinned Locked Moved IDS/IPS
    36 Posts 5 Posters 5.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • NollipfSenseN
      NollipfSense
      last edited by

      This is Snort 4.0 default, and values might be different from yours...did you changed yours?

      Screen Shot 2020-03-18 at 8.03.44 PM.png

      pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
      pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        Enable a log directory size limit as well as log sizes to be sure duplicate files don't fill the drive.

        Steve

        1 Reply Last reply Reply Quote 1
        • C
          cpom1
          last edited by

          Hello,

          I am also having this issue. My snort LAN interface logs are filling my disk. I have turned on Auto Log Management and changed the values and the issue persists. I have to go in and manually clean out the logs of the LAN interface. It generates multiple alert logs.

          ecc855e0-d003-4003-8208-3de2d33c372e-image.png
          I literally just cleaned out this directory and it has generated all of those within 30 min.
          It's rather annoying. Is there a fix for this ?

          Below is my current Log Mgmt settings:
          838709d4-40b9-4442-9ef7-81aec4abc668-image.png

          1 Reply Last reply Reply Quote 0
          • bmeeksB
            bmeeks
            last edited by bmeeks

            What versions of pfSense and the Snort package are you running? Some issues with log rotation were corrected several versions back in the Snort package. To my knowledge, users are not reporting widespread issues with logs management on either Snort or Suricata in the current package versions.

            There have been a few isolated cases that were caused by things the users had done (or more aptly, not done).

            How much space do you have allocated for /var on your firewall? Are you by chance using a RAM disk? If so, don't do that with Snort!

            C 1 Reply Last reply Reply Quote 0
            • C
              cpom1 @bmeeks
              last edited by

              @bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:

              What versions of pfSense and the Snort package are you running? Some issues with log rotation were corrected several versions back in the Snort package. To my knowledge, users are not reporting widespread issues with logs management on either Snort or Suricata in the current package versions.

              There have been a few isolated cases that were caused by things the users had done (or more aptly, not done).

              How much space do you have allocated for /var on your firewall? Are you by chance using a RAM disk? If so, don't do that with Snort!

              Hello and thank you for the quick response. I am using PFsense 2.4.5 and Snort 3.2.9.10_4. I am not using a RAM disk. Where would i check how space I have allocated for /var? I don't think i have ever set anything...

              1 Reply Last reply Reply Quote 0
              • bmeeksB
                bmeeks
                last edited by

                The quickest way is to look on the pfSense Dashboard page down towards the bottom. It will show you the amount of used and allocated space for / (which is the root system volume). The /var/log/snort/ directory will be allocated from that space.

                I have an SG-5100 and currently it shows me using 25% of 6.7 GB. That 25% is all of my installed software plus logs (including Snort logs). I have Snort running on three interfaces.

                C 1 Reply Last reply Reply Quote 0
                • C
                  cpom1 @bmeeks
                  last edited by

                  @bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:

                  The quickest way is to look on the pfSense Dashboard page down towards the bottom. It will show you the amount of used and allocated space for / (which is the root system volume). The /var/log/snort/ directory will be allocated from that space.

                  I have an SG-5100 and currently it shows me using 25% of 6.7 GB. That 25% is all of my installed software plus logs (including Snort logs). I have Snort running on three interfaces.

                  It shows 43GiB

                  1 Reply Last reply Reply Quote 0
                  • bmeeksB
                    bmeeks
                    last edited by bmeeks

                    If you have that much empty space on the system volume, then it sounds like the Snort log cleanup cron task is not executing, or else permissions are messed up somehow in the logging directory.

                    Are there any messages in the pfSense system log from Snort? The log cleanup job executes every 5 minutes if I remember correctly. So if your Snort logging path size currently exceeds the configured max value, that cron task should print some data about what it's doing to the system log every 5 minutes. Look and see if anything is being logged or if any error messages are showing up.

                    Look in the /etc/crontab file and see if the snort_check_cron_misc.inc file is showing in the list of executable jobs.

                    I am running the same version of pfSense and Snort as you and my logging directory is being kept below the configured limit. I have no log files in the directory older than 14 days (my retention limit setting) and the total size in bytes is below the 1340 MB limit I have configured. So the Logs Management process is working for me.

                    C 1 Reply Last reply Reply Quote 0
                    • bmeeksB
                      bmeeks
                      last edited by bmeeks

                      How much traffic is going through Snort? Just noticed the timestamps on those alert logs. You are generating a new file every few seconds! Your limit per log file in the interface says 500 KB. How big are the actual alert log files on the disk? Are they also 500 KB?

                      You have something seriously wrong if your system is generating 500 KBs/few seconds of alert logs. Those numbers on the end of each log file are Unix timestamps, so you can look at how close together the values are and see that your system is creating new files every few seconds.

                      C 1 Reply Last reply Reply Quote 0
                      • C
                        cpom1 @bmeeks
                        last edited by

                        @bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:

                        snort_check_cron_misc.inc

                        ok. i checked that directory and i do not have snort_check_cron_misc.inc file. Idk what happened to it or if there was ever one.

                        bmeeksB 1 Reply Last reply Reply Quote 0
                        • C
                          cpom1 @bmeeks
                          last edited by

                          @bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:

                          How much traffic is going through Snort? Just noticed the timestamps on those alert logs. You are generating a new file every few seconds! Your limit per log file in the interface says 500 KB. How big are the actual alert log files on the disk? Are they also 500 KB?

                          You have something seriously wrong if your system is generating 500 KBs/few seconds of alert logs. Those numbers on the end of each log file are Unix timestamps, so you can look at how close together the values are and see that your system is creating new files every few seconds.

                          This is the LAN interface logs so anytime someone browses or anything.

                          I have two Snort interfaces; WAN and LAN.

                          1 Reply Last reply Reply Quote 0
                          • bmeeksB
                            bmeeks @cpom1
                            last edited by bmeeks

                            @cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:

                            @bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:

                            snort_check_cron_misc.inc

                            ok. i checked that directory and i do not have snort_check_cron_misc.inc file. Idk what happened to it or if there was ever one.

                            No, it won't be a file in that directory. It will be one of several entries (lines) shown inside of the etc/crontab file. Go to DIAGNOSTICS > EDIT FILE and browse to and open the /etc/crontab file. Just look at its contents. Don't change anything or you can mess up your firewall. Just look at the lines and see if one contains the snort file I mentioned.

                            That file is installed as a periodic crontask.

                            Here is the bottom half of my file as an example:

                            crontab_example.png

                            The "*/5" means the file /usr/local/pkg/snort/snort_check_cron_misc.inc is executed every 5 minutes. That PHP file contains the code that does the logs management for Snort.

                            But you have another more serious problem like some runaway process or something if your firewall is generating that many alert log files in 30 minutes. I would suggest rebooting your firewall.

                            On a typical home network it should take at least several weeks to generate enough alert logs to fill up 1 GB of space.

                            C 1 Reply Last reply Reply Quote 0
                            • C
                              cpom1 @bmeeks
                              last edited by

                              @bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:

                              /etc/crontab

                              Ok I located it.

                              11137949-05af-40d9-8fdd-7975a9cf8b13-image.png

                              Do you have LAN setup on your Snort service?

                              Even when i have rebooted the service, the LAN generates multiple alert logs. And it doesn't seem like its getting overwritten or cleaned out by the log management.

                              I do have my WAN with Send Alerts to System Log enabled but LAN. Do you have that turned on ?

                              bmeeksB 1 Reply Last reply Reply Quote 0
                              • bmeeksB
                                bmeeks @cpom1
                                last edited by

                                @cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:

                                @bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:

                                /etc/crontab

                                Ok I located it.

                                11137949-05af-40d9-8fdd-7975a9cf8b13-image.png

                                Do you have LAN setup on your Snort service?

                                Even when i have rebooted the service, the LAN generates multiple alert logs. And it doesn't seem like its getting overwritten or cleaned out by the log management.

                                I do have my WAN with Send Alerts to System Log enabled but LAN. Do you have that turned on ?

                                Yes, I have logging enabled for all my Snort interfaces. I run Snort on the WAN using a few IP blacklists solely to generate logging data for me to test new Snort package updates with.

                                Are you on a home network? If so, there is absolutely no way you should be generating enough logs to fill up a 43 GB disk in 30 minutes. Even if you are not filling up the disk, there is no way in just 30 minutes you should be generating enough logs to cause you any problems whatsoever.

                                I seriously suggest rebooting your firewall to be sure you don't have some zombie process out there churning out logs.

                                You can also try removing and then reinstalling the Snort package. You won't lose any settings by doing that. All of your configuration will be preserved.

                                C 1 Reply Last reply Reply Quote 0
                                • C
                                  cpom1 @bmeeks
                                  last edited by

                                  @bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:

                                  @cpom1 said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:

                                  @bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:

                                  /etc/crontab

                                  Ok I located it.

                                  11137949-05af-40d9-8fdd-7975a9cf8b13-image.png

                                  Do you have LAN setup on your Snort service?

                                  Even when i have rebooted the service, the LAN generates multiple alert logs. And it doesn't seem like its getting overwritten or cleaned out by the log management.

                                  I do have my WAN with Send Alerts to System Log enabled but LAN. Do you have that turned on ?

                                  Yes, I have logging enabled for all my Snort interfaces. I run Snort on the WAN using a few IP blacklists solely to generate logging data for me to test new Snort package updates with.

                                  Are you on a home network? If so, there is absolutely no way you should be generating enough logs to fill up a 43 GB disk in 30 minutes. Even if you are not filling up the disk, there is no way in just 30 minutes you should be generating enough logs to cause you any problems whatsoever.

                                  I seriously suggest rebooting your firewall to be sure you don't have some zombie process out there churning out logs.

                                  You can also try removing and then reinstalling the Snort package. You won't lose any settings by doing that. All of your configuration will be preserved.

                                  Oh I'm sorry. I didn't mean that it fills up the disk that fast but overtime it does fill up the disk without no cleanup. Maybe monthly or two or so my disk will fill up with these logs.

                                  Yes, this is a home network.

                                  Do you have your logs being sent to system log on either or both of your interfaces?

                                  1 Reply Last reply Reply Quote 0
                                  • bmeeksB
                                    bmeeks
                                    last edited by

                                    No, I don't send any Snort alert logs to the pfSense system log.

                                    Get to a shell prompt on your firewall and execute this command to force the log cleanup task to run:

                                    php -f /usr/local/pkg/snort/snort_check_cron_misc.inc
                                    

                                    Then go look in the pfSense system log to see if anything is logged there. If your directory has files that exceed the limits shown on the LOG MGMT tab, then you should see some log messages showing a cleanup in progress.

                                    If you notice any weird messages on the console when you execute the command above, come back and post those.

                                    1 Reply Last reply Reply Quote 0
                                    • bmeeksB
                                      bmeeks
                                      last edited by

                                      The symptoms you describe indicate the log cleanup job is not running or else is generating some kind of error.

                                      I've never ever had Snort fill up my disk. And like I mentioned at the top of this exchange, the log management has to be working or else there would be lots of folks here complaining. There are over 24,000 Snort and Suricata installs on pfSense.

                                      C 1 Reply Last reply Reply Quote 0
                                      • C
                                        cpom1 @bmeeks
                                        last edited by

                                        @bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:

                                        The symptoms you describe indicate the log cleanup job is not running or else is generating some kind of error.

                                        I've never ever had Snort fill up my disk. And like I mentioned at the top of this exchange, the log management has to be working or else there would be lots of folks here complaining. There are over 24,000 Snort and Suricata installs on pfSense.

                                        okay so i ran that command and i cant even tell it executed; i didnt see anything in my system log. i ran it from putty as well as the pfsense shell command.

                                        1 Reply Last reply Reply Quote 0
                                        • bmeeksB
                                          bmeeks
                                          last edited by bmeeks

                                          Normally you won't see anything. And if the files currently in the log directory are below the limits (total megabytes space consumption and retention period), then nothing happens. Only when the log directory is beyond the limits does something happen.

                                          So give it a few days or weeks or however long it takes to accumulate more than 1 GB of files in the directory (specifically, 1024 MB as that is your currently configured limit) and then try running the command again manually.

                                          You might also have some zombie Snort process hanging on as well. I still suggest you reboot the firewall to be sure.

                                          C 2 Replies Last reply Reply Quote 0
                                          • C
                                            cpom1 @bmeeks
                                            last edited by

                                            @bmeeks said in Snort fills up disk with logs, Dashboard goes bye bye, interfaces lose their minds:

                                            Normally you won't see anything. And if the files currently in the log directory are below the limits (total megabytes space consumption and retention period), then nothing happens. Only when the log directory is beyond the limits does something happen.

                                            So give it a few days or weeks or however long it takes to accumulate more than 1 GB of files in the directory (specifically, 1024 MB as that is your currently configured limit) and then try running the command again manually.

                                            You might also have some zombie Snort process hanging on as well. I still suggest you reboot the firewall to be sure.

                                            okay. thanks for all your help. I will reboot and monitor and report back. Thanks again! :D

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.