Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2)

    Scheduled Pinned Locked Moved General pfSense Questions
    20 Posts 7 Posters 1.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      slu @w0w
      last edited by

      @w0w
      thanks for your reply, that's interesting.

      On my smaller network (only ~ 8 systems online) I see write throughput of approximately 10. But on my bigger network (screenshot above) its continuity around 1400 write throughput.

      pfSense Gold subscription

      w0wW 1 Reply Last reply Reply Quote 0
      • bmeeksB
        bmeeks @slu
        last edited by

        @slu said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):

        Do SNORT write on the disk to check the data flow @bmeeks ?

        No, Snort only writes what you see in the alert logs. It does no temporary writes other than when downloading and unpacking rules files updates. Those happen under /tmp and are cleaned up when the rules update completes. Snort logs are under /var/log/snort/.

        S 1 Reply Last reply Reply Quote 1
        • S
          slu @bmeeks
          last edited by

          @bmeeks said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):

          No, Snort only writes what you see in the alert logs.

          Thank you @bmeeks that confirmed what I see in top -m io with the Snort process.

          pfSense Gold subscription

          1 Reply Last reply Reply Quote 0
          • w0wW
            w0w @slu
            last edited by

            @slu
            Long-term monitoring of disk writes still showed that a significant portion of them is performed unbound. I'm not sure what percentage of the records are unbound, but it is clear that it's substantial, though I don't know how to track it precisely. The Samsung SSD 860 PRO 256GB drives are in a ZFS mirror, and 88% of the resource remains, which is generally non-critical, but...

            1 Reply Last reply Reply Quote 1
            • GertjanG
              Gertjan @slu
              last edited by Gertjan

              @slu said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):

              Any hint to disable the logging?

              Just checking :

              0bef5076-93bd-4ba2-ab41-273ab7a766cb-image.png

              = unbound log level setting : Right ?

              Level 3 and above logs a lot, and is only useful for temporary debug sessions. Setting it back to '1' is not 'optional'.

              Btw : logs files, also the /var/log/resolver.log file, are rotated by pfSense.
              My pfBlockerng log files (most of them are here /var/unbound/var/log/pfblockerng) are also rotated.
              I've never, over a decade now, saw a Tbytes file size on my pfSense ...

              No "help me" PM's please. Use the forum, the community will thank you.
              Edit : and where are the logs ??

              S 1 Reply Last reply Reply Quote 0
              • S
                slu @Gertjan
                last edited by slu

                @Gertjan said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):

                = unbound log level setting : Right ?

                Yes setting is "Level 1", I tried also "Level 0" but unbound write anyway according to "top -m io".

                Edit:
                @Gertjan said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):

                I've never, over a decade now, saw a Tbytes file size on my pfSense ...

                I do not have Tbytes file/log size on my pfSense, only see the SSD write TB's...

                pfSense Gold subscription

                w0wW 1 Reply Last reply Reply Quote 0
                • w0wW
                  w0w @slu
                  last edited by

                  @slu
                  I know this might not solve the root problem, but what about looking for a really long-lasting SSD? Maybe some of the older, reliable MLC or SLC variants? Which form factor do you have?

                  S 1 Reply Last reply Reply Quote 0
                  • S
                    SteveITS Galactic Empire @w0w
                    last edited by

                    @slu Or perhaps a RAM disk.

                    pfBlocker has some options that are on by default such as DNS Reply Logging to log all non-blocked queries (i.e. all valid DNS). Much like Suricata's HTTP request logging we disable that.

                    Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                    When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                    Upvote ๐Ÿ‘ helpful posts!

                    S 1 Reply Last reply Reply Quote 0
                    • S
                      slu @SteveITS
                      last edited by

                      @SteveITS said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):

                      pfBlocker has some options that are on by default such as DNS Reply Logging to log all non-blocked queries (i.e. all valid DNS).

                      I tried to disable this settings and reload pfBlockerNG + reboot pfSense, but unbound write and write again in python mode.

                      pfSense Gold subscription

                      S 1 Reply Last reply Reply Quote 0
                      • S
                        SteveITS Galactic Empire @slu
                        last edited by

                        @slu I don't know, have not noticed high disk writes. I have seen posts over the years though...here are a couple.

                        https://www.reddit.com/r/pfBlockerNG/comments/13di9c2/dnsbl_python_mode_and_disk_writes/
                        https://forum.netgate.com/topic/165993/should-i-be-using-unbound-python-mode-is-it-stable

                        Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                        When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                        Upvote ๐Ÿ‘ helpful posts!

                        S 1 Reply Last reply Reply Quote 0
                        • S
                          slu @SteveITS
                          last edited by

                          @SteveITS
                          thanks @SteveITS for the links.

                          Still not sure whats the root cause for this massive writes on the SSD, maybe I'm on the wrong way with pfBlockerNG and Unbound Python mode...

                          pfSense Gold subscription

                          w0wW 1 Reply Last reply Reply Quote 0
                          • w0wW
                            w0w @slu
                            last edited by

                            @slu
                            The settings can affect it. I just checked how often it updates. The Cron settings are set to once a day. This is probably significantly reducing the number of writes. Maybe there are some other settings that are affecting it as well.

                            S 1 Reply Last reply Reply Quote 0
                            • S
                              slu @w0w
                              last edited by

                              New topic here to investigate the issue since pfBlockerNG is not the cause.
                              https://forum.netgate.com/topic/189820/how-do-i-find-out-what-write-continuously-on-my-pfsense-ssd

                              pfSense Gold subscription

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post
                              Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.