Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    /var is low on disk space

    General pfSense Questions
    6
    40
    4.3k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J
      JonH
      last edited by

      SG-5100 w/ 8G sdd, running 23.05.1 on zfs.
      After upgrading from 23.05 my system is not running as smoothly as it was on 23.05 and earlier.

      I suspect my problem is low disk space, I've seen at least one error in pfBlockerNG referring to cannot write to disk.
      Now I notice my Disks Widget warning me about utilization threshold. I don't run Snort or any similar pkgs, my biggy is pfBlockerNG.
      I also run syslog-ng and my log files are set to keep 2 zipped archives which are in /var/log and all of them total only ~2.5MB.

      pfSense Disks.png

      /var/cache/pkg at 96% with the largest file pfSense-base-23.05.1~46a4a1f8e1.pkg at 141MB.

      I know this is not a big number but the level of utilization concerns me.

      My problem started with loss of connectivity on wifi (ubiquity AP on IGB1) but wired desktop worked fine. Restarting the AP didn't fix it so I restarted pfSense.
      That fixed the problem, until the next day when I discovered errors in pfBlockerNG and restarted pfSense again. pfBlocker uses space in /var for writing log files and presumably also writing temp files.

      I'm wondering if I can remove that base-23.05.1 pkg although I presume that is not wise. I am paying more attention to the size of /var but would feel better if I could free up more space.

      S 1 Reply Last reply Reply Quote 0
      • S
        SteveITS Galactic Empire @JonH
        last edited by

        @JonH Delete old System/Boot Environments:
        https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-shrink.html
        That’s easy to do.

        Are you using a RAM disk for /var? 50 MB seems rather low.

        Run du -h -d 1 /var to view usage.

        Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
        When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
        Upvote 👍 helpful posts!

        J 1 Reply Last reply Reply Quote 0
        • J
          JonH @SteveITS
          last edited by

          @SteveITS I don't know if I'm running a ram disk. It's basically a default install from Netgate, it's their disk layout. Thanks for boot environments suggestion. I forgot about that one. And I failed to see that filesystem-shrink in the doc's but did notice an impressive amount of info in that section.

          #du -h -d 1 /var
          
          512B	/var/games
          512B	/var/account
          3.1M	/var/dhcpd
          512B	/var/empty
          512B	/var/msgs
          4.7M	/var/log
          512B	/var/preserve
           14K	/var/mail
           39K	/var/tmp
          495M	/var/unbound
           12K	/var/spool
          512B	/var/authpf
          198M	/var/cache
          6.2M	/var/syslog-ng
           80M	/var/db
          1.0K	/var/lib
          512B	/var/crash
          512B	/var/yp
          9.8M	/var/backups
           40K	/var/etc
          135K	/var/run
          1.5K	/var/audit
          512B	/var/heimdal
          512B	/var/rwho
          1.5K	/var/at
          1.0K	/var/cron
          797M	/var
          
          S JonathanLeeJ 2 Replies Last reply Reply Quote 0
          • S
            SteveITS Galactic Empire @JonH
            last edited by

            @JonH You’d have to set a RAM disk in System/Advanced/Miscellaneous. Plus it would be tmpfs not ZFS. It just seems odd it’s 51 MB. But I don’t have access to a 5100.

            Did you reinstall, and change partitioning maybe? With ZFS I think it will reduce the “disk capacity” a la that doc page.

            My 2100 has /var/db and /var/cache/pkg at 4GB.

            Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
            When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
            Upvote 👍 helpful posts!

            J 1 Reply Last reply Reply Quote 0
            • J
              JonH @SteveITS
              last edited by

              @SteveITS said in /var is low on disk space:

              Did you reinstall, and change partitioning maybe? With ZFS I think it will reduce the “disk capacity” a la that doc page.

              There is no RAM disk active in advanced->misc.

              Last year I requested from Netgate a download to update my system to zfs. Unfortunately I cannot locate the details but suffice to say there have been a number of updates from Netgate since then and other than awhile ago having the DNS issue that some had, everything has been hunky-dorie until now.

              A few days after 23.05.1 was released I installed it. A few days after that I found my Apple mobile devices (also recently updated) unable to connect (no route to host). Checking my resolver.log shows no activity although I still have connectivity via wired desktop.

              I still have not clearly identified the culprit, be it a switch, an AP, pfSense, or pfBlocker or something else.

              I'll read that doc you linked to before I fool with removing boot environment files. The sizes of those files seem insignificant to my problem unless you are saying to remove the default environment also. I presume it is needed for booting but maybe this is essentially a backup to easily reinstall the system? I won't touch that until I get more info.

              BootEnviro.png

              I am planning to force a reinstall of pfBlocker before I do anything else.

              My AP's are 2 Unifi (and 1 older Asus). Unifi has a separate web interface with a dashboard that shows reliability. That indicates a loss of 'wifi experience' at 0015 hrs, the exact time pfBlocker runs an update. I cannot discount that perhaps the AP is about to croak and I'll fool around with that later in the week. My unifi also sends me a msg that a 'rogue AP is near my Unifi' (yes, that would be the Asus). All of my AP's have the same SSID to allow easy roaming throughout my property. Today I disabled the Asus radio's so I'll see if the situation changes in the next 24 hrs. So it's possible that my problem is not with pfSense but I'm still concerned by the very low free space showing in /var.

              S 1 Reply Last reply Reply Quote 0
              • S
                SteveITS Galactic Empire @JonH
                last edited by

                @JonH The boot environments may not help, they are part of ZFS and not in var. just a quick way to free space in general. I haven’t had many with ZFS as most of our clients are older but yesterday after upgrading at home I deleted a 23.01 line showing “32k” and it freed about 200 MB space per the dashboard widget. Don’t delete the one you’re using…I would hope it stops you.

                Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                Upvote 👍 helpful posts!

                J 1 Reply Last reply Reply Quote 0
                • J
                  JonH @SteveITS
                  last edited by

                  @SteveITS said in /var is low on disk space:

                  after upgrading at home I deleted a 23.01 line showing “32k” and it freed about 200 MB space per the dashboard widget.

                  Thanks! That helped. It's like taking off my socks and I can wiggle my toes again 👍

                  I have not gotten to reinstalling pfBlockerNG (maybe it's not a problem).
                  I removed a 23.01 from March (152k) and 22.05 from Feb (24k) from the environment.
                  It removed all my usage warnings and I my low disk space issue is solved.

                  Now to see what happens with my mobile devices over the next few days.

                  S 1 Reply Last reply Reply Quote 0
                  • S
                    SteveITS Galactic Empire @JonH
                    last edited by

                    @JonH Nice. Does the dashboard still show /var at 51 MB?

                    Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                    When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                    Upvote 👍 helpful posts!

                    J 1 Reply Last reply Reply Quote 0
                    • J
                      JonH @SteveITS
                      last edited by

                      @SteveITS 975MB, nearly 20x increase. No more red bars 😃

                      new disks.png

                      R S 2 Replies Last reply Reply Quote 0
                      • R
                        rcoleman-netgate Netgate @JonH
                        last edited by

                        @JonH We do recommend checking the eMMC health in our devices over time to make sure you do not exhaust the writes allowed on the storage before you've had a chance to add an SSD. Waiting until the eMMC has failed, on this model, may result in a system that cannot be booted at all when you get an SSD.

                        https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html#emmc

                        Ryan
                        Repeat, after me: MESH IS THE DEVIL! MESH IS THE DEVIL!
                        Requesting firmware for your Netgate device? https://go.netgate.com
                        Switching: Mikrotik, Netgear, Extreme
                        Wireless: Aruba, Ubiquiti

                        J 2 Replies Last reply Reply Quote 0
                        • J
                          JonH @rcoleman-netgate
                          last edited by

                          @rcoleman-netgate said in /var is low on disk space:

                          We do recommend checking the eMMC health in our devices

                          Thanks so much for this reminder. I looked at s.m.a.r.t. status but I guess it applies to scsi drive types.

                          I'll check out the link.

                          1 Reply Last reply Reply Quote 0
                          • S
                            SteveITS Galactic Empire @JonH
                            last edited by

                            @JonH interesting. That implies the issue could be 1) completely hidden by a large RAM disk, and 2) the RAM disk might not be able to copy 200MB of data to a “51MB” partition in the background.

                            Perhaps the self-cleaning of old BEs should happen sooner if /var is under a certain amount of total space. Or available space.

                            Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                            When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                            Upvote 👍 helpful posts!

                            J 1 Reply Last reply Reply Quote 0
                            • J
                              JonH @rcoleman-netgate
                              last edited by

                              @rcoleman-netgate said in /var is low on disk space:

                              Waiting until the eMMC has failed, on this model, may result in a system that cannot be booted at all when you get an SSD

                              My results are:

                              eMMC Firmware Version: ?
                              eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x0b
                              eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x0b
                              eMMC Pre EOL information [EXT_CSD_PRE_EOL_INFO]: 0x01

                              So 0x0b is "The disk has used 100%-110% of its estimated life time" and somewhat concerning.
                              and 0x01 is "The disk has consumed less than 80% of its reserved blocks" which is somewhat less concerning.

                              At any rate, I was planning on Monday checking with old-store.netgate to inquire about the feasibility of installing an M.2 2242 in my 5100, mainly because I want to run ntopng without continually removing the data and also have more logging storage. This 5100 replaced a failed 2440 which had a 20GB (I think) SSD and I never had to worry about space.

                              Although I was aware that SDD had an eventual fail possibility I never thought it could be so soon. All mechanical drives had an eventual fail possibility and I've yet to have that happen in 40+ years of various systems. So I probably have been too shortsighted in this possibility. The test I just ran shows me that the 8GB with logging, especially the filter.log, probably shortens the life considerably. In fact, maybe that is what happened to my 2440 and my previous 5100 that lasted 3 yrs (2440) and only 18 mo (first 5100). They both simply do not boot nor provide an output via the serial cable. I always ran a lot of logging & ntopng.

                              Thanks again for this heads-up.

                              S R 2 Replies Last reply Reply Quote 0
                              • S
                                SteveITS Galactic Empire @JonH
                                last edited by SteveITS

                                @JonH depends heavily on logging and packages, see
                                https://www.netgate.com/supported-pfsense-plus-packages
                                Edit: yeah ntopng, there you go, didn’t read down that far yet :)

                                Also:
                                https://docs.netgate.com/pfsense/en/latest/solutions/sg-5100/m-2-sata-installation.html

                                Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                                When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                                Upvote 👍 helpful posts!

                                1 Reply Last reply Reply Quote 0
                                • J
                                  JonH @SteveITS
                                  last edited by

                                  @SteveITS said in /var is low on disk space:

                                  Perhaps the self-cleaning of old BEs should happen sooner

                                  Indeed. I have been quite negligent in reviewing various changes and feature additions.
                                  I did see that BEs existed but never gave it much thought in the past. Now I also see that my negligence has poured over to SDD lifetime.

                                  I'm in my mid-70's now and keeping up with tech has become quite difficult. And with the amount of security patches for various devices coming fast and furious of late I now see that it will be difficult if not impossible to keep a system running for my own expected lifetime without needing to upgrade the hardware (I mostly use an Apple Intel machine which is going to be unsupported in the very near future).

                                  1 Reply Last reply Reply Quote 0
                                  • R
                                    rcoleman-netgate Netgate @JonH
                                    last edited by rcoleman-netgate

                                    @JonH said in /var is low on disk space:

                                    So 0x0b is "The disk has used 100%-110% of its estimated life time" and somewhat concerning.
                                    and 0x01 is "The disk has consumed less than 80% of its reserved blocks" which is somewhat less concerning.

                                    Get an m.2 SSD ASAP.

                                    Any mSATA m.2 SSD will work. https://docs.netgate.com/pfsense/en/latest/solutions/sg-5100/m-2-sata-installation.html

                                    Do not wait. if the eMMC on this goes read-only the only method you have (that I have determined so far) is to use a re-soldering station to heat the eMMC and remove it from the board. it's messy. I haven't gotten a second 5100 with a read-only eMMC yet to see if there's another option to block the BIOS from seeing the chip.

                                    Ryan
                                    Repeat, after me: MESH IS THE DEVIL! MESH IS THE DEVIL!
                                    Requesting firmware for your Netgate device? https://go.netgate.com
                                    Switching: Mikrotik, Netgear, Extreme
                                    Wireless: Aruba, Ubiquiti

                                    J 2 Replies Last reply Reply Quote 0
                                    • J
                                      JonH @rcoleman-netgate
                                      last edited by

                                      @rcoleman-netgate said in /var is low on disk space:

                                      Get an m.2 SSD ASAP.

                                      Thanks!

                                      A 1 Reply Last reply Reply Quote 0
                                      • A
                                        azdeltawye @JonH
                                        last edited by azdeltawye

                                        @JonH said in /var is low on disk space:

                                        @rcoleman-netgate said in /var is low on disk space:
                                        Get an m.2 SSD ASAP.

                                        I strongly agree with that advice. My SG-5100 exhibited this same behavior when I upgraded to 23.01 back in March. I installed a compatible SSD within a couple days but unfortunately it was too late. The eMMC had completely failed at that point and prevented the unit from booting. My beloved SG-5100 was a boat anchor after only 20 months of use!

                                        And the sad thing is, it really wasn't used that hard. Just a simple home environment with 40 or so client devices. The only write-heavy package installed was pfBlockerng.

                                        I would highly recommend that you get a backup plan together in case your unit suffers the same fate. I had to scramble and go to Best Buy to get a POS consumer-grade Asus router just to get back online. Needless to say, the whole ordeal was pretty annoying an stressful. I work from home and this happened in the middle of a busy week. The wife wasn't too happy either...

                                        Best of luck.

                                        R J 2 Replies Last reply Reply Quote 0
                                        • R
                                          rcoleman-netgate Netgate @azdeltawye
                                          last edited by

                                          @azdeltawye said in /var is low on disk space:

                                          I strongly agree with that advice. My SG-5100 exhibited this same behavior when I upgraded to 23.01 back in March. I installed a compatible SSD within a couple days but unfortunately it was too late. The eMMC had completely failed at that point and prevented the unit from booting.

                                          Unfortunately for @azdeltawye, but (and I think I speak for both of us here) we hope to be fortune for others, this very same 5100 is the one I mentioned above where I extracted the eMMC from the board and it immediately started perform normally.

                                          It's not a process I recommend everyone to do -- but for those who have the skills it can be done.

                                          This is the eMMC on the board (the bottom side) IMG_2268 Large.jpeg

                                          The board removed using a re-soldering station (a programmable heat gun)
                                          IMG_2269 Large.jpeg

                                          The empty space (partially cleaned up)
                                          IMG_2272 Large.jpeg

                                          The bottom of the eMMC.
                                          IMG_2274 Large.jpeg

                                          Ryan
                                          Repeat, after me: MESH IS THE DEVIL! MESH IS THE DEVIL!
                                          Requesting firmware for your Netgate device? https://go.netgate.com
                                          Switching: Mikrotik, Netgear, Extreme
                                          Wireless: Aruba, Ubiquiti

                                          A 1 Reply Last reply Reply Quote 1
                                          • J
                                            JonH @azdeltawye
                                            last edited by

                                            @azdeltawye said in /var is low on disk space:

                                            My beloved SG-5100 was a boat anchor after only 20 months of use!

                                            Indeed. I got 16 mo's out of my 1st 5100, this one is going on 27 mo.
                                            Same as you, my biggest write-hvy is pfBlockerNG. I am also simple home w/4 active devices but up to 30 because of smart devices.

                                            I was thinking of getting a SSD just to have more room, I had no idea how fragile the SSD was until now. I knew it was limited but didn't think it was this limited. A few minutes ago I cut my pfBlocker updates from 4x/day to once.

                                            Thanks

                                            S 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.