Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfsense VM disk becomes full - please help identify the culprit?

    Scheduled Pinned Locked Moved Virtualization
    22 Posts 8 Posters 3.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • GertjanG
      Gertjan @sloopbun
      last edited by

      @sloopbun

      What pfSense packages are you using ?

      Have a look, at the /var/log/ folder, and all the /var/log/ sub folders.
      There are packages that log 'network' activity. If you have a lot of network activity, log files can grow to very fast, within a day, to several Gbytes. The solution would be : stop using the tools that log a lot, or don't have them log anymore.

      You /var/ :

      10G	/var
      

      I bet these 10G are mostly in /var/log/

      [23.09.1-RELEASE][root@pfSense.bhf.tld]/root: df -hi
      Filesystem                         Size    Used   Avail Capacity iused ifree %iused  Mounted on
      ......
      pfSense/var/log                     94G     86M     94G     0%     116  197M    0%   /var/log
      ......
      

      No "help me" PM's please. Use the forum, the community will thank you.
      Edit : and where are the logs ??

      S 1 Reply Last reply Reply Quote 0
      • S
        serbus @sloopbun
        last edited by

        @sloopbun

        Hello!

        I have seen something similar (?) when running a sparse guest fs on top of a thin-provisioned host fs.
        I dont run zfs/zvol on proxmox, so I dont know if this applies.

        https://forum.netgate.com/topic/166274/pfsense-hyper-v-vhdx-growing-like-crazy

        John

        Lex parsimoniae

        1 Reply Last reply Reply Quote 0
        • S
          sloopbun @Gertjan
          last edited by

          @Gertjan said in pfsense VM disk becomes full - please help identify the culprit?:

          @sloopbun

          What pfSense packages are you using ?

          Have a look, at the /var/log/ folder, and all the /var/log/ sub folders.
          There are packages that log 'network' activity. If you have a lot of network activity, log files can grow to very fast, within a day, to several Gbytes. The solution would be : stop using the tools that log a lot, or don't have them log anymore.

          You /var/ :

          10G	/var
          

          I bet these 10G are mostly in /var/log/

          [23.09.1-RELEASE][root@pfSense.bhf.tld]/root: df -hi
          Filesystem                         Size    Used   Avail Capacity iused ifree %iused  Mounted on
          ......
          pfSense/var/log                     94G     86M     94G     0%     116  197M    0%   /var/log
          ......
          

          Here are my packages, there is some logging, and I have deleted a bunch just to free up space, but the reported size of the /var/ directory does not balloon, just the lack of available space reported by df.

          Screenshot 2024-02-12 at 19.40.44.png

          @serbus said in pfsense VM disk becomes full - please help identify the culprit?:

          @sloopbun

          Hello!

          I have seen something similar (?) when running a sparse guest fs on top of a thin-provisioned host fs.
          I dont run zfs/zvol on proxmox, so I dont know if this applies.

          https://forum.netgate.com/topic/166274/pfsense-hyper-v-vhdx-growing-like-crazy

          John

          This is much more along the lines of what I have in mine. In proxmox, with zvol, there is a setting for "discard" which I have enabled.

          I note that the last reply on that thread was:

          @LesserBloops said in PfSense Hyper-V .vhdx growing like crazy:

          I hit this when I tried using ZFS as the file system with expanding disks. It went runaway and eventually caused an outage.

          I've been meaning to go back and try again with a fixed size disk instead of an expanding one, but haven't got around to it yet, and EXT seems to be fine...

          There are some parameters that I am not confident in the effects of such as volmode and reservation which I am hesitant to mess with with a recommendation to do so. I not that some parameters like logicalused and volsize represent the size of the disk that pfsense sees.

          I don't think this would seem to explain why the available space shown by df / progressively grows over time, without any file usage, but I would be happy to take advice on resolving this.

          The odd thing to me is that I thought this was a fairly standard practice, as explained in the docs here: https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html

          The fact that this is not a common issue points to it being a configuration issue on my behalf, right?

          root@pvepbs:~# zfs get all rpool/data/vm-100-disk-1
          NAME                      PROPERTY              VALUE                     SOURCE
          rpool/data/vm-100-disk-1  type                  volume                    -
          rpool/data/vm-100-disk-1  creation              Mon Jan  8 13:18 2024     -
          rpool/data/vm-100-disk-1  used                  33.5G                     -
          rpool/data/vm-100-disk-1  available             684G                      -
          rpool/data/vm-100-disk-1  referenced            33.5G                     -
          rpool/data/vm-100-disk-1  compressratio         3.58x                     -
          rpool/data/vm-100-disk-1  reservation           none                      default
          rpool/data/vm-100-disk-1  volsize               128G                      local
          rpool/data/vm-100-disk-1  volblocksize          16K                       default
          rpool/data/vm-100-disk-1  checksum              on                        default
          rpool/data/vm-100-disk-1  compression           on                        inherited from rpool
          rpool/data/vm-100-disk-1  readonly              off                       default
          rpool/data/vm-100-disk-1  createtxg             2282                      -
          rpool/data/vm-100-disk-1  copies                1                         default
          rpool/data/vm-100-disk-1  refreservation        none                      default
          rpool/data/vm-100-disk-1  guid                  7625546012721003177       -
          rpool/data/vm-100-disk-1  primarycache          all                       default
          rpool/data/vm-100-disk-1  secondarycache        all                       default
          rpool/data/vm-100-disk-1  usedbysnapshots       0B                        -
          rpool/data/vm-100-disk-1  usedbydataset         33.5G                     -
          rpool/data/vm-100-disk-1  usedbychildren        0B                        -
          rpool/data/vm-100-disk-1  usedbyrefreservation  0B                        -
          rpool/data/vm-100-disk-1  logbias               latency                   default
          rpool/data/vm-100-disk-1  objsetid              2013                      -
          rpool/data/vm-100-disk-1  dedup                 off                       default
          rpool/data/vm-100-disk-1  mlslabel              none                      default
          rpool/data/vm-100-disk-1  sync                  standard                  inherited from rpool
          rpool/data/vm-100-disk-1  refcompressratio      3.58x                     -
          rpool/data/vm-100-disk-1  written               33.5G                     -
          rpool/data/vm-100-disk-1  logicalused           119G                      -
          rpool/data/vm-100-disk-1  logicalreferenced     119G                      -
          rpool/data/vm-100-disk-1  volmode               default                   default
          rpool/data/vm-100-disk-1  snapshot_limit        none                      default
          rpool/data/vm-100-disk-1  snapshot_count        none                      default
          rpool/data/vm-100-disk-1  snapdev               hidden                    default
          rpool/data/vm-100-disk-1  context               none                      default
          rpool/data/vm-100-disk-1  fscontext             none                      default
          rpool/data/vm-100-disk-1  defcontext            none                      default
          rpool/data/vm-100-disk-1  rootcontext           none                      default
          rpool/data/vm-100-disk-1  redundant_metadata    all                       default
          rpool/data/vm-100-disk-1  encryption            off                       default
          rpool/data/vm-100-disk-1  keylocation           none                      default
          rpool/data/vm-100-disk-1  keyformat             none                      default
          rpool/data/vm-100-disk-1  pbkdf2iters           0                         default
          rpool/data/vm-100-disk-1  snapshots_changed     Mon Feb 12  9:32:58 2024  -
          
          S 1 Reply Last reply Reply Quote 0
          • S
            SteveITS Galactic Empire @sloopbun
            last edited by

            @sloopbun I remembered this thread https://forum.netgate.com/topic/183318/free-up-space-disk-storage-80/ with a similar report I think but it doesn't look like there was a resolution.

            Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
            When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
            Upvote 👍 helpful posts!

            S 1 Reply Last reply Reply Quote 0
            • S
              sloopbun @SteveITS
              last edited by

              @SteveITS said in pfsense VM disk becomes full - please help identify the culprit?:

              @sloopbun I remembered this thread https://forum.netgate.com/topic/183318/free-up-space-disk-storage-80/ with a similar report I think but it doesn't look like there was a resolution.

              In that thread, it is suggested that the command geom part list might reveal something, but it looks normal to me but maybe someone else can spot something?:

              [2.7.2-RELEASE][root@pfSense.spacelab]/root: geom part list
              Geom name: vtbd0
              modified: false
              state: OK
              fwheads: 16
              fwsectors: 63
              last: 268435415
              first: 40
              entries: 128
              scheme: GPT
              Providers:
              1. Name: vtbd0p1
                 Mediasize: 272629760 (260M)
                 Sectorsize: 512
                 Stripesize: 0
                 Stripeoffset: 20480
                 Mode: r1w1e1
                 efimedia: HD(1,GPT,616156c8-afd5-11ee-9ccd-e9881622351f,0x28,0x82000)
                 rawuuid: 616156c8-afd5-11ee-9ccd-e9881622351f
                 rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
                 label: (null)
                 length: 272629760
                 offset: 20480
                 type: efi
                 index: 1
                 end: 532519
                 start: 40
              2. Name: vtbd0p2
                 Mediasize: 132871356416 (124G)
                 Sectorsize: 512
                 Stripesize: 0
                 Stripeoffset: 272650240
                 Mode: r1w1e2
                 efimedia: HD(2,GPT,6161bcfd-afd5-11ee-9ccd-e9881622351f,0x82028,0xf77e000)
                 rawuuid: 6161bcfd-afd5-11ee-9ccd-e9881622351f
                 rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
                 label: (null)
                 length: 132871356416
                 offset: 272650240
                 type: freebsd-ufs
                 index: 2
                 end: 260046887
                 start: 532520
              3. Name: vtbd0p3
                 Mediasize: 4294926336 (4.0G)
                 Sectorsize: 512
                 Stripesize: 0
                 Stripeoffset: 133144006656
                 Mode: r1w1e1
                 efimedia: HD(3,GPT,616220f2-afd5-11ee-9ccd-e9881622351f,0xf800028,0x7fffb0)
                 rawuuid: 616220f2-afd5-11ee-9ccd-e9881622351f
                 rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
                 label: (null)
                 length: 4294926336
                 offset: 133144006656
                 type: freebsd-swap
                 index: 3
                 end: 268435415
                 start: 260046888
              Consumers:
              1. Name: vtbd0
                 Mediasize: 137438953472 (128G)
                 Sectorsize: 512
                 Mode: r3w3e7
              
              [2.7.2-RELEASE][root@pfSense.spacelab]/root: gpart show -p
              =>       40  268435376    vtbd0  GPT  (128G)
                       40     532480  vtbd0p1  efi  (260M)
                   532520  259514368  vtbd0p2  freebsd-ufs  (124G)
                260046888    8388528  vtbd0p3  freebsd-swap  (4.0G)
              

              After deleting some old logs and such, dushows just 4.8G of disk usage but df shows 26G used...

              [2.7.2-RELEASE][root@pfSense.spacelab]/root: du -h -d1 /
              4.0K	/.snap
              101M	/boot
              4.5K	/dev
               16M	/rescue
              4.0K	/proc
               96M	/root
              2.1G	/var
              4.0K	/media
               12K	/conf.default
              4.0K	/mnt
               13M	/tmp
              4.6M	/sbin
               17M	/lib
              4.0K	/net
               15M	/cf
              164K	/libexec
              8.0M	/etc
              1.4M	/bin
              1.8G	/usr
               20K	/home
              1.1M	/tftpboot
              4.8G	/
              
              [2.7.2-RELEASE][root@pfSense.spacelab]/root: df -hi
              Filesystem                     Size    Used   Avail Capacity iused ifree %iused  Mounted on
              /dev/ufsid/659ec5d8d33f9720    120G     26G     85G    23%     55k   16M    0%   /
              devfs                          1.0K      0B    1.0K     0%       0     0     -   /dev
              /dev/vtbd0p1                   260M    1.3M    259M     1%       2   510    0%   /boot/efi
              tmpfs                          4.0M    192K    3.8M     5%      56   14k    0%   /var/run
              
              S 1 Reply Last reply Reply Quote 0
              • S
                sloopbun @sloopbun
                last edited by

                I am not 100% sure, but I don't think I had this problem when I had pfsense on zfs, on top of the zvol in proxmox. I believe this kind of zfs on zfs is not recommended because it leads to write amplification, but it would be an improvement for me if another solution cannot be found.

                I could go back to that. Or I could go back to bare-metal - but I am using this proxmox machine for backing up my other proxmox machine, so I would prefer not to at this stage. There is of course the other downside of losing proxmox managed backups and snapshots, but that is not a deal breaker.

                S 1 Reply Last reply Reply Quote 0
                • P
                  Popolou @sloopbun
                  last edited by

                  @sloopbun said in pfsense VM disk becomes full - please help identify the culprit?:

                  Since I had posted, my disk usage had gone from 13G to 19G.

                  Where are you checking this from, within pfsense or via the hypervisor/proxmox?

                  S 1 Reply Last reply Reply Quote 0
                  • S
                    sloopbun @Popolou
                    last edited by sloopbun

                    @Popolou said in pfsense VM disk becomes full - please help identify the culprit?:

                    @sloopbun said in pfsense VM disk becomes full - please help identify the culprit?:

                    Since I had posted, my disk usage had gone from 13G to 19G.

                    Where are you checking this from, within pfsense or via the hypervisor/proxmox?

                    From within pfsense. For example, below is the progression over a few hours today, but I do not see any change in the zvol reported on the host, proxmox.

                    root@pvepbs:~# date
                    Tue Feb 13 08:45:13 AM CET 2024
                    
                    root@pvepbs:~# zfs list rpool/data/vm-100-disk-1
                    NAME                       USED  AVAIL  REFER  MOUNTPOINT
                    rpool/data/vm-100-disk-1  33.4G   684G  33.4G  -
                    
                    root@pvepbs:~# date
                    Tue Feb 13 03:14:46 PM CET 2024
                    
                    root@pvepbs:~# zfs list rpool/data/vm-100-disk-1
                    NAME                       USED  AVAIL  REFER  MOUNTPOINT
                    rpool/data/vm-100-disk-1  33.2G   684G  33.2G  -
                    
                    [2.7.2-RELEASE][root@pfSense.spacelab]/root: date
                    Tue Feb 13 08:45:13 CET 2024
                    
                    [2.7.2-RELEASE][root@pfSense.spacelab]/root: df -hi /
                    Filesystem                     Size    Used   Avail Capacity iused ifree %iused  Mounted on
                    /dev/ufsid/659ec5d8d33f9720    120G     26G     85G    23%     55k   16M    0%   /
                    
                    [2.7.2-RELEASE][root@pfSense.spacelab]/root: date
                    Tue Feb 13 15:07:25 CET 2024
                    
                    [2.7.2-RELEASE][root@pfSense.spacelab]/root: df -hi /
                    Filesystem                     Size    Used   Avail Capacity iused ifree %iused  Mounted on
                    /dev/ufsid/659ec5d8d33f9720    120G     32G     79G    29%     55k   16M    0%   /
                    
                    [2.7.2-RELEASE][root@pfSense.spacelab]/root: du -h -d 1 /
                    4.0K	/.snap
                    101M	/boot
                    4.5K	/dev
                     16M	/rescue
                    4.0K	/proc
                     96M	/root
                    2.2G	/var
                    4.0K	/media
                     12K	/conf.default
                    4.0K	/mnt
                    9.8M	/tmp
                    4.6M	/sbin
                     17M	/lib
                    4.0K	/net
                     15M	/cf
                    164K	/libexec
                    8.0M	/etc
                    1.4M	/bin
                    1.8G	/usr
                     20K	/home
                    1.1M	/tftpboot
                    4.8G	/
                    
                    1 Reply Last reply Reply Quote 0
                    • S
                      serbus @sloopbun
                      last edited by

                      @sloopbun said in pfsense VM disk becomes full - please help identify the culprit?:

                      I believe this kind of zfs on zfs is not recommended

                      Hello!

                      It is not recommended.

                      Comparing the output from df and du can be problematic, such as when you mount over the top of a folder that already has files in it, or a file has been deleted but is still open.

                      You may need to try something like lsof or fstat to dig deeper.

                      John

                      Lex parsimoniae

                      S 1 Reply Last reply Reply Quote 0
                      • S
                        sloopbun @serbus
                        last edited by

                        @serbus Thanks for the suggestion. Any recommendation incantations of fstat and lsof? I thought I might find unlinked files with lsof, but lsof +L1 / did not show any 0 in the NLINK column.

                        Similarly, I am not sure what I am looking for with fstat...

                        [2.7.2-RELEASE][root@pfSense.spacelab]/root: fstat | awk '{print $5}' | sort | uniq -c | sort -nr
                         466 /
                         276 /dev
                         155 local
                          94 internet
                          83 pipe
                          44
                          39 /var/run
                          35 internet6
                          14 -
                          12 netgraph
                           3 route
                           3 pseudo-terminal
                           1 MOUNT
                        
                        S 1 Reply Last reply Reply Quote 0
                        • S
                          sloopbun @sloopbun
                          last edited by sloopbun

                          If I run five fsck from single user mode, as recommended in the docs, it shows no errors, even for the first run.

                          Screenshot 2024-02-13 at 19.43.57.png

                          But when I run fsck from the regular pfsense terminal, I have a lot of unreferenced files, though the free space is reported correctly because this is just after a fresh reboot.

                          That is weird, right?

                          fsck:

                          [2.7.2-RELEASE][root@pfSense.spacelab]/root: fsck -y /dev/ufsid/659ec5d8d33f9720
                          ** /dev/ufsid/659ec5d8d33f9720 (NO WRITE)
                          ** SU+J Recovering /dev/ufsid/659ec5d8d33f9720
                          
                          USE JOURNAL? no
                          
                          Skipping journal, falling through to full fsck
                          ** Last Mounted on /
                          ** Root file system
                          ** Phase 1 - Check Blocks and Sizes
                          INCORRECT BLOCK COUNT I=400686 (273856 should be 258944)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=641028 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=7371851 (383040 should be 382784)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=9615548 (896 should be 640)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=9615593 (144 should be 128)
                          CORRECT? no
                          
                          ** Phase 2 - Check Pathnames
                          ** Phase 3 - Check Connectivity
                          ** Phase 4 - Check Reference Counts
                          UNREF FILE I=400686  OWNER=root MODE=100600
                          SIZE=140112184 MTIME=Feb 13 19:53 2024
                          
                          CLEAR? no
                          
                          UNREF FILE I=15225275  OWNER=root MODE=100666
                          SIZE=0 MTIME=Feb 13 19:44 2024
                          
                          CLEAR? no
                          
                          ** Phase 5 - Check Cyl groups
                          FREE BLK COUNT(S) WRONG IN SUPERBLK
                          SALVAGE? no
                          
                          SUMMARY INFORMATION BAD
                          SALVAGE? no
                          
                          BLK(S) MISSING IN BIT MAPS
                          SALVAGE? no
                          
                          54742 files, 1317012 used, 30100372 free (4028 frags, 3762043 blocks, 0.0% fragmentation)
                          

                          [2.7.2-RELEASE][root@pfSense.spacelab]/root: fsck -y /
                          ** /dev/ufsid/659ec5d8d33f9720 (NO WRITE)
                          ** SU+J Recovering /dev/ufsid/659ec5d8d33f9720
                          
                          USE JOURNAL? no
                          
                          Skipping journal, falling through to full fsck
                          ** Last Mounted on /
                          ** Root file system
                          ** Phase 1 - Check Blocks and Sizes
                          INCORRECT BLOCK COUNT I=400686 (3136 should be 8)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=480777 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=480791 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=480799 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=480800 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=1041768 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=1041770 (2560 should be 768)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=1041771 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=2243678 (64 should be 8)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=3044945 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=3285252 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=3285261 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=3285294 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=3285301 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=3285324 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=6330120 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=6330142 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=7131638 (48 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=7131644 (960 should be 768)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=7131645 (16 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=7131718 (960 should be 768)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=7131722 (960 should be 768)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=7131734 (960 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=7131742 (960 should be 768)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=7131751 (960 should be 768)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=7371851 (380224 should be 380096)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=9615548 (624 should be 544)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=9615562 (48 should be 24)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=9695856 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=9855876 (2304 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=12100439 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=13542017 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=13542018 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=14102562 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=14102571 (48 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=14102821 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=14102834 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=15225288 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=15225289 (8 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=15225290 (48 should be 0)
                          CORRECT? no
                          
                          INCORRECT BLOCK COUNT I=15225291 (8 should be 0)
                          CORRECT? no
                          
                          ** Phase 2 - Check Pathnames
                          SETTING DIRTY FLAG IN READ_ONLY MODE
                          
                          UNEXPECTED SOFT UPDATE INCONSISTENCY
                          SETTING DIRTY FLAG IN READ_ONLY MODE
                          
                          UNEXPECTED SOFT UPDATE INCONSISTENCY
                          SETTING DIRTY FLAG IN READ_ONLY MODE
                          
                          UNEXPECTED SOFT UPDATE INCONSISTENCY
                          SETTING DIRTY FLAG IN READ_ONLY MODE
                          
                          UNEXPECTED SOFT UPDATE INCONSISTENCY
                          ** Phase 3 - Check Connectivity
                          UNREF DIR  I=7291717  OWNER=root MODE=40755
                          SIZE=512 MTIME=Feb 13 19:44 2024
                          
                          RECONNECT? no
                          
                          UNREF DIR  I=7291716  OWNER=root MODE=40755
                          SIZE=512 MTIME=Feb 13 19:44 2024
                          
                          RECONNECT? no
                          
                          UNREF DIR  I=7051269  OWNER=root MODE=40755
                          SIZE=512 MTIME=Feb 13 19:44 2024
                          
                          RECONNECT? no
                          
                          UNREF DIR  I=480792  OWNER=root MODE=40755
                          SIZE=512 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          UNREF DIR  I=480801  OWNER=root MODE=40755
                          SIZE=512 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          ** Phase 4 - Check Reference Counts
                          UNREF FILE  I=320567  OWNER=root MODE=100600
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=320569  OWNER=root MODE=100600
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=320570  OWNER=root MODE=100600
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=320571  OWNER=root MODE=100600
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=320572  OWNER=root MODE=100600
                          SIZE=888 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=320577  OWNER=root MODE=100600
                          SIZE=319 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=320579  OWNER=root MODE=100600
                          SIZE=503 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=320580  OWNER=root MODE=100600
                          SIZE=262 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          ZERO LENGTH DIR I=480777  OWNER=root MODE=40755
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          CLEAR? no
                          
                          ZERO LENGTH DIR I=480782  OWNER=root MODE=40755
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          CLEAR? no
                          
                          ZERO LENGTH DIR I=480783  OWNER=root MODE=40755
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          CLEAR? no
                          
                          ZERO LENGTH DIR I=480784  OWNER=root MODE=40755
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          CLEAR? no
                          
                          UNREF FILE  I=480785  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=480786  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=480787  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=480788  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=480789  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=480790  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          ZERO LENGTH DIR I=480791  OWNER=root MODE=40755
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          CLEAR? no
                          
                          UNREF DIR I=480792  OWNER=root MODE=40755
                          SIZE=512 MTIME=Feb 13 19:45 2024
                          
                          CLEAR? no
                          
                          ZERO LENGTH DIR I=480799  OWNER=root MODE=40755
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          CLEAR? no
                          
                          ZERO LENGTH DIR I=480800  OWNER=root MODE=40755
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          CLEAR? no
                          
                          UNREF DIR I=480801  OWNER=root MODE=40755
                          SIZE=512 MTIME=Feb 13 19:45 2024
                          
                          CLEAR? no
                          
                          UNREF FILE  I=961546  OWNER=root MODE=140666
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=1041770  OWNER=root MODE=100644
                          SIZE=1253376 MTIME=Feb  9 19:29 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=1041777  OWNER=root MODE=100644
                          SIZE=212992 MTIME=Dec  6 22:23 2023
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=1041802  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=1041810  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=2564108  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=2564118  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=2564120  OWNER=root MODE=140755
                          SIZE=0 MTIME=Feb 13 19:40 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=2564121  OWNER=root MODE=100600
                          SIZE=16384 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=2564122  OWNER=root MODE=140755
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          LINK COUNT DIR I=6330112  OWNER=root MODE=40755
                          SIZE=512 MTIME=Feb 13 19:45 2024  COUNT 6 SHOULD BE 5
                          ADJUST? no
                          
                          UNREF FILE  I=6330120  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=6330140  OWNER=root MODE=100644
                          SIZE=221 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=6330142  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=6810887  OWNER=root MODE=140666
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF DIR I=7051269  OWNER=root MODE=40755
                          SIZE=512 MTIME=Feb 13 19:44 2024
                          
                          CLEAR? no
                          
                          UNREF FILE  I=7131644  OWNER=root MODE=100644
                          SIZE=457710 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=7131656  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=7131662  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=7131681  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          LINK COUNT FILE I=7131688  OWNER=root MODE=100755
                          SIZE=348096 MTIME=Dec  8 18:07 2023  COUNT 4 SHOULD BE 3
                          ADJUST? no
                          
                          UNREF FILE  I=7131705  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=7131710  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=7131715  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=7131718  OWNER=root MODE=100644
                          SIZE=457244 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=7131722  OWNER=root MODE=100644
                          SIZE=457710 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=7131734  OWNER=root MODE=100644
                          SIZE=457710 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=7131742  OWNER=root MODE=100644
                          SIZE=457244 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=7131747  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=7131751  OWNER=root MODE=100644
                          SIZE=457710 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF DIR I=7291716  OWNER=root MODE=40755
                          SIZE=512 MTIME=Feb 13 19:44 2024
                          
                          CLEAR? no
                          
                          UNREF DIR I=7291717  OWNER=root MODE=40755
                          SIZE=512 MTIME=Feb 13 19:44 2024
                          
                          CLEAR? no
                          
                          LINK COUNT FILE I=9615549  OWNER=root MODE=100600
                          SIZE=11300 MTIME=Feb 12 22:11 2024  COUNT 2 SHOULD BE 1
                          ADJUST? no
                          
                          LINK COUNT FILE I=9615560  OWNER=root MODE=100600
                          SIZE=11448 MTIME=Feb 13 12:49 2024  COUNT 2 SHOULD BE 1
                          ADJUST? no
                          
                          UNREF FILE  I=9615576  OWNER=root MODE=100600
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          LINK COUNT FILE I=9615577  OWNER=root MODE=100600
                          SIZE=11293 MTIME=Feb 13 01:48 2024  COUNT 2 SHOULD BE 1
                          ADJUST? no
                          
                          LINK COUNT FILE I=9615578  OWNER=root MODE=100600
                          SIZE=11263 MTIME=Feb 13 09:11 2024  COUNT 2 SHOULD BE 1
                          ADJUST? no
                          
                          LINK COUNT FILE I=9615587  OWNER=root MODE=100600
                          SIZE=11258 MTIME=Feb 13 16:18 2024  COUNT 2 SHOULD BE 1
                          ADJUST? no
                          
                          LINK COUNT FILE I=9615588  OWNER=root MODE=100600
                          SIZE=11317 MTIME=Feb 13 05:42 2024  COUNT 2 SHOULD BE 1
                          ADJUST? no
                          
                          UNREF FILE  I=9615618  OWNER=root MODE=100600
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=9695987  OWNER=root MODE=100644
                          SIZE=2220 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          LINK COUNT DIR I=12099330  OWNER=root MODE=40755
                          SIZE=2560 MTIME=Feb 13 19:45 2024  COUNT 37 SHOULD BE 33
                          ADJUST? no
                          
                          UNREF FILE  I=12180487  OWNER=root MODE=100644
                          SIZE=568 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          ZERO LENGTH DIR I=12180505  OWNER=root MODE=40755
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          CLEAR? no
                          
                          ZERO LENGTH DIR I=12180506  OWNER=root MODE=40755
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          CLEAR? no
                          
                          ZERO LENGTH DIR I=12180507  OWNER=root MODE=40755
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          CLEAR? no
                          
                          UNREF FILE  I=12180508  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=12180509  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=12180510  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=12180511  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=12180512  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=12180513  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE I=15225275  OWNER=root MODE=100666
                          SIZE=0 MTIME=Feb 13 19:44 2024
                          
                          CLEAR? no
                          
                          UNREF FILE  I=15225276  OWNER=root MODE=100666
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=15225278  OWNER=root MODE=100644
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=15225299  OWNER=root MODE=100666
                          SIZE=0 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=15225300  OWNER=root MODE=100644
                          SIZE=1776 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          UNREF FILE  I=15225301  OWNER=root MODE=100644
                          SIZE=156665 MTIME=Feb 13 19:45 2024
                          
                          RECONNECT? no
                          
                          
                          CLEAR? no
                          
                          ** Phase 5 - Check Cyl groups
                          FREE BLK COUNT(S) WRONG IN SUPERBLK
                          SALVAGE? no
                          
                          SUMMARY INFORMATION BAD
                          SALVAGE? no
                          
                          BLK(S) MISSING IN BIT MAPS
                          SALVAGE? no
                          
                          ALLOCATED FILE 15225275 MARKED FREE
                          54697 files, 1280056 used, 30136626 free (4026 frags, 3766575 blocks, 0.0% fragmentation)
                          

                          [2.7.2-RELEASE][root@pfSense.spacelab]/root: df -hi /
                          Filesystem                     Size    Used   Avail Capacity iused ifree %iused  Mounted on
                          /dev/ufsid/659ec5d8d33f9720    120G    4.9G    105G     4%     55k   16M    0%   /
                          
                          S 1 Reply Last reply Reply Quote 0
                          • S
                            serbus @sloopbun
                            last edited by

                            @sloopbun

                            Hello!

                            This discussion and similar might have other leads you can follow.

                            John

                            Lex parsimoniae

                            S 1 Reply Last reply Reply Quote 0
                            • S
                              sloopbun @serbus
                              last edited by

                              @serbus Thanks. Open, or zombie procresses/files is a likely candidate, though I cannot find any. In that link there are a couple of commands for linux that makes sense to me, but either do not turn up any results, or do not work on freebsd. Any idea what their equivalent might be?

                              I am referring to are:

                              lsof | grep -i deleted
                              

                              which returns nothing, and:

                              [2.7.2-RELEASE][root@pfSense.spacelab]/: mount -o bind / /mnt
                              mount: /: mount option <bind> is unknown: Invalid argument
                              [2.7.2-RELEASE][root@pfSense.spacelab]/:
                              

                              which I think has an equivalent freebsd of:

                              mount -t nullfs / /mnt

                              Though once mounted, appear to be identical to /. No extra files in /mnt/var or anything...

                              Was there any other avenue of exploration you saw that might help?

                              Apart from the zombie processes/files, the other likely candidate in my mind is trim/discard/clean-up. What do you think about turning on trim for UFS in pfsense?

                              S 1 Reply Last reply Reply Quote 0
                              • S
                                sloopbun @sloopbun
                                last edited by

                                To add an unhelpful update. This problem was still occurring a few days ago, so I restarted as usual to start the ticking time bomb again.

                                I now have almost four days of uptime and available disk is in line with disk usage.

                                I swear I did not change anything or try to fix anything since this thread went quiet. In fact I have been convincing myself to go back to bare metal. Super weird and I don't trust it...

                                [2.7.2-RELEASE][root@pfSense.spacelab]/root: uptime
                                 9:19AM  up 3 days, 22:56, 2 users, load averages: 0.33, 0.22, 0.18
                                
                                [2.7.2-RELEASE][root@pfSense.spacelab]/root: df -h /
                                Filesystem                     Size    Used   Avail Capacity  Mounted on
                                /dev/ufsid/659ec5d8d33f9720    120G    5.8G    104G     5%    /
                                
                                [2.7.2-RELEASE][root@pfSense.spacelab]/root: du -h -d 1 /
                                4.0K	/.snap
                                101M	/boot
                                4.5K	/dev
                                 16M	/rescue
                                4.0K	/proc
                                 96M	/root
                                3.1G	/var
                                4.0K	/media
                                 12K	/conf.default
                                4.0K	/mnt
                                9.9M	/tmp
                                4.6M	/sbin
                                 17M	/lib
                                4.0K	/net
                                 16M	/cf
                                164K	/libexec
                                8.0M	/etc
                                1.4M	/bin
                                1.8G	/usr
                                 20K	/home
                                1.1M	/tftpboot
                                5.8G	/
                                
                                GPz1100G 1 Reply Last reply Reply Quote 0
                                • GPz1100G
                                  GPz1100 @sloopbun
                                  last edited by

                                  @sloopbun Did you ever get this resolved? I'm about to commit to a proxmox vm install myself.

                                  1 Reply Last reply Reply Quote 0
                                  • triksT
                                    triks
                                    last edited by

                                    system > advanced > miscellaneous >

                                    (https://yourpfsenseip/system_advanced_misc.php)

                                    ![alt text](pfsense_ram_disk_size.png image url)

                                    I set mine to 1024 and 1024.

                                    triksT 1 Reply Last reply Reply Quote 0
                                    • triksT
                                      triks @triks
                                      last edited by

                                      @triks for anyone in the same boat, it ended up being NTOPNG that was filling the RAMDISK. Followed many posts but couldn't get it to save to SSD so removed it and that resolved the issue.

                                      1 Reply Last reply Reply Quote 0
                                      • First post
                                        Last post
                                      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.