Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfsense VM disk becomes full - please help identify the culprit?

    Scheduled Pinned Locked Moved
    Virtualization
    8
    21
    1.8k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      Popolou @sloopbun
      last edited by

      @sloopbun said in pfsense VM disk becomes full - please help identify the culprit?:

      Since I had posted, my disk usage had gone from 13G to 19G.

      Where are you checking this from, within pfsense or via the hypervisor/proxmox?

      S 1 Reply Last reply Reply Quote 0
      • S
        sloopbun @Popolou
        last edited by sloopbun

        @Popolou said in pfsense VM disk becomes full - please help identify the culprit?:

        @sloopbun said in pfsense VM disk becomes full - please help identify the culprit?:

        Since I had posted, my disk usage had gone from 13G to 19G.

        Where are you checking this from, within pfsense or via the hypervisor/proxmox?

        From within pfsense. For example, below is the progression over a few hours today, but I do not see any change in the zvol reported on the host, proxmox.

        root@pvepbs:~# date
Tue Feb 13 08:45:13 AM CET 2024

root@pvepbs:~# zfs list rpool/data/vm-100-disk-1
NAME                       USED  AVAIL  REFER  MOUNTPOINT
rpool/data/vm-100-disk-1  33.4G   684G  33.4G  -

        root@pvepbs:~# date
Tue Feb 13 03:14:46 PM CET 2024

root@pvepbs:~# zfs list rpool/data/vm-100-disk-1
NAME                       USED  AVAIL  REFER  MOUNTPOINT
rpool/data/vm-100-disk-1  33.2G   684G  33.2G  -

        [2.7.2-RELEASE][root@pfSense.spacelab]/root: date
Tue Feb 13 08:45:13 CET 2024

[2.7.2-RELEASE][root@pfSense.spacelab]/root: df -hi /
Filesystem                     Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/ufsid/659ec5d8d33f9720    120G     26G     85G    23%     55k   16M    0%   /

        [2.7.2-RELEASE][root@pfSense.spacelab]/root: date
Tue Feb 13 15:07:25 CET 2024

[2.7.2-RELEASE][root@pfSense.spacelab]/root: df -hi /
Filesystem                     Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/ufsid/659ec5d8d33f9720    120G     32G     79G    29%     55k   16M    0%   /

        [2.7.2-RELEASE][root@pfSense.spacelab]/root: du -h -d 1 /
4.0K	/.snap
101M	/boot
4.5K	/dev
 16M	/rescue
4.0K	/proc
 96M	/root
2.2G	/var
4.0K	/media
 12K	/conf.default
4.0K	/mnt
9.8M	/tmp
4.6M	/sbin
 17M	/lib
4.0K	/net
 15M	/cf
164K	/libexec
8.0M	/etc
1.4M	/bin
1.8G	/usr
 20K	/home
1.1M	/tftpboot
4.8G	/
        1 Reply Last reply Reply Quote 0
        • S
          serbus @sloopbun
          last edited by

          @sloopbun said in pfsense VM disk becomes full - please help identify the culprit?:

          I believe this kind of zfs on zfs is not recommended

          Hello!

          It is not recommended.

          Comparing the output from df and du can be problematic, such as when you mount over the top of a folder that already has files in it, or a file has been deleted but is still open.

          You may need to try something like lsof or fstat to dig deeper.

          John

          Lex parsimoniae

          S 1 Reply Last reply Reply Quote 0
          • S
            sloopbun @serbus
            last edited by

            @serbus Thanks for the suggestion. Any recommendation incantations of fstat and lsof? I thought I might find unlinked files with lsof, but lsof +L1 / did not show any 0 in the NLINK column.

            Similarly, I am not sure what I am looking for with fstat...

            [2.7.2-RELEASE][root@pfSense.spacelab]/root: fstat | awk '{print $5}' | sort | uniq -c | sort -nr
 466 /
 276 /dev
 155 local
  94 internet
  83 pipe
  44
  39 /var/run
  35 internet6
  14 -
  12 netgraph
   3 route
   3 pseudo-terminal
   1 MOUNT
            S 1 Reply Last reply Reply Quote 0
            • S
              sloopbun @sloopbun
              last edited by sloopbun

              If I run five fsck from single user mode, as recommended in the docs, it shows no errors, even for the first run.

              Screenshot 2024-02-13 at 19.43.57.png

              But when I run fsck from the regular pfsense terminal, I have a lot of unreferenced files, though the free space is reported correctly because this is just after a fresh reboot.

              That is weird, right?

              fsck:

              [2.7.2-RELEASE][root@pfSense.spacelab]/root: fsck -y /dev/ufsid/659ec5d8d33f9720
** /dev/ufsid/659ec5d8d33f9720 (NO WRITE)
** SU+J Recovering /dev/ufsid/659ec5d8d33f9720

USE JOURNAL? no

Skipping journal, falling through to full fsck
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
INCORRECT BLOCK COUNT I=400686 (273856 should be 258944)
CORRECT? no

INCORRECT BLOCK COUNT I=641028 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=7371851 (383040 should be 382784)
CORRECT? no

INCORRECT BLOCK COUNT I=9615548 (896 should be 640)
CORRECT? no

INCORRECT BLOCK COUNT I=9615593 (144 should be 128)
CORRECT? no

** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
UNREF FILE I=400686  OWNER=root MODE=100600
SIZE=140112184 MTIME=Feb 13 19:53 2024

CLEAR? no

UNREF FILE I=15225275  OWNER=root MODE=100666
SIZE=0 MTIME=Feb 13 19:44 2024

CLEAR? no

** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? no

SUMMARY INFORMATION BAD
SALVAGE? no

BLK(S) MISSING IN BIT MAPS
SALVAGE? no

54742 files, 1317012 used, 30100372 free (4028 frags, 3762043 blocks, 0.0% fragmentation)


              [2.7.2-RELEASE][root@pfSense.spacelab]/root: fsck -y /
** /dev/ufsid/659ec5d8d33f9720 (NO WRITE)
** SU+J Recovering /dev/ufsid/659ec5d8d33f9720

USE JOURNAL? no

Skipping journal, falling through to full fsck
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
INCORRECT BLOCK COUNT I=400686 (3136 should be 8)
CORRECT? no

INCORRECT BLOCK COUNT I=480777 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=480791 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=480799 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=480800 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=1041768 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=1041770 (2560 should be 768)
CORRECT? no

INCORRECT BLOCK COUNT I=1041771 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=2243678 (64 should be 8)
CORRECT? no

INCORRECT BLOCK COUNT I=3044945 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=3285252 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=3285261 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=3285294 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=3285301 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=3285324 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=6330120 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=6330142 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=7131638 (48 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=7131644 (960 should be 768)
CORRECT? no

INCORRECT BLOCK COUNT I=7131645 (16 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=7131718 (960 should be 768)
CORRECT? no

INCORRECT BLOCK COUNT I=7131722 (960 should be 768)
CORRECT? no

INCORRECT BLOCK COUNT I=7131734 (960 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=7131742 (960 should be 768)
CORRECT? no

INCORRECT BLOCK COUNT I=7131751 (960 should be 768)
CORRECT? no

INCORRECT BLOCK COUNT I=7371851 (380224 should be 380096)
CORRECT? no

INCORRECT BLOCK COUNT I=9615548 (624 should be 544)
CORRECT? no

INCORRECT BLOCK COUNT I=9615562 (48 should be 24)
CORRECT? no

INCORRECT BLOCK COUNT I=9695856 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=9855876 (2304 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=12100439 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=13542017 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=13542018 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=14102562 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=14102571 (48 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=14102821 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=14102834 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=15225288 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=15225289 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=15225290 (48 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=15225291 (8 should be 0)
CORRECT? no

** Phase 2 - Check Pathnames
SETTING DIRTY FLAG IN READ_ONLY MODE

UNEXPECTED SOFT UPDATE INCONSISTENCY
SETTING DIRTY FLAG IN READ_ONLY MODE

UNEXPECTED SOFT UPDATE INCONSISTENCY
SETTING DIRTY FLAG IN READ_ONLY MODE

UNEXPECTED SOFT UPDATE INCONSISTENCY
SETTING DIRTY FLAG IN READ_ONLY MODE

UNEXPECTED SOFT UPDATE INCONSISTENCY
** Phase 3 - Check Connectivity
UNREF DIR  I=7291717  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:44 2024

RECONNECT? no

UNREF DIR  I=7291716  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:44 2024

RECONNECT? no

UNREF DIR  I=7051269  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:44 2024

RECONNECT? no

UNREF DIR  I=480792  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:45 2024

RECONNECT? no

UNREF DIR  I=480801  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:45 2024

RECONNECT? no

** Phase 4 - Check Reference Counts
UNREF FILE  I=320567  OWNER=root MODE=100600
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=320569  OWNER=root MODE=100600
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=320570  OWNER=root MODE=100600
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=320571  OWNER=root MODE=100600
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=320572  OWNER=root MODE=100600
SIZE=888 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=320577  OWNER=root MODE=100600
SIZE=319 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=320579  OWNER=root MODE=100600
SIZE=503 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=320580  OWNER=root MODE=100600
SIZE=262 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

ZERO LENGTH DIR I=480777  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

ZERO LENGTH DIR I=480782  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

ZERO LENGTH DIR I=480783  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

ZERO LENGTH DIR I=480784  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

UNREF FILE  I=480785  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=480786  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=480787  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=480788  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=480789  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=480790  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

ZERO LENGTH DIR I=480791  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

UNREF DIR I=480792  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:45 2024

CLEAR? no

ZERO LENGTH DIR I=480799  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

ZERO LENGTH DIR I=480800  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

UNREF DIR I=480801  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:45 2024

CLEAR? no

UNREF FILE  I=961546  OWNER=root MODE=140666
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=1041770  OWNER=root MODE=100644
SIZE=1253376 MTIME=Feb  9 19:29 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=1041777  OWNER=root MODE=100644
SIZE=212992 MTIME=Dec  6 22:23 2023

RECONNECT? no


CLEAR? no

UNREF FILE  I=1041802  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=1041810  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=2564108  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=2564118  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=2564120  OWNER=root MODE=140755
SIZE=0 MTIME=Feb 13 19:40 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=2564121  OWNER=root MODE=100600
SIZE=16384 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=2564122  OWNER=root MODE=140755
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

LINK COUNT DIR I=6330112  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:45 2024  COUNT 6 SHOULD BE 5
ADJUST? no

UNREF FILE  I=6330120  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=6330140  OWNER=root MODE=100644
SIZE=221 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=6330142  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=6810887  OWNER=root MODE=140666
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF DIR I=7051269  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:44 2024

CLEAR? no

UNREF FILE  I=7131644  OWNER=root MODE=100644
SIZE=457710 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131656  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131662  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131681  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

LINK COUNT FILE I=7131688  OWNER=root MODE=100755
SIZE=348096 MTIME=Dec  8 18:07 2023  COUNT 4 SHOULD BE 3
ADJUST? no

UNREF FILE  I=7131705  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131710  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131715  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131718  OWNER=root MODE=100644
SIZE=457244 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131722  OWNER=root MODE=100644
SIZE=457710 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131734  OWNER=root MODE=100644
SIZE=457710 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131742  OWNER=root MODE=100644
SIZE=457244 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131747  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131751  OWNER=root MODE=100644
SIZE=457710 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF DIR I=7291716  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:44 2024

CLEAR? no

UNREF DIR I=7291717  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:44 2024

CLEAR? no

LINK COUNT FILE I=9615549  OWNER=root MODE=100600
SIZE=11300 MTIME=Feb 12 22:11 2024  COUNT 2 SHOULD BE 1
ADJUST? no

LINK COUNT FILE I=9615560  OWNER=root MODE=100600
SIZE=11448 MTIME=Feb 13 12:49 2024  COUNT 2 SHOULD BE 1
ADJUST? no

UNREF FILE  I=9615576  OWNER=root MODE=100600
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

LINK COUNT FILE I=9615577  OWNER=root MODE=100600
SIZE=11293 MTIME=Feb 13 01:48 2024  COUNT 2 SHOULD BE 1
ADJUST? no

LINK COUNT FILE I=9615578  OWNER=root MODE=100600
SIZE=11263 MTIME=Feb 13 09:11 2024  COUNT 2 SHOULD BE 1
ADJUST? no

LINK COUNT FILE I=9615587  OWNER=root MODE=100600
SIZE=11258 MTIME=Feb 13 16:18 2024  COUNT 2 SHOULD BE 1
ADJUST? no

LINK COUNT FILE I=9615588  OWNER=root MODE=100600
SIZE=11317 MTIME=Feb 13 05:42 2024  COUNT 2 SHOULD BE 1
ADJUST? no

UNREF FILE  I=9615618  OWNER=root MODE=100600
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=9695987  OWNER=root MODE=100644
SIZE=2220 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

LINK COUNT DIR I=12099330  OWNER=root MODE=40755
SIZE=2560 MTIME=Feb 13 19:45 2024  COUNT 37 SHOULD BE 33
ADJUST? no

UNREF FILE  I=12180487  OWNER=root MODE=100644
SIZE=568 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

ZERO LENGTH DIR I=12180505  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

ZERO LENGTH DIR I=12180506  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

ZERO LENGTH DIR I=12180507  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

UNREF FILE  I=12180508  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=12180509  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=12180510  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=12180511  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=12180512  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=12180513  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE I=15225275  OWNER=root MODE=100666
SIZE=0 MTIME=Feb 13 19:44 2024

CLEAR? no

UNREF FILE  I=15225276  OWNER=root MODE=100666
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=15225278  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=15225299  OWNER=root MODE=100666
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=15225300  OWNER=root MODE=100644
SIZE=1776 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=15225301  OWNER=root MODE=100644
SIZE=156665 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? no

SUMMARY INFORMATION BAD
SALVAGE? no

BLK(S) MISSING IN BIT MAPS
SALVAGE? no

ALLOCATED FILE 15225275 MARKED FREE
54697 files, 1280056 used, 30136626 free (4026 frags, 3766575 blocks, 0.0% fragmentation)


              [2.7.2-RELEASE][root@pfSense.spacelab]/root: df -hi /
Filesystem                     Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/ufsid/659ec5d8d33f9720    120G    4.9G    105G     4%     55k   16M    0%   /
              S 1 Reply Last reply Reply Quote 0
              • S
                serbus @sloopbun
                last edited by

                @sloopbun

                Hello!

                This discussion and similar might have other leads you can follow.

                John

                Lex parsimoniae

                S 1 Reply Last reply Reply Quote 0
                • S
                  sloopbun @serbus
                  last edited by

                  @serbus Thanks. Open, or zombie procresses/files is a likely candidate, though I cannot find any. In that link there are a couple of commands for linux that makes sense to me, but either do not turn up any results, or do not work on freebsd. Any idea what their equivalent might be?

                  I am referring to are:

                  lsof | grep -i deleted

                  which returns nothing, and:

                  [2.7.2-RELEASE][root@pfSense.spacelab]/: mount -o bind / /mnt
mount: /: mount option <bind> is unknown: Invalid argument
[2.7.2-RELEASE][root@pfSense.spacelab]/:

                  which I think has an equivalent freebsd of:

                  mount -t nullfs / /mnt

                  Though once mounted, appear to be identical to /. No extra files in /mnt/var or anything...

                  Was there any other avenue of exploration you saw that might help?

                  Apart from the zombie processes/files, the other likely candidate in my mind is trim/discard/clean-up. What do you think about turning on trim for UFS in pfsense?

                  S 1 Reply Last reply Reply Quote 0
                  • S
                    sloopbun @sloopbun
                    last edited by

                    To add an unhelpful update. This problem was still occurring a few days ago, so I restarted as usual to start the ticking time bomb again.

                    I now have almost four days of uptime and available disk is in line with disk usage.

                    I swear I did not change anything or try to fix anything since this thread went quiet. In fact I have been convincing myself to go back to bare metal. Super weird and I don't trust it...

                    [2.7.2-RELEASE][root@pfSense.spacelab]/root: uptime
 9:19AM  up 3 days, 22:56, 2 users, load averages: 0.33, 0.22, 0.18

                    [2.7.2-RELEASE][root@pfSense.spacelab]/root: df -h /
Filesystem                     Size    Used   Avail Capacity  Mounted on
/dev/ufsid/659ec5d8d33f9720    120G    5.8G    104G     5%    /

                    [2.7.2-RELEASE][root@pfSense.spacelab]/root: du -h -d 1 /
4.0K	/.snap
101M	/boot
4.5K	/dev
 16M	/rescue
4.0K	/proc
 96M	/root
3.1G	/var
4.0K	/media
 12K	/conf.default
4.0K	/mnt
9.9M	/tmp
4.6M	/sbin
 17M	/lib
4.0K	/net
 16M	/cf
164K	/libexec
8.0M	/etc
1.4M	/bin
1.8G	/usr
 20K	/home
1.1M	/tftpboot
5.8G	/
                    G 1 Reply Last reply Reply Quote 0
                    • G
                      GPz1100 @sloopbun
                      last edited by

                      @sloopbun Did you ever get this resolved? I'm about to commit to a proxmox vm install myself.

                      1 Reply Last reply Reply Quote 0
                      • T
                        triks
                        last edited by

                        system > advanced > miscellaneous >

                        (https://yourpfsenseip/system_advanced_misc.php)

                        ![alt text](pfsense_ram_disk_size.png image url)

                        I set mine to 1024 and 1024.

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post