pfsense VM disk becomes full - please help identify the culprit?

sloopbun · Feb 13, 2024, 7:54 AM

Hello,

I have pfsense CE 2.7.2-RELEASE in a proxmox VM. Generally works well compared to bare metal I had before. The only difference is that on bare-metal I configured root-zfs for pfsense, but in this virtualised environment, I use UFS for boot, because proxmox use zfs and zvol for the VM.

The problem I am facing is that over a couple of weeks, the pfsense disk fills up until services like dns stop because no more writing is possible. I can check with df and it shows 102% disk usage.

When I check with du looking for the cause, it shows only 13G used.

Rebooting the proxmox machine resolves the issue temporarily.

I have tried everything but unable to identify what is consuming the space.

Can anyone help me please?

After a reboot, temporarily resolving the issue, this is what I have. Note that the df result is different, but the du result is unchanged:

[2.7.2-RELEASE][root@pfSense.spacelab]/root: df -h /
Filesystem                     Size    Used   Avail Capacity  Mounted on
/dev/ufsid/659ec5d8d33f9720    120G     13G     97G    12%    /

[2.7.2-RELEASE][root@pfSense.spacelab]/root: du -h -d 1 /
4.0K	/.snap
101M	/boot
4.5K	/dev
 16M	/rescue
4.0K	/proc
 48M	/root
 10G	/var
4.0K	/media
 12K	/conf.default
4.0K	/mnt
1.6M	/tmp
4.6M	/sbin
 17M	/lib
4.0K	/net
 15M	/cf
164K	/libexec
8.0M	/etc
1.4M	/bin
2.0G	/usr
 20K	/home
1.1M	/tftpboot
 13G	/

[2.7.2-RELEASE][root@pfSense.spacelab]/root: mount
/dev/ufsid/659ec5d8d33f9720 on / (ufs, local, noatime, soft-updates, journaled soft-updates)
devfs on /dev (devfs)
/dev/vtbd0p1 on /boot/efi (msdosfs, local)
tmpfs on /var/run (tmpfs, local)

root@pvepbs:~# cat /etc/pve/qemu-server/100.conf
balloon: 0
bios: ovmf
boot: order=ide2;virtio0
cores: 6
cpu: host,flags=+aes
efidisk0: local-zfs:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:01:00,pcie=1
hostpci1: 0000:03:00,pcie=1
ide2: none,media=cdrom
machine: q35
memory: 16384
meta: creation-qemu=8.1.2,ctime=1704716297
name: pfsense
numa: 0
onboot: 1
ostype: other
scsihw: virtio-scsi-single
serial0: socket
smbios1: uuid=dd5d2b62-e3a4-4da9-b796-e8c97bfb8358
sockets: 1
vga: qxl
virtio0: local-zfs:vm-100-disk-1,discard=on,iothread=1,size=128G
vmgenid: babe5976-8e38-42cb-9d4b-4f14f6e2cdfa

gstat:

dT: 1.010s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      6      0      0    0.0      6   1023    0.4    0.1| vtbd0
    0      0      0      0    0.0      0      0    0.0    0.0| vtbd0p1
    0      5      0      0    0.0      5   1023    0.3    0.1| vtbd0p2
    0      0      0      0    0.0      0      0    0.0    0.0| vtbd0p3
    0      5      0      0    0.0      5   1023    0.3    0.1| ufsid/659ec5d8d33f9720
    0      0      0      0    0.0      0      0    0.0    0.0| label/swap0
    0      0      0      0    0.0      0      0    0.0    0.0| cd0

systat -vmstat:

    2 users    Load  0.88  0.66  0.50                  Feb 10 12:56:58
   Mem usage:  17%Phy  2%Kmem                           VN PAGER   SWAP PAGER
Mem:      REAL           VIRTUAL                        in   out     in   out
       Tot   Share     Tot    Share     Free   count          28
Act  1414M    106M    521G     151M   13076M   pages     3    28
All  1423M    114M    521G     241M                       ioflt  Interrupts
Proc:                                                 186 cow   32032 total
  r   p   d    s   w   Csw  Trp  Sys  Int  Sof  Flt   958 zfod        atkbd0 1
  1          235       73K   1K  15K  30K  12K   1K       ozfod   270 cpu0:timer
                                                         %ozfod   320 cpu1:timer
 3.0%Sys   2.5%Intr  1.4%User  0.0%Nice 93.1%Idle         daefr   362 cpu2:timer
|    |    |    |    |    |    |    |    |    |    |   280 prcfr   283 cpu3:timer
=++                                                   541 totfr   279 cpu4:timer
                                       192 dtbuf          react   280 cpu5:timer
Namei     Name-cache   Dir-cache    348426 maxvn          pdwak  6559 igc0:rxq0
   Calls    hits   %    hits   %     68769 numvn      319 pdpgs    91 igc0:rxq1
    6778    6778 100                 55062 frevn          intrn  7885 igc0:rxq2
                                                    1197M wire    118 igc0:rxq3
Disks vtbd0   cd0 pass0                              771M act         igc0:aq 29
KB/t  89.07  0.00  0.00                              767M inact    83 igc1:rxq0
tps       3     0     0                                 0 laund  8450 igc1:rxq1
MB/s   0.24  0.00  0.00                               13G free    118 igc1:rxq2
%busy     0     0     0                              730M buf    6924 igc1:rxq3
                                                                      igc1:aq 34
                                                                      virtio_pci
                                                                    3 virtio_pci
                                                                    7 ahci0 39

SteveITS · Feb 10, 2024, 2:26 PM

@sloopbun Did you try an fsck? https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html

sloopbun · Feb 11, 2024, 10:18 PM

@SteveITS said in pfsense VM disk becomes full - please help identify the culprit?:

@sloopbun Did you try an fsck? https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html

Thanks for the suggestion. I had not done so. I just did the reboot with disk check from the gui.

Since I had posted, my disk usage had gone from 13G to 19G.

After the reboot with disk check, it is back to the 13G. Will see how it increases in the next day.

provels · Feb 12, 2024, 8:50 AM

Disk space in the host server isn't overcommitted, is it? For instance, in VMWare one can assign 2TB to each each of 2 VMs residing on a 3TB host disk and eventually someone will lose out.
Just spitballin'.

sloopbun · Feb 12, 2024, 8:50 AM

@provels
Since doing the reboot with filesystem check, my disk usage has jumped from 13G to 21G.

[2.7.2-RELEASE][root@pfSense.spacelab]/root: df -hi
Filesystem                     Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/ufsid/659ec5d8d33f9720    120G     22G     89G    20%     64k   16M    0%   /
devfs                          1.0K      0B    1.0K     0%       0     0     -   /dev
/dev/vtbd0p1                   260M    1.3M    259M     1%       2   510    0%   /boot/efi
tmpfs                          4.0M    192K    3.8M     5%      56   14k    0%   /var/run

[2.7.2-RELEASE][root@pfSense.spacelab]/root: du -h -d 1 /
4.0K	/.snap
101M	/boot
4.5K	/dev
 16M	/rescue
4.0K	/proc
 96M	/root
 10G	/var
4.0K	/media
 12K	/conf.default
4.0K	/mnt
1.1M	/tmp
4.6M	/sbin
 17M	/lib
4.0K	/net
 15M	/cf
164K	/libexec
8.0M	/etc
1.4M	/bin
1.8G	/usr
 20K	/home
1.1M	/tftpboot
 13G	/

Host storage looks normal to me:

root@pvepbs:~# zfs list
NAME                       USED  AVAIL  REFER  MOUNTPOINT
rpool                     41.2G   680G   104K  /rpool
rpool/ROOT                2.46G   680G    96K  /rpool/ROOT
rpool/ROOT/pve-1          2.46G   680G  2.46G  /
rpool/data                33.5G   680G    96K  /rpool/data
rpool/data/vm-100-disk-0    80K   680G    80K  -
rpool/data/vm-100-disk-1  33.5G   680G  33.5G  -
rpool/pbs_datastore         96K   600G    96K  /rpool/pbs_datastore
rpool/var-lib-vz          5.06G   680G  5.06G  /var/lib/vz

Gertjan · Feb 12, 2024, 9:15 AM

@sloopbun

What pfSense packages are you using ?

Have a look, at the /var/log/ folder, and all the /var/log/ sub folders.
There are packages that log 'network' activity. If you have a lot of network activity, log files can grow to very fast, within a day, to several Gbytes. The solution would be : stop using the tools that log a lot, or don't have them log anymore.

You /var/ :

10G	/var

I bet these 10G are mostly in /var/log/

[23.09.1-RELEASE][root@pfSense.bhf.tld]/root: df -hi
Filesystem                         Size    Used   Avail Capacity iused ifree %iused  Mounted on
......
pfSense/var/log                     94G     86M     94G     0%     116  197M    0%   /var/log
......

serbus · Feb 12, 2024, 6:32 PM

@sloopbun

Hello!

I have seen something similar (?) when running a sparse guest fs on top of a thin-provisioned host fs.
I dont run zfs/zvol on proxmox, so I dont know if this applies.

https://forum.netgate.com/topic/166274/pfsense-hyper-v-vhdx-growing-like-crazy

John

sloopbun · Feb 12, 2024, 7:12 PM

@Gertjan said in pfsense VM disk becomes full - please help identify the culprit?:

@sloopbun

What pfSense packages are you using ?

Have a look, at the /var/log/ folder, and all the /var/log/ sub folders.
There are packages that log 'network' activity. If you have a lot of network activity, log files can grow to very fast, within a day, to several Gbytes. The solution would be : stop using the tools that log a lot, or don't have them log anymore.

You /var/ :
10G	/var
I bet these 10G are mostly in /var/log/
[23.09.1-RELEASE][root@pfSense.bhf.tld]/root: df -hi
Filesystem                         Size    Used   Avail Capacity iused ifree %iused  Mounted on
......
pfSense/var/log                     94G     86M     94G     0%     116  197M    0%   /var/log
......

Here are my packages, there is some logging, and I have deleted a bunch just to free up space, but the reported size of the /var/ directory does not balloon, just the lack of available space reported by df.

login-to-view

@serbus said in pfsense VM disk becomes full - please help identify the culprit?:

@sloopbun

Hello!

I have seen something similar (?) when running a sparse guest fs on top of a thin-provisioned host fs.
I dont run zfs/zvol on proxmox, so I dont know if this applies.

https://forum.netgate.com/topic/166274/pfsense-hyper-v-vhdx-growing-like-crazy

John

This is much more along the lines of what I have in mine. In proxmox, with zvol, there is a setting for "discard" which I have enabled.

I note that the last reply on that thread was:

@LesserBloops said in PfSense Hyper-V .vhdx growing like crazy:

I hit this when I tried using ZFS as the file system with expanding disks. It went runaway and eventually caused an outage.

I've been meaning to go back and try again with a fixed size disk instead of an expanding one, but haven't got around to it yet, and EXT seems to be fine...

There are some parameters that I am not confident in the effects of such as volmode and reservation which I am hesitant to mess with with a recommendation to do so. I not that some parameters like logicalused and volsize represent the size of the disk that pfsense sees.

I don't think this would seem to explain why the available space shown by df / progressively grows over time, without any file usage, but I would be happy to take advice on resolving this.

The odd thing to me is that I thought this was a fairly standard practice, as explained in the docs here: https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html

The fact that this is not a common issue points to it being a configuration issue on my behalf, right?

root@pvepbs:~# zfs get all rpool/data/vm-100-disk-1
NAME                      PROPERTY              VALUE                     SOURCE
rpool/data/vm-100-disk-1  type                  volume                    -
rpool/data/vm-100-disk-1  creation              Mon Jan  8 13:18 2024     -
rpool/data/vm-100-disk-1  used                  33.5G                     -
rpool/data/vm-100-disk-1  available             684G                      -
rpool/data/vm-100-disk-1  referenced            33.5G                     -
rpool/data/vm-100-disk-1  compressratio         3.58x                     -
rpool/data/vm-100-disk-1  reservation           none                      default
rpool/data/vm-100-disk-1  volsize               128G                      local
rpool/data/vm-100-disk-1  volblocksize          16K                       default
rpool/data/vm-100-disk-1  checksum              on                        default
rpool/data/vm-100-disk-1  compression           on                        inherited from rpool
rpool/data/vm-100-disk-1  readonly              off                       default
rpool/data/vm-100-disk-1  createtxg             2282                      -
rpool/data/vm-100-disk-1  copies                1                         default
rpool/data/vm-100-disk-1  refreservation        none                      default
rpool/data/vm-100-disk-1  guid                  7625546012721003177       -
rpool/data/vm-100-disk-1  primarycache          all                       default
rpool/data/vm-100-disk-1  secondarycache        all                       default
rpool/data/vm-100-disk-1  usedbysnapshots       0B                        -
rpool/data/vm-100-disk-1  usedbydataset         33.5G                     -
rpool/data/vm-100-disk-1  usedbychildren        0B                        -
rpool/data/vm-100-disk-1  usedbyrefreservation  0B                        -
rpool/data/vm-100-disk-1  logbias               latency                   default
rpool/data/vm-100-disk-1  objsetid              2013                      -
rpool/data/vm-100-disk-1  dedup                 off                       default
rpool/data/vm-100-disk-1  mlslabel              none                      default
rpool/data/vm-100-disk-1  sync                  standard                  inherited from rpool
rpool/data/vm-100-disk-1  refcompressratio      3.58x                     -
rpool/data/vm-100-disk-1  written               33.5G                     -
rpool/data/vm-100-disk-1  logicalused           119G                      -
rpool/data/vm-100-disk-1  logicalreferenced     119G                      -
rpool/data/vm-100-disk-1  volmode               default                   default
rpool/data/vm-100-disk-1  snapshot_limit        none                      default
rpool/data/vm-100-disk-1  snapshot_count        none                      default
rpool/data/vm-100-disk-1  snapdev               hidden                    default
rpool/data/vm-100-disk-1  context               none                      default
rpool/data/vm-100-disk-1  fscontext             none                      default
rpool/data/vm-100-disk-1  defcontext            none                      default
rpool/data/vm-100-disk-1  rootcontext           none                      default
rpool/data/vm-100-disk-1  redundant_metadata    all                       default
rpool/data/vm-100-disk-1  encryption            off                       default
rpool/data/vm-100-disk-1  keylocation           none                      default
rpool/data/vm-100-disk-1  keyformat             none                      default
rpool/data/vm-100-disk-1  pbkdf2iters           0                         default
rpool/data/vm-100-disk-1  snapshots_changed     Mon Feb 12  9:32:58 2024  -

SteveITS · Feb 12, 2024, 7:51 PM

@sloopbun I remembered this thread https://forum.netgate.com/topic/183318/free-up-space-disk-storage-80/ with a similar report I think but it doesn't look like there was a resolution.

sloopbun · Feb 13, 2024, 7:43 AM

@SteveITS said in pfsense VM disk becomes full - please help identify the culprit?:

@sloopbun I remembered this thread https://forum.netgate.com/topic/183318/free-up-space-disk-storage-80/ with a similar report I think but it doesn't look like there was a resolution.

In that thread, it is suggested that the command geom part list might reveal something, but it looks normal to me but maybe someone else can spot something?:

[2.7.2-RELEASE][root@pfSense.spacelab]/root: geom part list
Geom name: vtbd0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 268435415
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: vtbd0p1
   Mediasize: 272629760 (260M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r1w1e1
   efimedia: HD(1,GPT,616156c8-afd5-11ee-9ccd-e9881622351f,0x28,0x82000)
   rawuuid: 616156c8-afd5-11ee-9ccd-e9881622351f
   rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
   label: (null)
   length: 272629760
   offset: 20480
   type: efi
   index: 1
   end: 532519
   start: 40
2. Name: vtbd0p2
   Mediasize: 132871356416 (124G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 272650240
   Mode: r1w1e2
   efimedia: HD(2,GPT,6161bcfd-afd5-11ee-9ccd-e9881622351f,0x82028,0xf77e000)
   rawuuid: 6161bcfd-afd5-11ee-9ccd-e9881622351f
   rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 132871356416
   offset: 272650240
   type: freebsd-ufs
   index: 2
   end: 260046887
   start: 532520
3. Name: vtbd0p3
   Mediasize: 4294926336 (4.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 133144006656
   Mode: r1w1e1
   efimedia: HD(3,GPT,616220f2-afd5-11ee-9ccd-e9881622351f,0xf800028,0x7fffb0)
   rawuuid: 616220f2-afd5-11ee-9ccd-e9881622351f
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 4294926336
   offset: 133144006656
   type: freebsd-swap
   index: 3
   end: 268435415
   start: 260046888
Consumers:
1. Name: vtbd0
   Mediasize: 137438953472 (128G)
   Sectorsize: 512
   Mode: r3w3e7

[2.7.2-RELEASE][root@pfSense.spacelab]/root: gpart show -p
=>       40  268435376    vtbd0  GPT  (128G)
         40     532480  vtbd0p1  efi  (260M)
     532520  259514368  vtbd0p2  freebsd-ufs  (124G)
  260046888    8388528  vtbd0p3  freebsd-swap  (4.0G)

After deleting some old logs and such, dushows just 4.8G of disk usage but df shows 26G used...

[2.7.2-RELEASE][root@pfSense.spacelab]/root: du -h -d1 /
4.0K	/.snap
101M	/boot
4.5K	/dev
 16M	/rescue
4.0K	/proc
 96M	/root
2.1G	/var
4.0K	/media
 12K	/conf.default
4.0K	/mnt
 13M	/tmp
4.6M	/sbin
 17M	/lib
4.0K	/net
 15M	/cf
164K	/libexec
8.0M	/etc
1.4M	/bin
1.8G	/usr
 20K	/home
1.1M	/tftpboot
4.8G	/

[2.7.2-RELEASE][root@pfSense.spacelab]/root: df -hi
Filesystem                     Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/ufsid/659ec5d8d33f9720    120G     26G     85G    23%     55k   16M    0%   /
devfs                          1.0K      0B    1.0K     0%       0     0     -   /dev
/dev/vtbd0p1                   260M    1.3M    259M     1%       2   510    0%   /boot/efi
tmpfs                          4.0M    192K    3.8M     5%      56   14k    0%   /var/run

sloopbun · Feb 13, 2024, 7:47 AM

I am not 100% sure, but I don't think I had this problem when I had pfsense on zfs, on top of the zvol in proxmox. I believe this kind of zfs on zfs is not recommended because it leads to write amplification, but it would be an improvement for me if another solution cannot be found.

I could go back to that. Or I could go back to bare-metal - but I am using this proxmox machine for backing up my other proxmox machine, so I would prefer not to at this stage. There is of course the other downside of losing proxmox managed backups and snapshots, but that is not a deal breaker.

Popolou · Feb 13, 2024, 2:16 PM

@sloopbun said in pfsense VM disk becomes full - please help identify the culprit?:

Since I had posted, my disk usage had gone from 13G to 19G.

Where are you checking this from, within pfsense or via the hypervisor/proxmox?

sloopbun · Feb 13, 2024, 2:16 PM

@Popolou said in pfsense VM disk becomes full - please help identify the culprit?:

@sloopbun said in pfsense VM disk becomes full - please help identify the culprit?:

Since I had posted, my disk usage had gone from 13G to 19G.

Where are you checking this from, within pfsense or via the hypervisor/proxmox?

From within pfsense. For example, below is the progression over a few hours today, but I do not see any change in the zvol reported on the host, proxmox.

root@pvepbs:~# date
Tue Feb 13 08:45:13 AM CET 2024

root@pvepbs:~# zfs list rpool/data/vm-100-disk-1
NAME                       USED  AVAIL  REFER  MOUNTPOINT
rpool/data/vm-100-disk-1  33.4G   684G  33.4G  -

root@pvepbs:~# date
Tue Feb 13 03:14:46 PM CET 2024

root@pvepbs:~# zfs list rpool/data/vm-100-disk-1
NAME                       USED  AVAIL  REFER  MOUNTPOINT
rpool/data/vm-100-disk-1  33.2G   684G  33.2G  -

[2.7.2-RELEASE][root@pfSense.spacelab]/root: date
Tue Feb 13 08:45:13 CET 2024

[2.7.2-RELEASE][root@pfSense.spacelab]/root: df -hi /
Filesystem                     Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/ufsid/659ec5d8d33f9720    120G     26G     85G    23%     55k   16M    0%   /

[2.7.2-RELEASE][root@pfSense.spacelab]/root: date
Tue Feb 13 15:07:25 CET 2024

[2.7.2-RELEASE][root@pfSense.spacelab]/root: df -hi /
Filesystem                     Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/ufsid/659ec5d8d33f9720    120G     32G     79G    29%     55k   16M    0%   /

[2.7.2-RELEASE][root@pfSense.spacelab]/root: du -h -d 1 /
4.0K	/.snap
101M	/boot
4.5K	/dev
 16M	/rescue
4.0K	/proc
 96M	/root
2.2G	/var
4.0K	/media
 12K	/conf.default
4.0K	/mnt
9.8M	/tmp
4.6M	/sbin
 17M	/lib
4.0K	/net
 15M	/cf
164K	/libexec
8.0M	/etc
1.4M	/bin
1.8G	/usr
 20K	/home
1.1M	/tftpboot
4.8G	/

serbus · Feb 13, 2024, 2:46 PM

@sloopbun said in pfsense VM disk becomes full - please help identify the culprit?:

I believe this kind of zfs on zfs is not recommended

Hello!

It is not recommended.

Comparing the output from df and du can be problematic, such as when you mount over the top of a folder that already has files in it, or a file has been deleted but is still open.

You may need to try something like lsof or fstat to dig deeper.

John

sloopbun · Feb 13, 2024, 6:35 PM

@serbus Thanks for the suggestion. Any recommendation incantations of fstat and lsof? I thought I might find unlinked files with lsof, but lsof +L1 / did not show any 0 in the NLINK column.

Similarly, I am not sure what I am looking for with fstat...

[2.7.2-RELEASE][root@pfSense.spacelab]/root: fstat | awk '{print $5}' | sort | uniq -c | sort -nr
 466 /
 276 /dev
 155 local
  94 internet
  83 pipe
  44
  39 /var/run
  35 internet6
  14 -
  12 netgraph
   3 route
   3 pseudo-terminal
   1 MOUNT

sloopbun · Feb 13, 2024, 6:54 PM

If I run five fsck from single user mode, as recommended in the docs, it shows no errors, even for the first run.

login-to-view

But when I run fsck from the regular pfsense terminal, I have a lot of unreferenced files, though the free space is reported correctly because this is just after a fresh reboot.

That is weird, right?

fsck:

[2.7.2-RELEASE][root@pfSense.spacelab]/root: fsck -y /dev/ufsid/659ec5d8d33f9720
** /dev/ufsid/659ec5d8d33f9720 (NO WRITE)
** SU+J Recovering /dev/ufsid/659ec5d8d33f9720

USE JOURNAL? no

Skipping journal, falling through to full fsck
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
INCORRECT BLOCK COUNT I=400686 (273856 should be 258944)
CORRECT? no

INCORRECT BLOCK COUNT I=641028 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=7371851 (383040 should be 382784)
CORRECT? no

INCORRECT BLOCK COUNT I=9615548 (896 should be 640)
CORRECT? no

INCORRECT BLOCK COUNT I=9615593 (144 should be 128)
CORRECT? no

** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
UNREF FILE I=400686  OWNER=root MODE=100600
SIZE=140112184 MTIME=Feb 13 19:53 2024

CLEAR? no

UNREF FILE I=15225275  OWNER=root MODE=100666
SIZE=0 MTIME=Feb 13 19:44 2024

CLEAR? no

** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? no

SUMMARY INFORMATION BAD
SALVAGE? no

BLK(S) MISSING IN BIT MAPS
SALVAGE? no

54742 files, 1317012 used, 30100372 free (4028 frags, 3762043 blocks, 0.0% fragmentation)

[2.7.2-RELEASE][root@pfSense.spacelab]/root: fsck -y /
** /dev/ufsid/659ec5d8d33f9720 (NO WRITE)
** SU+J Recovering /dev/ufsid/659ec5d8d33f9720

USE JOURNAL? no

Skipping journal, falling through to full fsck
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
INCORRECT BLOCK COUNT I=400686 (3136 should be 8)
CORRECT? no

INCORRECT BLOCK COUNT I=480777 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=480791 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=480799 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=480800 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=1041768 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=1041770 (2560 should be 768)
CORRECT? no

INCORRECT BLOCK COUNT I=1041771 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=2243678 (64 should be 8)
CORRECT? no

INCORRECT BLOCK COUNT I=3044945 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=3285252 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=3285261 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=3285294 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=3285301 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=3285324 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=6330120 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=6330142 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=7131638 (48 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=7131644 (960 should be 768)
CORRECT? no

INCORRECT BLOCK COUNT I=7131645 (16 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=7131718 (960 should be 768)
CORRECT? no

INCORRECT BLOCK COUNT I=7131722 (960 should be 768)
CORRECT? no

INCORRECT BLOCK COUNT I=7131734 (960 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=7131742 (960 should be 768)
CORRECT? no

INCORRECT BLOCK COUNT I=7131751 (960 should be 768)
CORRECT? no

INCORRECT BLOCK COUNT I=7371851 (380224 should be 380096)
CORRECT? no

INCORRECT BLOCK COUNT I=9615548 (624 should be 544)
CORRECT? no

INCORRECT BLOCK COUNT I=9615562 (48 should be 24)
CORRECT? no

INCORRECT BLOCK COUNT I=9695856 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=9855876 (2304 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=12100439 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=13542017 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=13542018 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=14102562 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=14102571 (48 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=14102821 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=14102834 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=15225288 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=15225289 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=15225290 (48 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=15225291 (8 should be 0)
CORRECT? no

** Phase 2 - Check Pathnames
SETTING DIRTY FLAG IN READ_ONLY MODE

UNEXPECTED SOFT UPDATE INCONSISTENCY
SETTING DIRTY FLAG IN READ_ONLY MODE

UNEXPECTED SOFT UPDATE INCONSISTENCY
SETTING DIRTY FLAG IN READ_ONLY MODE

UNEXPECTED SOFT UPDATE INCONSISTENCY
SETTING DIRTY FLAG IN READ_ONLY MODE

UNEXPECTED SOFT UPDATE INCONSISTENCY
** Phase 3 - Check Connectivity
UNREF DIR  I=7291717  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:44 2024

RECONNECT? no

UNREF DIR  I=7291716  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:44 2024

RECONNECT? no

UNREF DIR  I=7051269  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:44 2024

RECONNECT? no

UNREF DIR  I=480792  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:45 2024

RECONNECT? no

UNREF DIR  I=480801  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:45 2024

RECONNECT? no

** Phase 4 - Check Reference Counts
UNREF FILE  I=320567  OWNER=root MODE=100600
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=320569  OWNER=root MODE=100600
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=320570  OWNER=root MODE=100600
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=320571  OWNER=root MODE=100600
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=320572  OWNER=root MODE=100600
SIZE=888 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=320577  OWNER=root MODE=100600
SIZE=319 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=320579  OWNER=root MODE=100600
SIZE=503 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=320580  OWNER=root MODE=100600
SIZE=262 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

ZERO LENGTH DIR I=480777  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

ZERO LENGTH DIR I=480782  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

ZERO LENGTH DIR I=480783  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

ZERO LENGTH DIR I=480784  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

UNREF FILE  I=480785  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=480786  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=480787  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=480788  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=480789  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=480790  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

ZERO LENGTH DIR I=480791  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

UNREF DIR I=480792  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:45 2024

CLEAR? no

ZERO LENGTH DIR I=480799  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

ZERO LENGTH DIR I=480800  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

UNREF DIR I=480801  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:45 2024

CLEAR? no

UNREF FILE  I=961546  OWNER=root MODE=140666
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=1041770  OWNER=root MODE=100644
SIZE=1253376 MTIME=Feb  9 19:29 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=1041777  OWNER=root MODE=100644
SIZE=212992 MTIME=Dec  6 22:23 2023

RECONNECT? no


CLEAR? no

UNREF FILE  I=1041802  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=1041810  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=2564108  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=2564118  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=2564120  OWNER=root MODE=140755
SIZE=0 MTIME=Feb 13 19:40 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=2564121  OWNER=root MODE=100600
SIZE=16384 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=2564122  OWNER=root MODE=140755
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

LINK COUNT DIR I=6330112  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:45 2024  COUNT 6 SHOULD BE 5
ADJUST? no

UNREF FILE  I=6330120  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=6330140  OWNER=root MODE=100644
SIZE=221 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=6330142  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=6810887  OWNER=root MODE=140666
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF DIR I=7051269  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:44 2024

CLEAR? no

UNREF FILE  I=7131644  OWNER=root MODE=100644
SIZE=457710 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131656  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131662  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131681  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

LINK COUNT FILE I=7131688  OWNER=root MODE=100755
SIZE=348096 MTIME=Dec  8 18:07 2023  COUNT 4 SHOULD BE 3
ADJUST? no

UNREF FILE  I=7131705  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131710  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131715  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131718  OWNER=root MODE=100644
SIZE=457244 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131722  OWNER=root MODE=100644
SIZE=457710 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131734  OWNER=root MODE=100644
SIZE=457710 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131742  OWNER=root MODE=100644
SIZE=457244 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131747  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=7131751  OWNER=root MODE=100644
SIZE=457710 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF DIR I=7291716  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:44 2024

CLEAR? no

UNREF DIR I=7291717  OWNER=root MODE=40755
SIZE=512 MTIME=Feb 13 19:44 2024

CLEAR? no

LINK COUNT FILE I=9615549  OWNER=root MODE=100600
SIZE=11300 MTIME=Feb 12 22:11 2024  COUNT 2 SHOULD BE 1
ADJUST? no

LINK COUNT FILE I=9615560  OWNER=root MODE=100600
SIZE=11448 MTIME=Feb 13 12:49 2024  COUNT 2 SHOULD BE 1
ADJUST? no

UNREF FILE  I=9615576  OWNER=root MODE=100600
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

LINK COUNT FILE I=9615577  OWNER=root MODE=100600
SIZE=11293 MTIME=Feb 13 01:48 2024  COUNT 2 SHOULD BE 1
ADJUST? no

LINK COUNT FILE I=9615578  OWNER=root MODE=100600
SIZE=11263 MTIME=Feb 13 09:11 2024  COUNT 2 SHOULD BE 1
ADJUST? no

LINK COUNT FILE I=9615587  OWNER=root MODE=100600
SIZE=11258 MTIME=Feb 13 16:18 2024  COUNT 2 SHOULD BE 1
ADJUST? no

LINK COUNT FILE I=9615588  OWNER=root MODE=100600
SIZE=11317 MTIME=Feb 13 05:42 2024  COUNT 2 SHOULD BE 1
ADJUST? no

UNREF FILE  I=9615618  OWNER=root MODE=100600
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=9695987  OWNER=root MODE=100644
SIZE=2220 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

LINK COUNT DIR I=12099330  OWNER=root MODE=40755
SIZE=2560 MTIME=Feb 13 19:45 2024  COUNT 37 SHOULD BE 33
ADJUST? no

UNREF FILE  I=12180487  OWNER=root MODE=100644
SIZE=568 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

ZERO LENGTH DIR I=12180505  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

ZERO LENGTH DIR I=12180506  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

ZERO LENGTH DIR I=12180507  OWNER=root MODE=40755
SIZE=0 MTIME=Feb 13 19:45 2024

CLEAR? no

UNREF FILE  I=12180508  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=12180509  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=12180510  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=12180511  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=12180512  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=12180513  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE I=15225275  OWNER=root MODE=100666
SIZE=0 MTIME=Feb 13 19:44 2024

CLEAR? no

UNREF FILE  I=15225276  OWNER=root MODE=100666
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=15225278  OWNER=root MODE=100644
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=15225299  OWNER=root MODE=100666
SIZE=0 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=15225300  OWNER=root MODE=100644
SIZE=1776 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

UNREF FILE  I=15225301  OWNER=root MODE=100644
SIZE=156665 MTIME=Feb 13 19:45 2024

RECONNECT? no


CLEAR? no

** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? no

SUMMARY INFORMATION BAD
SALVAGE? no

BLK(S) MISSING IN BIT MAPS
SALVAGE? no

ALLOCATED FILE 15225275 MARKED FREE
54697 files, 1280056 used, 30136626 free (4026 frags, 3766575 blocks, 0.0% fragmentation)

[2.7.2-RELEASE][root@pfSense.spacelab]/root: df -hi /
Filesystem                     Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/ufsid/659ec5d8d33f9720    120G    4.9G    105G     4%     55k   16M    0%   /

serbus · Feb 15, 2024, 8:28 AM

@sloopbun

Hello!

This discussion and similar might have other leads you can follow.

John

sloopbun · Mar 4, 2024, 8:36 AM

@serbus Thanks. Open, or zombie procresses/files is a likely candidate, though I cannot find any. In that link there are a couple of commands for linux that makes sense to me, but either do not turn up any results, or do not work on freebsd. Any idea what their equivalent might be?

I am referring to are:

lsof | grep -i deleted

which returns nothing, and:

[2.7.2-RELEASE][root@pfSense.spacelab]/: mount -o bind / /mnt
mount: /: mount option <bind> is unknown: Invalid argument
[2.7.2-RELEASE][root@pfSense.spacelab]/:

which I think has an equivalent freebsd of:

mount -t nullfs / /mnt

Though once mounted, appear to be identical to /. No extra files in /mnt/var or anything...

Was there any other avenue of exploration you saw that might help?

Apart from the zombie processes/files, the other likely candidate in my mind is trim/discard/clean-up. What do you think about turning on trim for UFS in pfsense?

sloopbun · Mar 4, 2024, 8:36 AM

To add an unhelpful update. This problem was still occurring a few days ago, so I restarted as usual to start the ticking time bomb again.

I now have almost four days of uptime and available disk is in line with disk usage.

I swear I did not change anything or try to fix anything since this thread went quiet. In fact I have been convincing myself to go back to bare metal. Super weird and I don't trust it...

[2.7.2-RELEASE][root@pfSense.spacelab]/root: uptime
 9:19AM  up 3 days, 22:56, 2 users, load averages: 0.33, 0.22, 0.18

[2.7.2-RELEASE][root@pfSense.spacelab]/root: df -h /
Filesystem                     Size    Used   Avail Capacity  Mounted on
/dev/ufsid/659ec5d8d33f9720    120G    5.8G    104G     5%    /

[2.7.2-RELEASE][root@pfSense.spacelab]/root: du -h -d 1 /
4.0K	/.snap
101M	/boot
4.5K	/dev
 16M	/rescue
4.0K	/proc
 96M	/root
3.1G	/var
4.0K	/media
 12K	/conf.default
4.0K	/mnt
9.9M	/tmp
4.6M	/sbin
 17M	/lib
4.0K	/net
 16M	/cf
164K	/libexec
8.0M	/etc
1.4M	/bin
1.8G	/usr
 20K	/home
1.1M	/tftpboot
5.8G	/

GPz1100 · May 9, 2024, 8:52 AM

@sloopbun Did you ever get this resolved? I'm about to commit to a proxmox vm install myself.