Disk Full - but not seeing how

Boethius

I noticed problem first when attempting to login to the webui, which would not load beyond the sign-in page. I connect via serial and determine I've got a full disk. Hardware is an APU4c4 with 12G m2 SSD on ZFS. Only package is ntopng

[2.5.0-RELEASE][root@pfsense-xxx]/: df -a
Filesystem         1K-blocks     Used Avail Capacity  Mounted on
zroot/ROOT/default  12613408 12613408     0   100%    /
devfs                      1        1     0   100%    /dev
zroot/tmp                204      204     0   100%    /tmp
zroot/var              82156    82156     0   100%    /var
zroot                     88       88     0   100%    /zroot
/dev/md0                3484      108  3100     3%    /var/run
devfs                      1        1     0   100%    /var/dhcpd/dev

[2.5.0-RELEASE][root@pfsense-xxx]/var/db: zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zroot  12.5G  12.1G   400M        -         -    74%    96%  1.00x  ONLINE  -

[2.5.0-RELEASE][root@pfsense-xxx]/: zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
zroot               12.1G      0    88K  /zroot
zroot/ROOT          12.0G      0    88K  none
zroot/ROOT/default  12.0G      0  12.0G  /
zroot/tmp            204K      0   204K  /tmp
zroot/var           80.2M      0  80.2M  /var

Below, the tail output of du -h from within /

60K    /etc/inc/web
52K    /etc/inc/priv
1.2M    /etc/inc
512B    /etc/ntp
629K    /etc/rc.d
36K    /etc/devd
72K    /etc/pam.d
5.5K    /etc/gss
3.8M    /etc
512B    /net
36K    /libexec/resolvconf
141K    /libexec
512B    /proc
152M    /root
484K    /cf/conf/backup
636K    /cf/conf
636K    /cf
512B    /lib/casper
22K    /lib/nvmecontrol
241K    /lib/geom
7.3M    /lib
1.1G    /

Any ideas?

Boethius

Ruling out snapshots.

/: zfs list -t snapshot zroot/ROOT/default
cannot open 'zroot/ROOT/default': missing '@' delimiter in snapshot name

and

zfs list -t snapshot 
no datasets available

Boethius

I never updated this with resolution because I ended up reinstalling pfSense.

I do believe I have figured out the cause however. I'd classify this as a bug.

Scenario: Run a PCAP with no limit on capture size or # of packets, allowing it to completely fill local storage.
Via Console connection or, if working (sometimes it doesn't with full filesystem) SSH into pfSense, run rm capture.cap

Then, you will run df -h and see no change. Only when you go to the webgui and "stop capture" will the change in storage be realized and the system be usable again. I encountered this same problem having left a cap running on accident and after deleting the 13GB capture file, still not having space show in df, then stopping the capture in webgui, voila, everything as expected.

Derelict

@boethius Likely because the file being removed is still open by the capture process.

I don't think that is going to be considered a bug.

Boethius

@derelict whatever it is, it was unclear what was causing the system to be unusable. Seeing output of df -h showing an abundance of space, but still having file system full errors, that's confusing. Can you agree?

Derelict

@boethius Always put a packet count on your packet captures.

If it is a "bug" it is a bug in the FreeeBSD implementation of ZFS.

Boethius

@derelict No, it isn't ZFS related... I rediscovered on this subsequent installation, in which I opted for UFS, thinking the previous file system full had something to do with ZFS. I know I was a barbarian when initiating an uncapped packet capture. But still this behavior is odd.

Derelict

@boethius Can only go by what you're showing us.

If I really looked at it that is probably standard behavior for removing a file with an open file descriptor.

Boethius

@derelict

I had lost webui access entirely... so I didn't have the opportunity to see the capture process still running. I had forgotten about it.

Likely because the file being removed is still open by the capture process.

Is there a command to see what files have open file descriptors?