Disk Full - but not seeing how
-
I noticed problem first when attempting to login to the webui, which would not load beyond the sign-in page. I connect via serial and determine I've got a full disk. Hardware is an APU4c4 with 12G m2 SSD on ZFS. Only package is ntopng
[2.5.0-RELEASE][root@pfsense-xxx]/: df -a Filesystem 1K-blocks Used Avail Capacity Mounted on zroot/ROOT/default 12613408 12613408 0 100% / devfs 1 1 0 100% /dev zroot/tmp 204 204 0 100% /tmp zroot/var 82156 82156 0 100% /var zroot 88 88 0 100% /zroot /dev/md0 3484 108 3100 3% /var/run devfs 1 1 0 100% /var/dhcpd/dev
[2.5.0-RELEASE][root@pfsense-xxx]/var/db: zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zroot 12.5G 12.1G 400M - - 74% 96% 1.00x ONLINE -
[2.5.0-RELEASE][root@pfsense-xxx]/: zfs list NAME USED AVAIL REFER MOUNTPOINT zroot 12.1G 0 88K /zroot zroot/ROOT 12.0G 0 88K none zroot/ROOT/default 12.0G 0 12.0G / zroot/tmp 204K 0 204K /tmp zroot/var 80.2M 0 80.2M /var
Below, the tail output of du -h from within /
60K /etc/inc/web 52K /etc/inc/priv 1.2M /etc/inc 512B /etc/ntp 629K /etc/rc.d 36K /etc/devd 72K /etc/pam.d 5.5K /etc/gss 3.8M /etc 512B /net 36K /libexec/resolvconf 141K /libexec 512B /proc 152M /root 484K /cf/conf/backup 636K /cf/conf 636K /cf 512B /lib/casper 22K /lib/nvmecontrol 241K /lib/geom 7.3M /lib 1.1G /
Any ideas?
-
Ruling out snapshots.
/: zfs list -t snapshot zroot/ROOT/default cannot open 'zroot/ROOT/default': missing '@' delimiter in snapshot name
and
zfs list -t snapshot no datasets available
-
I never updated this with resolution because I ended up reinstalling pfSense.
I do believe I have figured out the cause however. I'd classify this as a bug.
Scenario: Run a PCAP with no limit on capture size or # of packets, allowing it to completely fill local storage.
Via Console connection or, if working (sometimes it doesn't with full filesystem) SSH into pfSense, runrm capture.cap
Then, you will run df -h and see no change. Only when you go to the webgui and "stop capture" will the change in storage be realized and the system be usable again. I encountered this same problem having left a cap running on accident and after deleting the 13GB capture file, still not having space show in
df
, then stopping the capture in webgui, voila, everything as expected. -
@boethius Likely because the file being removed is still open by the capture process.
I don't think that is going to be considered a bug.
-
@derelict whatever it is, it was unclear what was causing the system to be unusable. Seeing output of df -h showing an abundance of space, but still having file system full errors, that's confusing. Can you agree?
-
@boethius Always put a packet count on your packet captures.
If it is a "bug" it is a bug in the FreeeBSD implementation of ZFS.
-
@derelict No, it isn't ZFS related... I rediscovered on this subsequent installation, in which I opted for UFS, thinking the previous file system full had something to do with ZFS. I know I was a barbarian when initiating an uncapped packet capture. But still this behavior is odd.
-
@boethius Can only go by what you're showing us.
If I really looked at it that is probably standard behavior for removing a file with an open file descriptor.
-
I had lost webui access entirely... so I didn't have the opportunity to see the capture process still running. I had forgotten about it.
Likely because the file being removed is still open by the capture process.
Is there a command to see what files have open file descriptors?