DHCPD errors /: filesystem full - but really it is not!
-
I keep getting the following errors:
kernel: pid 40848 (dhcpd), uid 1002 inumber 9963101 on /: filesystem full
df -ih tells me I don't have a problem with the root filesystem /
$ df -ih
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/ad0s1a 81G 63G 11G 85% 9.5M 1.5M 86% /
devfs 1.0k 1.0k 0B 100% 0 0 100% /dev
/dev/md0 3.6M 46k 3.3M 1% 27 739 4% /var/run
devfs 1.0k 1.0k 0B 100% 0 0 100% /var/dhcpd/devThis PFSense runs inside a proxmox VM that all sits on a samsung SSD… I have checked the SMART status of the drive and it is happy.. Zero write errors no wear-out.
I am running a significant squid proxy which accounts for most of the hard disk use.. but i am not near 100% yet.. Heaps of RAM and cpu resources assigned. Any help would be appreciated.. The system crashes every 6 hours with loads of kernel: pid 40848 (dhcpd), uid 1002 inumber 9963101 on /: filesystem full.
Today I turned off all the logging and RRD in the hope it stabilizes. Needless to say this is not the way I want to run long term.
Cheers -
It fell over again last night with all the logging turned off. Any suggestions people!
-
Weird. First thing I would do is blow my cache away and then recreate it with some reasonable values like 200-500 MB, and see if it lives more than a day or two before failing again. Your cache hit rate, if it's anything like mine or what I see online, is going to be in the area of 4-7% – not great. Using up many gigs of space for a tiny hit rate is a waste. I find Squid's value these days is more as a filtering platform with SquidGuard.
-
When a disk is claiming to be full but does not appear to be full, there are two possibilities:
1. The disk is not out of capacity, but out of inodes (too many small files, for example). Use "df -hi" to confirm.
2. The disk is failing and the OS cannot contact the disk to make a write, or the disk returned a write error.2 is far more common than 1.
There is also a third option but it does not apply to your setup:
3. The /var RAM disk (NanoBSD or full w/the option to use it on) is full, not / -
Thanks Jimp and KOM.
KOM, When reducing the squid cache to 500mb and flushing squid did not free up any space I started to hunt for the culprit.
Jimp I wish I looked at the inodes before i started deleting 60 Gig of SARG files (Just to be sure). There were a zillion small files in there. running a single rm -R * on the sarg-reports sub directories took 7 hours to complete. Now my disk space is pitifully empty:
$ df -hi
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/ad0s1a 81G 1.9G 72G 3% 42k 11M 0% /
devfs 1.0k 1.0k 0B 100% 0 0 100% /dev
/dev/md0 3.6M 46k 3.3M 1% 27 739 4% /var/run
devfs 1.0k 1.0k 0B 100% 0 0 100% /var/dhcpd/devThanks for your help! looks like we solved this one.. I am putting it down to the inodes being full.
Needless to say I removed the zero in the reporting options
Cheers..