DHCPD errors /: filesystem full - but really it is not!



  • I keep getting the following errors:
    kernel: pid 40848 (dhcpd), uid 1002 inumber 9963101 on /: filesystem full
    df -ih tells me I don't have a problem with the root filesystem /
    $ df -ih
    Filesystem    Size    Used  Avail Capacity iused ifree %iused  Mounted on
    /dev/ad0s1a    81G    63G    11G    85%    9.5M  1.5M  86%  /
    devfs          1.0k    1.0k      0B  100%      0    0  100%  /dev
    /dev/md0      3.6M    46k    3.3M    1%      27  739    4%  /var/run
    devfs          1.0k    1.0k      0B  100%      0    0  100%  /var/dhcpd/dev

    This PFSense runs inside a proxmox VM that all sits on a samsung SSD…  I have checked the SMART status of the drive and it is happy.. Zero write errors no wear-out. 
    I am running a significant squid proxy which accounts for most of the hard disk use.. but i am not near 100% yet.. Heaps of RAM and cpu resources assigned. Any help would be appreciated.. The system crashes every 6 hours with loads of  kernel: pid 40848 (dhcpd), uid 1002 inumber 9963101 on /: filesystem full.
    Today I turned off all the logging and RRD in the hope it stabilizes. Needless to say this is not the way I want to run long term.
    Cheers



  • It fell over again last night with all the logging turned off. Any suggestions people!



  • Weird.  First thing I would do is blow my cache away and then recreate it with some reasonable values like 200-500 MB, and see if it lives more than a day or two before failing again.  Your cache hit rate, if it's anything like mine or what I see online, is going to be in the area of 4-7% – not great.  Using up many gigs of space for a tiny hit rate is a waste.  I find Squid's value these days is more as a filtering platform with SquidGuard.


  • Rebel Alliance Developer Netgate

    When a disk is claiming to be full but does not appear to be full, there are two possibilities:

    1. The disk is not out of capacity, but out of inodes (too many small files, for example). Use "df -hi" to confirm.
    2. The disk is failing and the OS cannot contact the disk to make a write, or the disk returned a write error.

    2 is far more common than 1.

    There is also a third option but it does not apply to your setup:
    3. The /var RAM disk (NanoBSD or full w/the option to use it on) is full, not /



  • Thanks Jimp and KOM.

    KOM, When reducing the squid cache to 500mb and flushing squid did not free up any space I started to hunt for the culprit.
    Jimp I wish I looked at the inodes before i started deleting 60 Gig of SARG files (Just to be sure). There were a zillion small files in there. running a single rm -R * on the sarg-reports sub directories took 7 hours to complete.  Now my disk space is pitifully empty:
    $ df -hi
    Filesystem    Size    Used  Avail Capacity iused ifree %iused  Mounted on
    /dev/ad0s1a    81G    1.9G    72G    3%    42k  11M    0%  /
    devfs          1.0k    1.0k      0B  100%      0    0  100%  /dev
    /dev/md0      3.6M    46k    3.3M    1%      27  739    4%  /var/run
    devfs          1.0k    1.0k      0B  100%      0    0  100%  /var/dhcpd/dev

    Thanks for your help! looks like we solved this one.. I am putting it down to the inodes being full.
    Needless to say I removed the zero in the reporting options
    Cheers..