/: create/symlink failed, no inodes free



  • At random occasions I got this message when I login over ssh to pfsense

    /: create/symlink failed, no inodes free
    [Wed Mar  3 00:04:51 2010] [apc-error] apc_fcntl_create: open(/tmp/.apc.mBZJOa, O_RDWR|O_CREAT, 0666) failed: No space left on device

    shell works partially, but DHCP server is not working anymore, after restart everything becomes normal again until next crash that can happen sometimes in few hours or more.

    # df -h
    Filesystem     Size    Used   Avail Capacity  Mounted on
    /dev/ad0s1a    3.8G    309M    3.2G     9%    /
    devfs          1.0K    1.0K      0B   100%    /dev
    /dev/md0       3.6M     42K    3.3M     1%    /var/run
    devfs          1.0K    1.0K      0B   100%    /var/dhcpd/dev
    
    # du
    /usr/bin/du: Device not configured.
    
    # ls -la
    ls: .cshrc: Device not configured
    ls: .cvsignore: Device not configured
    ls: .profile: Device not configured
    ls: .snap: Device not configured
    ls: COPYRIGHT: Device not configured
    ls: RELENG_1_2: Device not configured
    ls: boot.config: Device not configured
    ls: conf.default: Device not configured
    ls: kernels: Device not configured
    ls: media: Device not configured
    ls: mnt: Device not configured
    ls: needs_package_sync: Device not configured
    ls: ntop.core: Device not configured
    ls: rescue: Device not configured
    ls: scripts: Device not configured
    ls: touch: Device not configured
    total 33
    drwxr-xr-x  23 root  wheel  1024 Feb 26 07:29 .
    drwxr-xr-x  23 root  wheel  1024 Feb 26 07:29 ..
    drwxr-xr-x   2 root  wheel  1024 Nov 11 13:56 bin
    drwxr-xr-x   7 root  wheel   512 Feb 26 07:29 boot
    drwxr-xr-x   3 root  wheel   512 Jul  7  2009 cf
    lrwxr-xr-x   1 root  wheel     8 Jul  7  2009 conf -> /cf/conf
    dr-xr-xr-x   5 root  wheel   512 Mar  2 08:59 dev
    drwxr-xr-x  25 root  wheel  4096 Mar  2 09:00 etc
    drwxr-xr-x   4 root  wheel   512 Feb 26 03:28 home
    drwxr-xr-x   3 root  wheel  1536 Nov 11 13:56 lib
    drwxr-xr-x   2 root  wheel   512 Nov 11 13:56 libexec
    dr-xr-xr-x   2 root  wheel   512 Feb 26 03:27 proc
    drwxr-xr-x   2 root  wheel   512 Feb 26 13:57 root
    drwxr-xr-x   2 root  wheel  1536 Nov 11 13:56 sbin
    drwxrwxrwt   4 root  wheel  1024 Mar  2 16:12 tmp
    drwxr-xr-x  11 root  wheel   512 Jan  9  2009 usr
    drwxr-xr-x  25 root  wheel   512 Feb 26 03:13 var
    
    

    I also noticed  that open files count increases with usage of web configurator but never decreases.

    sysctl kern.openfiles

    kern.openfiles: 2882


  • Rebel Alliance Developer Netgate

    You may need to upgrade to a new snapshot and try that again.

    The messages you are seeing though indicate you are out of inodes, not out of space. There are only so many inodes per file system, each inode points to a single file. So if you have a few million small files you can run out of inodes long before running out of space. A loose analogy: Like a parking lot, there are only so many spaces, it doesn't matter if you have motorcycles or SUVs, the number of vehicles the lot should hold is the same.

    Use "df -i" to check inode usage.



  • before restart:

    # df -i
    Filesystem  1K-blocks   Used   Avail Capacity iused  ifree %iused  Mounted on
    /dev/ad0s1a   3997382 316750 3360842     9%    8356 509786    2%   /
    devfs               1      1       0   100%       0      0  100%   /dev
    /dev/md0         3694     42    3358     1%      20    746    3%   /var/run
    devfs               1      1       0   100%       0      0  100%   /var/dhcpd/dev
    
    

    after restart:

    /dev/ad0s1a   3997382 316750 3360842     9%    8352 509790    2%   /
    devfs               1      1       0   100%       0      0  100%   /dev
    /dev/md0         3694     42    3358     1%      20    746    3%   /var/run
    devfs               1      1       0   100%       0      0  100%   /var/dhcpd/dev
    
    

    Now I will try to get latest snapshot and see what happens….


Log in to reply