/: create/symlink failed, no inodes free

[ablaskovic]

At random occasions I got this message when I login over ssh to pfsense

/: create/symlink failed, no inodes free
[Wed Mar  3 00:04:51 2010] [apc-error] apc_fcntl_create: open(/tmp/.apc.mBZJOa, O_RDWR|O_CREAT, 0666) failed: No space left on device

shell works partially, but DHCP server is not working anymore, after restart everything becomes normal again until next crash that can happen sometimes in few hours or more.

# df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ad0s1a    3.8G    309M    3.2G     9%    /
devfs          1.0K    1.0K      0B   100%    /dev
/dev/md0       3.6M     42K    3.3M     1%    /var/run
devfs          1.0K    1.0K      0B   100%    /var/dhcpd/dev

# du
/usr/bin/du: Device not configured.

# ls -la
ls: .cshrc: Device not configured
ls: .cvsignore: Device not configured
ls: .profile: Device not configured
ls: .snap: Device not configured
ls: COPYRIGHT: Device not configured
ls: RELENG_1_2: Device not configured
ls: boot.config: Device not configured
ls: conf.default: Device not configured
ls: kernels: Device not configured
ls: media: Device not configured
ls: mnt: Device not configured
ls: needs_package_sync: Device not configured
ls: ntop.core: Device not configured
ls: rescue: Device not configured
ls: scripts: Device not configured
ls: touch: Device not configured
total 33
drwxr-xr-x  23 root  wheel  1024 Feb 26 07:29 .
drwxr-xr-x  23 root  wheel  1024 Feb 26 07:29 ..
drwxr-xr-x   2 root  wheel  1024 Nov 11 13:56 bin
drwxr-xr-x   7 root  wheel   512 Feb 26 07:29 boot
drwxr-xr-x   3 root  wheel   512 Jul  7  2009 cf
lrwxr-xr-x   1 root  wheel     8 Jul  7  2009 conf -> /cf/conf
dr-xr-xr-x   5 root  wheel   512 Mar  2 08:59 dev
drwxr-xr-x  25 root  wheel  4096 Mar  2 09:00 etc
drwxr-xr-x   4 root  wheel   512 Feb 26 03:28 home
drwxr-xr-x   3 root  wheel  1536 Nov 11 13:56 lib
drwxr-xr-x   2 root  wheel   512 Nov 11 13:56 libexec
dr-xr-xr-x   2 root  wheel   512 Feb 26 03:27 proc
drwxr-xr-x   2 root  wheel   512 Feb 26 13:57 root
drwxr-xr-x   2 root  wheel  1536 Nov 11 13:56 sbin
drwxrwxrwt   4 root  wheel  1024 Mar  2 16:12 tmp
drwxr-xr-x  11 root  wheel   512 Jan  9  2009 usr
drwxr-xr-x  25 root  wheel   512 Feb 26 03:13 var

I also noticed  that open files count increases with usage of web configurator but never decreases.

sysctl kern.openfiles

kern.openfiles: 2882

[jimp]

You may need to upgrade to a new snapshot and try that again.

The messages you are seeing though indicate you are out of inodes, not out of space. There are only so many inodes per file system, each inode points to a single file. So if you have a few million small files you can run out of inodes long before running out of space. A loose analogy: Like a parking lot, there are only so many spaces, it doesn't matter if you have motorcycles or SUVs, the number of vehicles the lot should hold is the same.

Use "df -i" to check inode usage.

[ablaskovic]

before restart:

# df -i
Filesystem  1K-blocks   Used   Avail Capacity iused  ifree %iused  Mounted on
/dev/ad0s1a   3997382 316750 3360842     9%    8356 509786    2%   /
devfs               1      1       0   100%       0      0  100%   /dev
/dev/md0         3694     42    3358     1%      20    746    3%   /var/run
devfs               1      1       0   100%       0      0  100%   /var/dhcpd/dev

after restart:

/dev/ad0s1a   3997382 316750 3360842     9%    8352 509790    2%   /
devfs               1      1       0   100%       0      0  100%   /dev
/dev/md0         3694     42    3358     1%      20    746    3%   /var/run
devfs               1      1       0   100%       0      0  100%   /var/dhcpd/dev

Now I will try to get latest snapshot and see what happens….