/: create/symlink failed, no inodes free
-
At random occasions I got this message when I login over ssh to pfsense
/: create/symlink failed, no inodes free
[Wed Mar 3 00:04:51 2010] [apc-error] apc_fcntl_create: open(/tmp/.apc.mBZJOa, O_RDWR|O_CREAT, 0666) failed: No space left on deviceshell works partially, but DHCP server is not working anymore, after restart everything becomes normal again until next crash that can happen sometimes in few hours or more.
# df -h Filesystem Size Used Avail Capacity Mounted on /dev/ad0s1a 3.8G 309M 3.2G 9% / devfs 1.0K 1.0K 0B 100% /dev /dev/md0 3.6M 42K 3.3M 1% /var/run devfs 1.0K 1.0K 0B 100% /var/dhcpd/dev # du /usr/bin/du: Device not configured. # ls -la ls: .cshrc: Device not configured ls: .cvsignore: Device not configured ls: .profile: Device not configured ls: .snap: Device not configured ls: COPYRIGHT: Device not configured ls: RELENG_1_2: Device not configured ls: boot.config: Device not configured ls: conf.default: Device not configured ls: kernels: Device not configured ls: media: Device not configured ls: mnt: Device not configured ls: needs_package_sync: Device not configured ls: ntop.core: Device not configured ls: rescue: Device not configured ls: scripts: Device not configured ls: touch: Device not configured total 33 drwxr-xr-x 23 root wheel 1024 Feb 26 07:29 . drwxr-xr-x 23 root wheel 1024 Feb 26 07:29 .. drwxr-xr-x 2 root wheel 1024 Nov 11 13:56 bin drwxr-xr-x 7 root wheel 512 Feb 26 07:29 boot drwxr-xr-x 3 root wheel 512 Jul 7 2009 cf lrwxr-xr-x 1 root wheel 8 Jul 7 2009 conf -> /cf/conf dr-xr-xr-x 5 root wheel 512 Mar 2 08:59 dev drwxr-xr-x 25 root wheel 4096 Mar 2 09:00 etc drwxr-xr-x 4 root wheel 512 Feb 26 03:28 home drwxr-xr-x 3 root wheel 1536 Nov 11 13:56 lib drwxr-xr-x 2 root wheel 512 Nov 11 13:56 libexec dr-xr-xr-x 2 root wheel 512 Feb 26 03:27 proc drwxr-xr-x 2 root wheel 512 Feb 26 13:57 root drwxr-xr-x 2 root wheel 1536 Nov 11 13:56 sbin drwxrwxrwt 4 root wheel 1024 Mar 2 16:12 tmp drwxr-xr-x 11 root wheel 512 Jan 9 2009 usr drwxr-xr-x 25 root wheel 512 Feb 26 03:13 varI also noticed that open files count increases with usage of web configurator but never decreases.
sysctl kern.openfiles
kern.openfiles: 2882
-
You may need to upgrade to a new snapshot and try that again.
The messages you are seeing though indicate you are out of inodes, not out of space. There are only so many inodes per file system, each inode points to a single file. So if you have a few million small files you can run out of inodes long before running out of space. A loose analogy: Like a parking lot, there are only so many spaces, it doesn't matter if you have motorcycles or SUVs, the number of vehicles the lot should hold is the same.
Use "df -i" to check inode usage.
-
before restart:
# df -i Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/ad0s1a 3997382 316750 3360842 9% 8356 509786 2% / devfs 1 1 0 100% 0 0 100% /dev /dev/md0 3694 42 3358 1% 20 746 3% /var/run devfs 1 1 0 100% 0 0 100% /var/dhcpd/devafter restart:
/dev/ad0s1a 3997382 316750 3360842 9% 8352 509790 2% / devfs 1 1 0 100% 0 0 100% /dev /dev/md0 3694 42 3358 1% 20 746 3% /var/run devfs 1 1 0 100% 0 0 100% /var/dhcpd/devNow I will try to get latest snapshot and see what happens….
