Ran out of inodes

ewildgoose

Hi, using a snapshot around early Dec 2010 ish (2.0 beta4), I have tried (and failed) to login to the web interface. Give me an error message:

Warning: fopen(/tmp/config.lock): failed to open stream: Device not configured in /etc/inc/util.inc on line 123 Warning: flock() expects parameter 1 to be resource, null given in /etc/inc/util.inc on line 134 Warning: fclose(): supplied argument is not a valid stream resource in /etc/inc/util.inc on line 135 Warning: session_start(): open(/var/tmp//sess_c9cf1581880d10889fcf3ddcb324b7ee, O_RDWR) failed: No space left on device (28) in /etc/inc/auth.inc on line 1204

Logging in via ssh, I see:
: df -hi
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/ad0s1a 1.9G 208M 1.6G 11% 6.2k 276k 2% /
devfs 1.0K 1.0K 0B 100% 0 0 100% /dev
/dev/md0 3.6M 38K 3.3M 1% 24 742 3% /var/run

So I presume that something is creating and deleting files, but keeping the file open? How else can I be out of inodes?

Any suggestions on things to check quickly before rebooting (which I presume will fix the issue?). Is there an equiv of "lsof" available? (Sorry, I'm a linux user normally)

Note:

: ps x
PID TT STAT TIME COMMAND
0 ?? DLs 98:47.32 [kernel]
1 ?? ILs 0:00.43 /sbin/init –
2 ?? DL 2:08.30 [g_event]
3 ?? DL 1:02.35 [g_up]
4 ?? DL 0:29.10 [g_down]
5 ?? DL 0:00.00 [crypto]
6 ?? DL 0:00.00 [crypto returns]
7 ?? DL 1:15.11 [pfpurge]
8 ?? DL 0:00.00 [xpt_thrd]
9 ?? DL 0:08.30 [pagedaemon]
10 ?? RL 155078:06.10 [idle]
11 ?? WL 153:00.53 [intr]
12 ?? DL 0:04.47 [ng_queue]
13 ?? DL 12:23.39 [yarrow]
14 ?? DL 0:45.90 [usb]
15 ?? DL 0:00.00 [vmdaemon]
16 ?? DL 0:04.93 [idlepoll]
17 ?? DL 0:00.04 [pagezero]
18 ?? DL 0:15.11 [bufdaemon]
19 ?? DL 2:35.25 [syncer]
20 ?? DL 0:16.11 [vnlru]
21 ?? DL 0:18.18 [softdepflush]
32 ?? DL 0:02.23 [md0]
236 ?? INs 0:00.14 /usr/local/sbin/check_reload_status
241 ?? IN 0:00.00 check_reload_status: Monitoring daemon of check_reload_status
249 ?? Is 0:00.00 /sbin/devd
4375 ?? Ss 0:03.11 ntpd: [priv] (ntpd)
5394 ?? IN 29:24.89 /bin/sh /var/db/rrd/updaterrd.sh
7260 ?? INs 0:00.01 /usr/sbin/sshd
7591 ?? Ss 1:10.11 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_wan.conf -p /var/run/pppoe_wan.pid -s ppp pppoe
8082 ?? Ss 8:32.12 /usr/sbin/syslogd -c -l /var/dhcpd/var/run/log -f /var/etc/syslog.conf
11841 ?? I 0:13.55 /usr/local/bin/php
13505 ?? Is 0:22.24 /usr/sbin/inetd -wW -R 0 -a 127.0.0.1 /var/etc/inetd.conf
15076 ?? Is 0:00.02 /usr/local/sbin/sshlockout_pf
16761 ?? SN 1:17.49 /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf
22087 ?? Is 0:00.12 /usr/local/bin/php
22713 ?? Is 0:00.18 /usr/local/bin/php
26611 ?? SNs 0:00.17 sshd: root@pts/0 (sshd)
29425 ?? I 0:12.55 /usr/local/bin/php
33502 ?? Ss 0:25.24 openvpn –config /var/etc/openvpn/server1.conf
34120 ?? Is 0:04.40 dhclient: em0 [priv] (dhclient)
34456 ?? Is 0:23.56 /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf
34508 ?? SNs 4:27.36 /usr/local/sbin/apinger -c /var/etc/apinger.conf
34603 ?? IN 0:06.89 /usr/local/bin/rrdtool -
36488 ?? IN 0:00.01 sleep 60
36564 ?? IN 0:00.01 /usr/bin/top -d 2 -s 1 0
36775 ?? IN 0:00.00 [awk]
37446 ?? I 0:10.84 /usr/local/bin/php
45148 ?? Is 0:03.41 /usr/sbin/cron -s
46805 ?? I 0:04.38 /usr/local/bin/php
58168 ?? Is 0:02.51 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh
58746 ?? Is 0:00.16 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /etc/rc.expireaccounts
58951 ?? Is 0:00.01 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /etc/rc.update_alias_url_dat
9358 u0- S 3:26.01 /usr/sbin/tcpdump -s 256 -v -l -n -e -ttt -i pflog0
9487 u0- S 2:54.79 logger -t pf -p local0.info
62539 u0 Is 0:00.02 login [pam] (login)
62604 u0 I 0:00.01 -sh (sh)
63719 u0 I+ 0:00.01 /bin/sh /etc/rc.initial
19916 0 S 0:00.11 /bin/tcsh
26713 0 Is 0:00.01 -sh (sh)
27681 0 I 0:00.02 /bin/sh /etc/rc.initial
36930 0 R+ 0:00.00 ps x

Thanks

mikesamo

This is a common problem when you have many files of a small size
Rebooting will not fix this.

try delete huge log file after you will be able to access webgui.

ewildgoose

Thanks for the thoughts, but your advice seems contradictory?

Firstly it starts working again after I reboot

Secondly you say it's a problem of lots of small files (why?), but then your suggestion is to delete a single (huge) file to recover from that situation? Don't understand how both of these can be correct?

Finally, looking at the df output we can clearly see loads of free inodes? That seems inconsistent with your theory?

Here is the inode count after rebooting:

: df -ih
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/ad0s1a 1.9G 206M 1.6G 11% 6.1k 277k 2% /
devfs 1.0K 1.0K 0B 100% 0 0 100% /dev
/dev/md0 3.6M 38K 3.3M 1% 24 742 3% /var/run

So, I think my question remains? Why am I running out of inodes? I suspect either disk corruption or a huge amount of temp files created which are deleted but not closed?

Thanks for any thoughts?

danswartz

why do you think it is out of inodes? i don't see that in text you quoted?

cmb

You're not out of inodes (devfs isn't a file system, it will always show that), must have a package on there that's dropping a lot of data.

ewildgoose

Aha, apparently I slightly skipped a bit of debug. Firstly notice the main error I mention about the failure to get into the web interface, this is a symptom of being unable to create a file in /tmp due to being out of inodes

Additionally ssh-ing in and a simple "touch /tmp/abcd" would give an error that I don't have to hand, but something like "cannot create file, out of inodes"

Additionally the log files are stuffed full of errors along the lines of "cannot create file xxxx"

Before rebooting I killed almost every process that I could and despite that I still couldn't create any new files on my root filesystem. After a reboot all was fine…

Any thoughts?

danswartz

How about answering my question as to why you believe you are out of inodes?

Cry Havok

@ewildgoose:

: df -ih
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/ad0s1a 1.9G 208M 1.6G 11% 6.2k 276k 2% /
devfs 1.0K 1.0K 0B 100% 0 0 100% /dev
/dev/md0 3.6M 38K 3.3M 1% 24 742 3% /var/run

I'm with danswartz, there's nothing there to show you're running out of inodes (the entry for devfs is normal - that's not a real file system). Whatever your problem is, it is not related to running out of inodes (or disk space I suspect).

ewildgoose

OK, it's just got itself back into that state again:

Login via SSH

#: touch /tmp/abcd

/: create/symlink failed, no inodes free
touch: /tmp/abcd: No space left on device

#: df -ih
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/ad0s1a 1.9G 206M 1.6G 11% 6.1k 277k 2% /
devfs 1.0K 1.0K 0B 100% 0 0 100% /dev
/dev/md0 3.6M 38K 3.3M 1% 24 742 3% /var/run

Web interface says:

Warning: fopen(/tmp/config.lock): failed to open stream: Device not configured in /etc/inc/util.inc on line 123 Warning: flock() expects parameter 1 to be resource, null given in /etc/inc/util.inc on line 134 Warning: fclose(): supplied argument is not a valid stream resource in /etc/inc/util.inc on line 135 Warning: session_start(): open(/var/tmp//sess_26b30bec4d389d7fca66d4afd1b97385, O_RDWR) failed: No space left on device (28) in /etc/inc/auth.inc on line 1204

Warning: Unknown: open(/var/tmp//sess_26b30bec4d389d7fca66d4afd1b97385, O_RDWR) failed: No space left on device (28) in Unknown on line 0 Warning: Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct () in Unknown on line 0

I think this is either corruption of the filesystem or some other wierdness? Odd that it seems to only appear after a day or so and then clears after a reboot

Any ideas?

ewildgoose

Ahh, further wierdness I forgot to mention. Can't reboot from the SSH prompt, the reboot binary gives an error when you try and access it… In fact so do quite a lot of other binaries (remember this clears after I reboot)

#: ls -al /usr/sbin/
ls: ancontrol: Device not configured
ls: boot0cfg: Device not configured
ls: bsnmpd: Device not configured
ls: chown: Device not configured
ls: chroot: Device not configured
ls: clear_locks: Device not configured
ls: config_lock.sh: Device not configured
ls: config_unlock.sh: Device not configured
ls: cpucontrol: Device not configured
ls: crashinfo: Device not configured
ls: daemon: Device not configured
ls: devinfo: Device not configured
ls: digictl: Device not configured
ls: diskinfo: Device not configured
ls: dtrace: Device not configured
ls: dumpcis: Device not configured
ls: faithd: Device not configured
ls: fifolog_create: Device not configured
ls: fifolog_reader: Device not configured
ls: fifolog_writer: Device not configured
ls: freebsd-update: Device not configured
ls: ftp-proxy: Device not configured
ls: gensnmptree: Device not configured
ls: getextattr: Device not configured
ls: hostapd: Device not configured
ls: hostapd_cli: Device not configured
ls: i2c: Device not configured
ls: iostat: Device not configured
ls: ip6addrctl: Device not configured
ls: ipfwpcap: Device not configured
ls: jail: Device not configured
ls: jexec: Device not configured
ls: jls: Device not configured
ls: kbdcontrol: Device not configured
ls: kbdmap: Device not configured
ls: kldxref: Device not configured
ls: lmcconfig: Device not configured
ls: lockstat: Device not configured
ls: makefs: Device not configured
ls: mfiutil: Device not configured
ls: mountd: Device not configured
ls: mptutil: Device not configured
ls: mtree: Device not configured
ls: ndis_events: Device not configured
ls: ndiscvt: Device not configured
ls: ndisgen: Device not configured
ls: nfscbd: Device not configured
ls: nfsd: Device not configured
ls: nfsdumpstate: Device not configured
ls: nfsrevoke: Device not configured
ls: nfsuserd: Device not configured
ls: ngctl: Device not configured
ls: nghook: Device not configured
ls: nologin: Device not configured
ls: ntp-keygen: Device not configured
ls: ntpd: Device not configured
ls: ntpdc: Device not configured
ls: ntptime: Device not configured
ls: ntptrace: Device not configured
ls: pciconf: Device not configured
ls: pkg_add: Device not configured
ls: pkg_create: Device not configured
ls: pkg_delete: Device not configured
ls: pkg_info: Device not configured
ls: pkg_updating: Device not configured
ls: pkg_version: Device not configured
ls: pmcannotate: Device not configured
ls: ppp: Device not configured
ls: pppctl: Device not configured
ls: rpcbind: Device not configured
ls: rtadvd: Device not configured
ls: sade: Device not configured
ls: service: Device not configured
ls: services_mkdb: Device not configured
ls: setfib: Device not configured
ls: snapinfo: Device not configured
ls: traceroute: Device not configured
ls: traceroute6: Device not configured
ls: uathload: Device not configured
ls: uhsoctl: Device not configured
ls: vidcontrol: Device not configured
ls: vidfont: Device not configured
ls: vipw: Device not configured
ls: wake: Device not configured
ls: watchdog: Device not configured
ls: watchdogd: Device not configured
ls: wlandebug: Device not configured
ls: wpa_cli: Device not configured
ls: wpa_passphrase: Device not configured
ls: wpa_supplicant: Device not configured
ls: zdb: Device not configured
total 1344
drwxr-xr-x 2 root wheel 2048 Dec 2 09:03 .
drwxr-xr-x 12 root wheel 512 Dec 1 18:25 ..
-r-xr-xr-x 1 root wheel 16744 Dec 1 07:53 arp
-rwxr-xr-x 1 root wheel 11998 Dec 1 09:05 clog
-r-xr-xr-x 1 root wheel 39656 Dec 1 07:53 cron
-r-xr-xr-x 1 root wheel 47920 Dec 1 07:53 inetd
-r-xr-xr-x 1 root wheel 57720 Dec 1 07:53 ntpdate
-r-xr-xr-x 1 root wheel 14288 Dec 1 07:53 powerd
-r-xr-xr-x 2 root wheel 13808 Dec 1 07:53 pstat
-r-xr-xr-x 1 root wheel 63312 Dec 1 07:53 pw
-r-xr-xr-x 1 root wheel 19184 Dec 1 07:53 pwd_mkdb
-r-xr-xr-x 1 root wheel 236416 Dec 1 07:53 sshd
-r-xr-xr-x 2 root wheel 13808 Dec 1 07:53 swapinfo
-rwxr-xr-x 1 root wheel 40744 Dec 1 09:05 syslogd
-r-xr-xr-x 1 root wheel 709192 Dec 1 07:53 tcpdump
-r-xr-xr-x 1 root wheel 24544 Dec 1 07:53 usbconfig

wallabybob

@ewildgoose:

Hi, using a snapshot around early Dec 2010 ish (2.0 beta4), I have tried (and failed) to login to the web interface.

That is now a fairly old build. I suggest you upgrade to a much more recent snapshot.

Speculation: You have run into a problem where the kernel has exhausted one or more resources. Allocation of some critical resource (for example, a chunk of heap memory) fails and that allocation failure gets reported up the line as "out of inodes" because the allocation failure occurs in some file system related code and the code authors didn't take care to distinguish the cases "out of inodes" and "can't allocate heap memory for inode processing"

Which build are you using: i386 or amd64? If i386, can you use amd64?

sleeprae

While I don't want to complicate matters with an issue that has similar symptoms but perhaps completely unrelated causes, I thought I would share my recent experience as well. In my case, after running for 2 or 3 days, I would get similar messages–no space left on device, etc. In my case, I couldn't even drop into a shell on the console, but a reboot would clear it up. After a couple occurrences, I left an SSH session open. The next time it happened, I looked at dmesg output, and saw the following:

unknown: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=2257839
ata2: timeout waiting to issue command
ata2: error issuing WRITE_DMA command
g_vfs_done():ad4s1a[WRITE(offset=578043904, length=16384)]error = 6
g_vfs_done():ad4s1a[WRITE(offset=963330048, length=16384)]error = 6
g_vfs_done():ad4s1a[WRITE(offset=1155956736, length=16384)]error = 6
g_vfs_done():ad4s1a[WRITE(offset=1541521408, length=16384)]error = 6
g_vfs_done():ad4s1a[WRITE(offset=1541947392, length=16384)]error = 6
g_vfs_done():ad4s1a[WRITE(offset=1541980160, length=16384)]error = 6
g_vfs_done():ad4s1a[WRITE(offset=1733967872, length=16384)]error = 6
g_vfs_done():ad4s1a[WRITE(offset=1733984256, length=16384)]error = 6
g_vfs_done():ad4s1a[WRITE(offset=1734000640, length=16384)]error = 6
g_vfs_done():ad4s1a[WRITE(offset=1734098944, length=16384)]error = 6
g_vfs_done():ad4s1a[WRITE(offset=1734115328, length=16384)]error = 6
g_vfs_done():ad4s1a[WRITE(offset=2311847936, length=16384)]error = 6
g_vfs_done():ad4s1a[WRITE(offset=2311864320, length=16384)]error = 6
g_vfs_done():ad4s1a[WRITE(offset=2311880704, length=16384)]error = 6
g_vfs_done():ad4s1a[WRITE(offset=2311946240, length=16384)]error = 6
g_vfs_done():ad4s1a[WRITE(offset=2697117696, length=16384)]error = 6
g_vfs_done():ad4s1a[WRITE(offset=2889760768, length=16384)]error = 6
g_vfs_done():ad4s1a[WRITE(offset=2889777152, length=16384)]error = 6
g_vfs_done():ad4s1a[WRITE(offset=1155973120, length=16384)]error = 5
Device ad4s1a went missing before all of the data could be written to it; expect data loss.
pid 26192 (php), uid 0 inumber 141312 on /: out of inodes
pid 38718 (php), uid 0 inumber 141312 on /: out of inodes
pid 17290 (php), uid 0 inumber 141312 on /: out of inodes
pid 499 (php), uid 0 inumber 141312 on /: out of inodes
pid 43275 (sh), uid 0 inumber 353389 on /: out of inodes
pid 7655 (php), uid 0 inumber 141312 on /: out of inodes
pid 55110 (php), uid 0 inumber 141312 on /: out of inodes
pid 56772 (php), uid 0 inumber 141312 on /: out of inodes
pid 60221 (php), uid 0 inumber 141312 on /: out of inodes
pid 63456 (php), uid 0 inumber 141312 on /: out of inodes
pid 11435 (php), uid 0 inumber 141312 on /: out of inodes
pid 24754 (php), uid 0 inumber 141312 on /: out of inodes
pid 27519 (php), uid 0 inumber 141312 on /: out of inodes
pid 42681 (php), uid 0 inumber 141312 on /: out of inodes
pid 17285 (php), uid 0 inumber 141312 on /: out of inodes
pid 18570 (php), uid 0 inumber 141312 on /: out of inodes

In my case, pfSense was installed on an older nForce3 Ultra board and a new Kingston 8GB V100 series SSD. I was running one of the late December -BETA4 (x64) builds initially, and upgrading to newer -BETA5 builds didn't seem to help. (latest installed: 2.0-BETA5 (amd64) built on Tue Dec 28 03:03:03 EST 2010). Four days ago, I cloned the SSD to a generic magnetic HDD, and have had no problem since. I chalked it up to some incompatibility between the older motherboard and the new SSD, but it's similar enough to your issue that I thought I would at least mention it.

wallabybob

sleeprae: If the system hard drive suddenly goes walkabout all sorts of wierdness will happen. Its not hard to imagine that "the hard drive has gone" might translate into "out of inodes". Thanks for reporting that, I think its similar enough to the orgininally reported problem to be interesting.

wildgoose: are you running off a solid state drive (e.g. a flash card or ssd)?

Cry Havok

ewildgoose, I think you've got hardware problems there.

ewildgoose

Hmm, checking dmesg further back I see the same problem. Snippet from the log files:

em1: link state changed to UP
g_vfs_done():ad0s1a[WRITE(offset=1926676480, length=16384)]error = 6
ug_vfs_done():ad0s1a[WRITE(offset=1926692864, length=16384)]error = 6
ng_vfs_done():ad0s1a[WRITE(offset=k770686976, length=16384)]error = 6
g_vfs_done():ad0s1a[WRITE(offset=1163100160, length=16384)]error = 6
ng_vfs_done():ad0s1a[WRITE(offset=1163771904, length=16384)]error = 6
owg_vfs_done():ad0s1a[WRITE(offset=1166508032, length=16384)]error = 6
n:g_vfs_done():ad0s1a[WRITE(offset=1166704640, length=16384)]error = 6
g_vfs_done():ad0s1a[WRITE(offset=1166901248, length=16384)]error = 6
g_vfs_done():ad0s1a[WRITE(offset=1167097856, length=16384)]error = 6
Tg_vfs_done():ad0s1a[WRITE(offset=1167294464, length=16384)]error = 6
Ig_vfs_done():ad0s1a[WRITE(offset=1167491072, length=16384)]error = 6
MEg_vfs_done():ad0s1a[WRITE(offset=1171144704, length=16384)]error = 6
g_vfs_done():ad0s1a[WRITE(offset=1171341312, length=16384)]error = 6
Og_vfs_done():ad0s1a[WRITE(offset=1168130048, length=16384)]error = 6
Ug_vfs_done():ad0s1a[WRITE(offset=1167687680, length=32768)]error = 6
T g_vfs_done():ad0s1a[WRITE(offset=1167998976, length=32768)]error = 6

g_vfs_done():ad0s1a[WRITE(offset=1168310272, length=32768)]error = 6
Wg_vfs_done():ad0s1a[WRITE(offset=1168637952, length=16384)]error = 6
Rg_vfs_done():ad0s1a[WRITE(offset=1168867328, length=16384)]error = 6
Ig_vfs_done():ad0s1a[WRITE(offset=1169113088, length=16384)]error = 6
g_vfs_done():ad0s1a[WRITE(offset=777910272, length=2048)]error = 6
Tg_vfs_done():ad0s1a[WRITE(offset=777889792, length=6144)]error = 6
g_vfs_done():ad0s1a[WRITE(offset=6144000, length=2048)]error = 6
Eg_vfs_done():ad0s1a[WRITE(offset=65536, length=2048)]error = 6
_g_vfs_done():ad0s1a[READ(offset=1164279808, length=4096)]error = 6
Dvnode_pager_getpages: I/O read error
vm_fault: pager read error, pid 29556 (rrdtool)
Mg_vfs_done():ad0s1a[WRITE(offset=578158592, length=16384)]error = 6
A
pid 29556 (rrdtool), uid 0: exited on signal 11
retrying (1 retry left) LBA=3762959
ata0: timeout waiting to issue command
ata0: error issuing WRITE_DMA command
g_vfs_done():ad0s1a[WRITE(offset=1926594560, length=16384)]error = 5
Device ad0s1a went missing before all of the data could be written to it; expect data loss.
pid 24822 (php), uid 0 inumber 94249 on /: out of inodes
vnode_pager_getpages: I/O read error
vm_fault: pager read error, pid 46772 (rrdtool)
pid 46772 (rrdtool), uid 0 inumber 70656 on /: out of inodes
pid 46772 (rrdtool), uid 0: exited on signal 11
arp: 192.168.105.56 moved from 58:b0:35:78:0d:f5 to 00:24:36:9e:fe:13 on em1
pid 51725 (php), uid 0 inumber 94249 on /: out of inodes
vnode_pager_getpages: I/O read error
vm_fault: pager read error, pid 35067 (rrdtool)
pid 35067 (rrdtool), uid 0 inumber 70656 on /: out of inodes
pid 35067 (rrdtool), uid 0: exited on signal 11
pid 58977 (php), uid 0 inumber 94249 on /: out of inodes
vnode_pager_getpages: I/O read error

Yes, I'm using the CF card slot on this Lanner board. I'm using a more expensive SLC card though, it's a brand new card and no reason to think it should have gone bad. Seems more likely that there might be a driver issue with the controller?

atapci0: <intel ich8m="" udma100="" controller="">port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0
ata0: <ata 0="" channel="">on atapci0
ata0: [ITHREAD]

Hmm..</ata></intel>

Cry Havok

It isn't a driver issue, or many other people would have reported problems. Just because your card is new doesn't stop it going bad - hardware can fail at any time.

Try another card. If there are no problems then you know the problem was with that card. If there are then it may be a problem with your motherboard.

cmb

"out of inodes" can indeed be reported on a failing disk/CF, I've seen that on occasion.

jimp

Between the out of inode error and the other errors you are seeing, all signs point toward failing media.

wallabybob

Here are a couple of tests you could do on your "hard drive". These examples assume you hard drive is /dev/ad1 (change name as appropriate for your configuration).

Read the whole hard drive (copy the whole drive to /dev/null, the "null" device):

# dd bs=65536 if=/dev/ad1 of=/dev/null

write zeroes to free space, then free up the space filled with zeroes:

# dd bs=65536 if=/dev/zero of=/tmp/zero; rm /tmp/zero

If the drive is good neither of these tests should produce any error report relating to the drive.

I believe some types of solid state "disks" do some sort of wear levelling which could involve the drive being "busy" for a while. I have no idea what appropriate standards say about this in relation to how long a drive might be allowed to "lock out" i/o requests while it is busy with its housekeeping. Its possible the FreeBSD disk driver might need some tweaking to accommodate some types of solid state media.

I have two pfSense boxes using Transcend DOM 1GB solid state disk modules. I've not seen this sort of problem on them. But these devices are intended for high i/o rate and sustained i/o environments. I suspect commodity type memory cards are not intended for high i/o rate and sustained i/o environments and consequently the designers might have taken some shortcuts.

Tidder

@wallabybob:

wildgoose: are you running off a solid state drive (e.g. a flash card or ssd)?

I just wanted to mention here that we had 2 routers running a recent build of BETA5 working flawless for a couple of weeks. I have switched from a normal platter-based 2.5" hard drive to a Transend 2.5" solid state 8gb IDE drive (SLC) and have installed RC1 fresh onto both of them. Both of them are now exhibiting this behavior. I do believe this problem could be linked to SSDs. Anything I can do to help/fix? Start a new thread and not hijack this one? ;)