Ran out of inodes



  • Hi, using a snapshot around early Dec 2010 ish (2.0 beta4), I have tried (and failed) to login to the web interface.  Give me an error message:

    Warning: fopen(/tmp/config.lock): failed to open stream: Device not configured in /etc/inc/util.inc on line 123 Warning: flock() expects parameter 1 to be resource, null given in /etc/inc/util.inc on line 134 Warning: fclose(): supplied argument is not a valid stream resource in /etc/inc/util.inc on line 135 Warning: session_start(): open(/var/tmp//sess_c9cf1581880d10889fcf3ddcb324b7ee, O_RDWR) failed: No space left on device (28) in /etc/inc/auth.inc on line 1204

    Logging in via ssh, I see:
    : df -hi
    Filesystem    Size    Used  Avail Capacity iused ifree %iused  Mounted on
    /dev/ad0s1a    1.9G    208M    1.6G    11%    6.2k  276k    2%  /
    devfs          1.0K    1.0K      0B  100%      0    0  100%  /dev
    /dev/md0      3.6M    38K    3.3M    1%      24  742    3%  /var/run

    So I presume that something is creating and deleting files, but keeping the file open? How else can I be out of inodes?

    Any suggestions on things to check quickly before rebooting (which I presume will fix the issue?).  Is there an equiv of "lsof" available? (Sorry, I'm a linux user normally)

    Note:

    : ps x
      PID  TT  STAT      TIME COMMAND
        0  ??  DLs  98:47.32 [kernel]
        1  ??  ILs    0:00.43 /sbin/init –
        2  ??  DL    2:08.30 [g_event]
        3  ??  DL    1:02.35 [g_up]
        4  ??  DL    0:29.10 [g_down]
        5  ??  DL    0:00.00 [crypto]
        6  ??  DL    0:00.00 [crypto returns]
        7  ??  DL    1:15.11 [pfpurge]
        8  ??  DL    0:00.00 [xpt_thrd]
        9  ??  DL    0:08.30 [pagedaemon]
      10  ??  RL  155078:06.10 [idle]
      11  ??  WL  153:00.53 [intr]
      12  ??  DL    0:04.47 [ng_queue]
      13  ??  DL    12:23.39 [yarrow]
      14  ??  DL    0:45.90 [usb]
      15  ??  DL    0:00.00 [vmdaemon]
      16  ??  DL    0:04.93 [idlepoll]
      17  ??  DL    0:00.04 [pagezero]
      18  ??  DL    0:15.11 [bufdaemon]
      19  ??  DL    2:35.25 [syncer]
      20  ??  DL    0:16.11 [vnlru]
      21  ??  DL    0:18.18 [softdepflush]
      32  ??  DL    0:02.23 [md0]
      236  ??  INs    0:00.14 /usr/local/sbin/check_reload_status
      241  ??  IN    0:00.00 check_reload_status: Monitoring daemon of check_reload_status
      249  ??  Is    0:00.00 /sbin/devd
    4375  ??  Ss    0:03.11 ntpd: [priv] (ntpd)
    5394  ??  IN    29:24.89 /bin/sh /var/db/rrd/updaterrd.sh
    7260  ??  INs    0:00.01 /usr/sbin/sshd
    7591  ??  Ss    1:10.11 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_wan.conf -p /var/run/pppoe_wan.pid -s ppp pppoe
    8082  ??  Ss    8:32.12 /usr/sbin/syslogd -c -l /var/dhcpd/var/run/log -f /var/etc/syslog.conf
    11841  ??  I      0:13.55 /usr/local/bin/php
    13505  ??  Is    0:22.24 /usr/sbin/inetd -wW -R 0 -a 127.0.0.1 /var/etc/inetd.conf
    15076  ??  Is    0:00.02 /usr/local/sbin/sshlockout_pf
    16761  ??  SN    1:17.49 /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf
    22087  ??  Is    0:00.12 /usr/local/bin/php
    22713  ??  Is    0:00.18 /usr/local/bin/php
    26611  ??  SNs    0:00.17 sshd: root@pts/0 (sshd)
    29425  ??  I      0:12.55 /usr/local/bin/php
    33502  ??  Ss    0:25.24 openvpn –config /var/etc/openvpn/server1.conf
    34120  ??  Is    0:04.40 dhclient: em0 [priv] (dhclient)
    34456  ??  Is    0:23.56 /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf
    34508  ??  SNs    4:27.36 /usr/local/sbin/apinger -c /var/etc/apinger.conf
    34603  ??  IN    0:06.89 /usr/local/bin/rrdtool -
    36488  ??  IN    0:00.01 sleep 60
    36564  ??  IN    0:00.01 /usr/bin/top -d 2 -s 1 0
    36775  ??  IN    0:00.00 [awk]
    37446  ??  I      0:10.84 /usr/local/bin/php
    45148  ??  Is    0:03.41 /usr/sbin/cron -s
    46805  ??  I      0:04.38 /usr/local/bin/php
    58168  ??  Is    0:02.51 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh
    58746  ??  Is    0:00.16 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /etc/rc.expireaccounts
    58951  ??  Is    0:00.01 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /etc/rc.update_alias_url_dat
    9358  u0- S      3:26.01 /usr/sbin/tcpdump -s 256 -v -l -n -e -ttt -i pflog0
    9487  u0- S      2:54.79 logger -t pf -p local0.info
    62539  u0  Is    0:00.02 login [pam] (login)
    62604  u0  I      0:00.01 -sh (sh)
    63719  u0  I+    0:00.01 /bin/sh /etc/rc.initial
    19916  0  S      0:00.11 /bin/tcsh
    26713  0  Is    0:00.01 -sh (sh)
    27681  0  I      0:00.02 /bin/sh /etc/rc.initial
    36930  0  R+    0:00.00 ps x

    Thanks



  • This is a common problem when you have many files of a small size
    Rebooting will not fix this.

    try delete huge log file after you will be able to access webgui.



  • Thanks for the thoughts, but your advice seems contradictory?

    Firstly it starts working again after I reboot

    Secondly you say it's a problem of lots of small files (why?), but then your suggestion is to delete a single (huge) file to recover from that situation?  Don't understand how both of these can be correct?

    Finally, looking at the df output we can clearly see loads of free inodes?  That seems inconsistent with your theory?

    Here is the inode count after rebooting:

    : df -ih
    Filesystem    Size    Used  Avail Capacity iused ifree %iused  Mounted on
    /dev/ad0s1a    1.9G    206M    1.6G    11%    6.1k  277k    2%  /
    devfs          1.0K    1.0K      0B  100%      0    0  100%  /dev
    /dev/md0      3.6M    38K    3.3M    1%      24  742    3%  /var/run

    So, I think my question remains?  Why am I running out of inodes?  I suspect either disk corruption or a huge amount of temp files created which are deleted but not closed?

    Thanks for any thoughts?



  • why do you think it is out of inodes? i don't see that in text you quoted?



  • You're not out of inodes (devfs isn't a file system, it will always show that), must have a package on there that's dropping a lot of data.



  • Aha, apparently I slightly skipped a bit of debug.  Firstly notice the main error I mention about the failure to get into the web interface, this is a symptom of being unable to create a file in /tmp due to being out of inodes

    Additionally ssh-ing in and a simple "touch /tmp/abcd" would give an error that I don't have to hand, but something like "cannot create file, out of inodes"

    Additionally the log files are stuffed full of errors along the lines of "cannot create file xxxx"

    Before rebooting I killed almost every process that I could and despite that I still couldn't create any new files on my root filesystem.  After a reboot all was fine…

    Any thoughts?



  • How about answering my question as to why you believe you are out of inodes?



  • @ewildgoose:

    : df -ih
    Filesystem    Size    Used  Avail Capacity iused ifree %iused  Mounted on
    /dev/ad0s1a    1.9G    208M    1.6G    11%    6.2k  276k    2%  /
    devfs          1.0K    1.0K      0B  100%      0    0  100%  /dev
    /dev/md0      3.6M    38K    3.3M    1%      24  742    3%  /var/run

    I'm with danswartz, there's nothing there to show you're running out of inodes (the entry for devfs is normal - that's not a real file system).  Whatever your problem is, it is not related to running out of inodes (or disk space I suspect).



  • OK, it's just got itself back into that state again:

    Login via SSH

    #: touch /tmp/abcd

    /: create/symlink failed, no inodes free
    touch: /tmp/abcd: No space left on device

    #: df -ih
    Filesystem    Size    Used  Avail Capacity iused ifree %iused  Mounted on
    /dev/ad0s1a    1.9G    206M    1.6G    11%    6.1k  277k    2%  /
    devfs          1.0K    1.0K      0B  100%      0    0  100%  /dev
    /dev/md0      3.6M    38K    3.3M    1%      24  742    3%  /var/run

    Web interface says:

    Warning: fopen(/tmp/config.lock): failed to open stream: Device not configured in /etc/inc/util.inc on line 123 Warning: flock() expects parameter 1 to be resource, null given in /etc/inc/util.inc on line 134 Warning: fclose(): supplied argument is not a valid stream resource in /etc/inc/util.inc on line 135 Warning: session_start(): open(/var/tmp//sess_26b30bec4d389d7fca66d4afd1b97385, O_RDWR) failed: No space left on device (28) in /etc/inc/auth.inc on line 1204

    Warning: Unknown: open(/var/tmp//sess_26b30bec4d389d7fca66d4afd1b97385, O_RDWR) failed: No space left on device (28) in Unknown on line 0 Warning: Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct () in Unknown on line 0

    I think this is either corruption of the filesystem or some other wierdness?  Odd that it seems to only appear after a day or so and then clears after a reboot

    Any ideas?



  • Ahh, further wierdness I forgot to mention.  Can't reboot from the SSH prompt, the reboot binary gives an error when you try and access it… In fact so do quite a lot of other binaries (remember this clears after I reboot)

    #: ls -al /usr/sbin/
    ls: ancontrol: Device not configured
    ls: boot0cfg: Device not configured
    ls: bsnmpd: Device not configured
    ls: chown: Device not configured
    ls: chroot: Device not configured
    ls: clear_locks: Device not configured
    ls: config_lock.sh: Device not configured
    ls: config_unlock.sh: Device not configured
    ls: cpucontrol: Device not configured
    ls: crashinfo: Device not configured
    ls: daemon: Device not configured
    ls: devinfo: Device not configured
    ls: digictl: Device not configured
    ls: diskinfo: Device not configured
    ls: dtrace: Device not configured
    ls: dumpcis: Device not configured
    ls: faithd: Device not configured
    ls: fifolog_create: Device not configured
    ls: fifolog_reader: Device not configured
    ls: fifolog_writer: Device not configured
    ls: freebsd-update: Device not configured
    ls: ftp-proxy: Device not configured
    ls: gensnmptree: Device not configured
    ls: getextattr: Device not configured
    ls: hostapd: Device not configured
    ls: hostapd_cli: Device not configured
    ls: i2c: Device not configured
    ls: iostat: Device not configured
    ls: ip6addrctl: Device not configured
    ls: ipfwpcap: Device not configured
    ls: jail: Device not configured
    ls: jexec: Device not configured
    ls: jls: Device not configured
    ls: kbdcontrol: Device not configured
    ls: kbdmap: Device not configured
    ls: kldxref: Device not configured
    ls: lmcconfig: Device not configured
    ls: lockstat: Device not configured
    ls: makefs: Device not configured
    ls: mfiutil: Device not configured
    ls: mountd: Device not configured
    ls: mptutil: Device not configured
    ls: mtree: Device not configured
    ls: ndis_events: Device not configured
    ls: ndiscvt: Device not configured
    ls: ndisgen: Device not configured
    ls: nfscbd: Device not configured
    ls: nfsd: Device not configured
    ls: nfsdumpstate: Device not configured
    ls: nfsrevoke: Device not configured
    ls: nfsuserd: Device not configured
    ls: ngctl: Device not configured
    ls: nghook: Device not configured
    ls: nologin: Device not configured
    ls: ntp-keygen: Device not configured
    ls: ntpd: Device not configured
    ls: ntpdc: Device not configured
    ls: ntptime: Device not configured
    ls: ntptrace: Device not configured
    ls: pciconf: Device not configured
    ls: pkg_add: Device not configured
    ls: pkg_create: Device not configured
    ls: pkg_delete: Device not configured
    ls: pkg_info: Device not configured
    ls: pkg_updating: Device not configured
    ls: pkg_version: Device not configured
    ls: pmcannotate: Device not configured
    ls: ppp: Device not configured
    ls: pppctl: Device not configured
    ls: rpcbind: Device not configured
    ls: rtadvd: Device not configured
    ls: sade: Device not configured
    ls: service: Device not configured
    ls: services_mkdb: Device not configured
    ls: setfib: Device not configured
    ls: snapinfo: Device not configured
    ls: traceroute: Device not configured
    ls: traceroute6: Device not configured
    ls: uathload: Device not configured
    ls: uhsoctl: Device not configured
    ls: vidcontrol: Device not configured
    ls: vidfont: Device not configured
    ls: vipw: Device not configured
    ls: wake: Device not configured
    ls: watchdog: Device not configured
    ls: watchdogd: Device not configured
    ls: wlandebug: Device not configured
    ls: wpa_cli: Device not configured
    ls: wpa_passphrase: Device not configured
    ls: wpa_supplicant: Device not configured
    ls: zdb: Device not configured
    total 1344
    drwxr-xr-x  2 root  wheel    2048 Dec  2 09:03 .
    drwxr-xr-x  12 root  wheel    512 Dec  1 18:25 ..
    -r-xr-xr-x  1 root  wheel  16744 Dec  1 07:53 arp
    -rwxr-xr-x  1 root  wheel  11998 Dec  1 09:05 clog
    -r-xr-xr-x  1 root  wheel  39656 Dec  1 07:53 cron
    -r-xr-xr-x  1 root  wheel  47920 Dec  1 07:53 inetd
    -r-xr-xr-x  1 root  wheel  57720 Dec  1 07:53 ntpdate
    -r-xr-xr-x  1 root  wheel  14288 Dec  1 07:53 powerd
    -r-xr-xr-x  2 root  wheel  13808 Dec  1 07:53 pstat
    -r-xr-xr-x  1 root  wheel  63312 Dec  1 07:53 pw
    -r-xr-xr-x  1 root  wheel  19184 Dec  1 07:53 pwd_mkdb
    -r-xr-xr-x  1 root  wheel  236416 Dec  1 07:53 sshd
    -r-xr-xr-x  2 root  wheel  13808 Dec  1 07:53 swapinfo
    -rwxr-xr-x  1 root  wheel  40744 Dec  1 09:05 syslogd
    -r-xr-xr-x  1 root  wheel  709192 Dec  1 07:53 tcpdump
    -r-xr-xr-x  1 root  wheel  24544 Dec  1 07:53 usbconfig



  • @ewildgoose:

    Hi, using a snapshot around early Dec 2010 ish (2.0 beta4), I have tried (and failed) to login to the web interface.

    That is now a fairly old build. I suggest you upgrade to a much more recent snapshot.

    Speculation: You have run into a problem where the kernel has exhausted one or more resources. Allocation of some critical resource (for example, a chunk of heap memory) fails and that allocation failure gets reported up the line as "out of inodes" because the allocation failure occurs in some file system related code and the code authors didn't take care to distinguish the cases "out of inodes" and "can't allocate heap memory for inode processing"

    Which build are you using: i386 or amd64? If i386, can you use amd64?



  • While I don't want to complicate matters with an issue that has similar symptoms but perhaps completely unrelated causes, I thought I would share my recent experience as well. In my case, after running for 2 or 3 days, I would get similar messages–no space left on device, etc. In my case, I couldn't even drop into a shell on the console, but a reboot would clear it up. After a couple occurrences, I left an SSH session open. The next time it happened, I looked at dmesg output, and saw the following:

    unknown: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=2257839
    ata2: timeout waiting to issue command
    ata2: error issuing WRITE_DMA command
    g_vfs_done():ad4s1a[WRITE(offset=578043904, length=16384)]error = 6
    g_vfs_done():ad4s1a[WRITE(offset=963330048, length=16384)]error = 6
    g_vfs_done():ad4s1a[WRITE(offset=1155956736, length=16384)]error = 6
    g_vfs_done():ad4s1a[WRITE(offset=1541521408, length=16384)]error = 6
    g_vfs_done():ad4s1a[WRITE(offset=1541947392, length=16384)]error = 6
    g_vfs_done():ad4s1a[WRITE(offset=1541980160, length=16384)]error = 6
    g_vfs_done():ad4s1a[WRITE(offset=1733967872, length=16384)]error = 6
    g_vfs_done():ad4s1a[WRITE(offset=1733984256, length=16384)]error = 6
    g_vfs_done():ad4s1a[WRITE(offset=1734000640, length=16384)]error = 6
    g_vfs_done():ad4s1a[WRITE(offset=1734098944, length=16384)]error = 6
    g_vfs_done():ad4s1a[WRITE(offset=1734115328, length=16384)]error = 6
    g_vfs_done():ad4s1a[WRITE(offset=2311847936, length=16384)]error = 6
    g_vfs_done():ad4s1a[WRITE(offset=2311864320, length=16384)]error = 6
    g_vfs_done():ad4s1a[WRITE(offset=2311880704, length=16384)]error = 6
    g_vfs_done():ad4s1a[WRITE(offset=2311946240, length=16384)]error = 6
    g_vfs_done():ad4s1a[WRITE(offset=2697117696, length=16384)]error = 6
    g_vfs_done():ad4s1a[WRITE(offset=2889760768, length=16384)]error = 6
    g_vfs_done():ad4s1a[WRITE(offset=2889777152, length=16384)]error = 6
    g_vfs_done():ad4s1a[WRITE(offset=1155973120, length=16384)]error = 5
    Device ad4s1a went missing before all of the data could be written to it; expect data loss.
    pid 26192 (php), uid 0 inumber 141312 on /: out of inodes
    pid 38718 (php), uid 0 inumber 141312 on /: out of inodes
    pid 17290 (php), uid 0 inumber 141312 on /: out of inodes
    pid 499 (php), uid 0 inumber 141312 on /: out of inodes
    pid 43275 (sh), uid 0 inumber 353389 on /: out of inodes
    pid 7655 (php), uid 0 inumber 141312 on /: out of inodes
    pid 55110 (php), uid 0 inumber 141312 on /: out of inodes
    pid 56772 (php), uid 0 inumber 141312 on /: out of inodes
    pid 60221 (php), uid 0 inumber 141312 on /: out of inodes
    pid 63456 (php), uid 0 inumber 141312 on /: out of inodes
    pid 11435 (php), uid 0 inumber 141312 on /: out of inodes
    pid 24754 (php), uid 0 inumber 141312 on /: out of inodes
    pid 27519 (php), uid 0 inumber 141312 on /: out of inodes
    pid 42681 (php), uid 0 inumber 141312 on /: out of inodes
    pid 17285 (php), uid 0 inumber 141312 on /: out of inodes
    pid 18570 (php), uid 0 inumber 141312 on /: out of inodes
    

    In my case, pfSense was installed on an older nForce3 Ultra board and a new Kingston 8GB V100 series SSD. I was running one of the late December -BETA4 (x64) builds initially, and upgrading to newer -BETA5 builds didn't seem to help. (latest installed: 2.0-BETA5 (amd64) built on Tue Dec 28 03:03:03 EST 2010). Four days ago, I cloned the SSD to a generic magnetic HDD, and have had no problem since. I chalked it up to some incompatibility between the older motherboard and the new SSD, but it's similar enough to your issue that I thought I would at least mention it.



  • sleeprae: If the system hard drive suddenly goes walkabout all sorts of wierdness will happen. Its not hard to imagine that "the hard drive has gone" might translate into "out of inodes". Thanks for reporting that, I think its similar enough to the orgininally reported problem to be interesting.

    wildgoose: are you running off a solid state drive (e.g. a flash card or ssd)?



  • ewildgoose, I think you've got hardware problems there.



  • Hmm, checking dmesg further back I see the same problem.  Snippet from the log files:

    em1: link state changed to UP
    g_vfs_done():ad0s1a[WRITE(offset=1926676480, length=16384)]error = 6
    ug_vfs_done():ad0s1a[WRITE(offset=1926692864, length=16384)]error = 6
    ng_vfs_done():ad0s1a[WRITE(offset=k770686976, length=16384)]error = 6
    g_vfs_done():ad0s1a[WRITE(offset=1163100160, length=16384)]error = 6
    ng_vfs_done():ad0s1a[WRITE(offset=1163771904, length=16384)]error = 6
    owg_vfs_done():ad0s1a[WRITE(offset=1166508032, length=16384)]error = 6
    n:g_vfs_done():ad0s1a[WRITE(offset=1166704640, length=16384)]error = 6
    g_vfs_done():ad0s1a[WRITE(offset=1166901248, length=16384)]error = 6
    g_vfs_done():ad0s1a[WRITE(offset=1167097856, length=16384)]error = 6
    Tg_vfs_done():ad0s1a[WRITE(offset=1167294464, length=16384)]error = 6
    Ig_vfs_done():ad0s1a[WRITE(offset=1167491072, length=16384)]error = 6
    MEg_vfs_done():ad0s1a[WRITE(offset=1171144704, length=16384)]error = 6
    g_vfs_done():ad0s1a[WRITE(offset=1171341312, length=16384)]error = 6
    Og_vfs_done():ad0s1a[WRITE(offset=1168130048, length=16384)]error = 6
    Ug_vfs_done():ad0s1a[WRITE(offset=1167687680, length=32768)]error = 6
    T g_vfs_done():ad0s1a[WRITE(offset=1167998976, length=32768)]error = 6

    • g_vfs_done():ad0s1a[WRITE(offset=1168310272, length=32768)]error = 6
      Wg_vfs_done():ad0s1a[WRITE(offset=1168637952, length=16384)]error = 6
      Rg_vfs_done():ad0s1a[WRITE(offset=1168867328, length=16384)]error = 6
      Ig_vfs_done():ad0s1a[WRITE(offset=1169113088, length=16384)]error = 6
      g_vfs_done():ad0s1a[WRITE(offset=777910272, length=2048)]error = 6
      Tg_vfs_done():ad0s1a[WRITE(offset=777889792, length=6144)]error = 6
      g_vfs_done():ad0s1a[WRITE(offset=6144000, length=2048)]error = 6
      Eg_vfs_done():ad0s1a[WRITE(offset=65536, length=2048)]error = 6
      _g_vfs_done():ad0s1a[READ(offset=1164279808, length=4096)]error = 6
      Dvnode_pager_getpages: I/O read error
      vm_fault: pager read error, pid 29556 (rrdtool)
      Mg_vfs_done():ad0s1a[WRITE(offset=578158592, length=16384)]error = 6
      A
      pid 29556 (rrdtool), uid 0: exited on signal 11
      retrying (1 retry left) LBA=3762959
      ata0: timeout waiting to issue command
      ata0: error issuing WRITE_DMA command
      g_vfs_done():ad0s1a[WRITE(offset=1926594560, length=16384)]error = 5
      Device ad0s1a went missing before all of the data could be written to it; expect data loss.
      pid 24822 (php), uid 0 inumber 94249 on /: out of inodes
      vnode_pager_getpages: I/O read error
      vm_fault: pager read error, pid 46772 (rrdtool)
      pid 46772 (rrdtool), uid 0 inumber 70656 on /: out of inodes
      pid 46772 (rrdtool), uid 0: exited on signal 11
      arp: 192.168.105.56 moved from 58:b0:35:78:0d:f5 to 00:24:36:9e:fe:13 on em1
      pid 51725 (php), uid 0 inumber 94249 on /: out of inodes
      vnode_pager_getpages: I/O read error
      vm_fault: pager read error, pid 35067 (rrdtool)
      pid 35067 (rrdtool), uid 0 inumber 70656 on /: out of inodes
      pid 35067 (rrdtool), uid 0: exited on signal 11
      pid 58977 (php), uid 0 inumber 94249 on /: out of inodes
      vnode_pager_getpages: I/O read error

    Yes, I'm using the CF card slot on this Lanner board.  I'm using a more expensive SLC card though, it's a brand new card and no reason to think it should have gone bad.  Seems more likely that there might be a driver issue with the controller?

    atapci0: <intel ich8m="" udma100="" controller="">port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0
    ata0: <ata 0="" channel="">on atapci0
    ata0: [ITHREAD]

    Hmm..</ata></intel>



  • It isn't a driver issue, or many other people would have reported problems.  Just because your card is new doesn't stop it going bad - hardware can fail at any time.

    Try another card. If there are no problems then you know the problem was with that card. If there are then it may be a problem with your motherboard.



  • "out of inodes" can indeed be reported on a failing disk/CF, I've seen that on occasion.


  • Rebel Alliance Developer Netgate

    Between the out of inode error and the other errors you are seeing, all signs point toward failing media.



  • Here are a couple of tests you could do on your "hard drive". These examples assume you hard drive is /dev/ad1 (change name as appropriate for your configuration).

    Read the whole hard drive (copy the whole drive to /dev/null, the "null" device):

    # dd bs=65536 if=/dev/ad1 of=/dev/null
    

    write zeroes to free space, then free up the space filled with zeroes:

    # dd bs=65536 if=/dev/zero of=/tmp/zero; rm /tmp/zero
    

    If the drive is good neither of these tests should produce any error report relating to the drive.

    I believe some types of solid state "disks" do some sort of wear levelling which could involve the drive being "busy" for a while. I have no idea what appropriate standards say about this in relation to how long a drive might be allowed to "lock out" i/o requests while it is busy with its housekeeping. Its possible the FreeBSD disk driver might need some tweaking to accommodate some types of solid state media.

    I have two pfSense boxes using Transcend DOM 1GB solid state disk modules. I've not seen this sort of problem on them. But these devices are intended for high i/o rate and sustained i/o environments. I suspect commodity type memory cards are not intended for high i/o rate and sustained i/o environments and consequently the designers might have taken some shortcuts.



  • @wallabybob:

    wildgoose: are you running off a solid state drive (e.g. a flash card or ssd)?

    I just wanted to mention here that we had 2 routers running a recent build of BETA5 working flawless for a couple of weeks.  I have switched from a normal platter-based 2.5" hard drive to a Transend 2.5" solid state 8gb IDE drive (SLC) and have installed RC1 fresh onto both of them.  Both of them are now exhibiting this behavior.  I do believe this problem could be linked to SSDs.  Anything I can do to help/fix?  Start a new thread and not hijack this one? ;)


Log in to reply