Too many open files in system - Possible Culprit

  • A few times over the past year or so running PFSense (2.1-RELEASE, currently), I've run into the problem of squid complaining "Too many open files in system" and the firewall not passing traffic. Restarting the box will make things work again, but after some time (usually months, this is a home set up), it'll do it again.

    After figuring out how to know what has a file handle open, and how many max/current files there are open, I used the mailreport package to send me some fstat and pstat info. Over the past week, this is what I've seen:

    The first two columns are the last two lines from: fstat | awk '{print $2}' | sort | uniq -c | sort -n | tail -n 5
    The third column is from: pstat -T
    (The next lowest entry from fstat is 'sh', and is in the low 200s, fwiw).

    | filterdns | ipfw-classifyd | pstat |
    | 894 | 1786 | 3303/12328 |
    | 990 | 1978 | 3448/12328 |
    | 1086 | 2170 | 3748/12328 |
    | 1182 | 2362 | 4148/12328 |
    | 1278 | 2554 | 4395/12328 |
    | 1374 | 2746 | 4655/12328 |

    As you can see, in a few more weeks, I'll run out of handles and have to restart.

    Unfortunately, this is where my knowledge runs out - I'm not sure why it's not killing these off (or even if it should). I could easily write a script that 'kill's off old processes, but that's probably not the right thing to do.

    I'd appreciate any help in narrowing this down and solving it. I've searched the forums, and seen other instances of this problem, but never any solutions (other than: restart the box).