Help?? PFS has gotten into strange nearly unusable state out of the blue (FIXED

ck42

No idea what has happened, but it as always it seems, the problem started as soon as I left the house…

Symptoms:
PFS does not reply to ICMP PINGs, yet can successfully PING other devices on the network
My Outlook client remains connected (outlook server is accessed via the Internet), but I cannot access any web pages...including PFS web interface.

Despite having enabled SSHD, I cannot get in via SSH, but from a prompt in PFS, I can successfully SSH out to another device on the network.

I've got a monitor hooked up and am looking around. Not sure if disk storage is any issue since I've never had to deal with it with PFS, but it's down to 710MB free (total 19GB). If I try to run an upgrade from console, I end up getting the message "write failed, filesystem is full".

I'm pretty stuck at this point and don't know how to proceed next. If at all possible, I'd like to repair the problem and avoid having to reinstall (though I do have current config backups if it comes to that)

cmb

Sounds like the symptoms of a dead hard drive possibly. Shouldn't be anything that would need anywhere near 700 MB free disk space (if upgrading from the console, hence not doing a full backup). Dying disks sometimes exhibit the problem in such a way that the OS returns the "filesystem is full" message. That could also explain all the other issues, as various other things will be failing as well. Already-active connections like your Outlook could be unaffected.

Power cycling the system may bring it back to life for some period of time, or it may completely kill the drive and be unbootable. Regardless, that's what I'd do next, especially since you have a config backup. Just be prepared to replace the drive and reinstall should that become necessary.

ck42

I've cycled power twice now. No change.
I'm currently trying to find the right syntax (for freebsd) to have it search the drive and show me the largest files.

cmb

Maybe it's a different partition that's out of space? Though by default you'd only have one. Double check 'df -h' output.

To find how big the root directories are, run:

du -hd1 /

then check subdirectories from there as necessary.

du -hd1 /usr

ck42

Okay…biggest files on the drive:

1.6GB dhcpleases.core ( in the / directory)
970MB access.log (in my squid directory)
203MB (a pfs full backup .tar.gz file)

Everything else is 13MB and smaller and are very innocuous looking files.

du -hd1 /
14GB are in /usr (surprise surprise)
1.4GB in /var

So those two directories alone seem to hold 90% of my available space.

Can I nuke the dhcpleases.core file safely?

cmb

Yeah you can delete that core file.

ck42

Largest file found in /usr is only 5.7MB….so there's apparently just an absolutely huge number of files in that directory then.
I'll nuke the dhcpleases file and restart. fingers crossed.

ck42

Came back up…interfaces were no longer assigned.
Rebuilt the LAN interface and can PING/SSH out once again....but still cannot PING or SSH into PFS.

Did a config restore from about a week ago.
System still angry.
Rebooted again...system looks like it might be running again! Maybe?

ck42

Looks like I'm back up!!
Also completed a firmware update (was about 2 releases behind).

Would still like to know what happened though so I can avoid it in the future.