PfSense melt down overnight
-
For some reason my pfSense box at my central site had a meltdown. When i got in today everything was offline and nothing was working. Phones were offline and weren't reconnecting to the voip server, exchange wasnt passing email, workstations were slow to get on, and no internet. Logged into pfSense and main page came up quick but all the other pages are taking a long time to load. the IPSec status page will not come up, the IPSec config page takes forever to load, any firewall page takes extremely long to pull up, and the interface page shows everything is connected. I rebooted the box and hooked a monitor up to the machine to see what it does and it gets to configuring firewall and freezes. after 5 minutes maybe a little more it finishes configuring the firewall and then moves on and then hangs 5 lines down when it tried to configure IPSec VPN which again takes 5 minutes plus to load and then moves past that and loads and then stops again for several minutes at the Final filter pass. When it does finishes booting the local console shows an external IP and internally I can SSH into the box but NO TRAFFIC is coming in out out. I can't explain it or what happened as everything was working smoothly when I left lastnight at 10pm. It won't let me in to look at the IPSec logs and the firewall log page takes forever to come up.
I am running pfSense v1.2.3RC1 on a dell gx60 box. p4 1.8ghz with 512mb ram and 30 gb harddrive. I have 1.2
.3RC1 loaded on 2 other boxes pretty much identical to this one and those are still online. -
Not saying this is always the case, but typically random weirdness of that nature is indicative of hardware issues.
If you can get to a shell, running top -S might help figure out what is taking so long. Pay particular attention to RAM usage.
I'm not sure if it works during the boot process, but pressing Ctrl-T should print on the console what the current active process is when you're waiting on it.