SG1100 becomes inaccessible (web and ssh). can't regain access. logs not showing much, if anything.

[jaaasshh]

I've posted related posts here and here - but I no longer think it has anything to do with the login interface or my WAN (Verizon came out and replaced all of their hardware and has been stable ever since).

Today, I tried accessing the web UI and the login page loaded. I entered credentials and submitted the form. The Web UI Hung. At no point did the web UI return anything after the initial login form. I refreshed, opened in new tabs, never got anything.

At this point, I tried SSHing into the firewall. SSH didn't respond. Couldn't get in.

So I hooked up a USB cable and logged in to a serial console. tail /var/log/*log showed no movement if I re-requested a page or tried SSH. I ran: /etc/rc.restart_webgui - it seemed to execute successfully, (showed: Restarting webConfigurator... done.) - however the Web UI wasn't accessible. I ps aux|grep nginx and ps aux|grep php and killed everything I could find. Re-ran /etc/rc.restart_webgui Still nothing.

Okay, how about sshd? ps aux|grep sshd showed it was running. Ran /etc/rc.d/sshd onestop. Didn't stop. kill -9 it - that got it, and /etc/rc.d/sshd onestart restarted it, but still couldn't ssh in.

So at that point, frustrated, and needing the internet to work, I ran reboot from the serial console. That killed the serial connection but definitely didn't reboot the firewall. ~5 minutes later I power cycled it and it came back up seemingly working well.

I'm a bit at a loss trying to track down what's going wrong. At no point in this debugging did I see any movement while tail -f /var/log/*log - I can't say I saw any meaningful looking errors either. After the power cycle and bringing it back up i see plenty of old messages in those logs, but nothing within the last few days.

Any suggestions on what to look for next time?