Intermittent system hang



  • Hi guys,

    I have been running pfSense for a long time now and never had this happen before.

    We are running a pfSense appliance (based round an Intel Atom) with multiple LAN connections (via VLANs) and a number of openVPN connections. About 48 hours ago it started hanging - for about 5 to 10 minutes. No traffic through the firewall, can not get at the web interface or SSH and the serial output (it has not monitor) also hangs.

    When it comes back the system does not appear to have crashed - it still says 28 days up time. The log has three apinger alarms (the three gateways on the system) and then three alarm clears - all with the same timestamp. The RRD graphs have nice little blank spots in them where they failed to get any data.

    My thoughts is that something is hogging the CPU for a while but I can not see what or where?

    Firmware is 2.0.1-RELEASE (i386) built on Mon Dec 12 17:53:52 EST 2011 FreeBSD 8.1-RELEASE-p6.

    I am waiting for a response from the supplier as to whether it is easy/possible to update to 2.1.x

    Anyone got any ideas as to how I can work out what is going on?



  • No solutions but I would look at hardware first, replace NICs / switch & cables.
    If nothing changed (config / software) in the last couple days, than it is probably hardware.

    Have you tried a cold reboot (take power out for ~30sec) ?

    Peter



  • Well,

    After 15 hours hunting, it turned out to be a digital asset management system (photo and video management) that had gone odd. It runs in a virtual machine hidden in the depths of our system. I ended up turning connections off at the routing switch in the heart of our network until the firewall started up again, then followed the network until I hit on that machine. One reboot later and a happy network.

    Why is it always a simple solution that takes hours to find :)


Log in to reply