Panic after upgrading to 2.2.4



  • Did the 2.2.3 -> 2.2.4 upgrade yesterday, and I woke up this morning to outage alerts.  When I got on site, the box had panic'd.  Sadly I wasn't able to do any diagnostics on it, and I didn't think to capture the stack trace it left on screen before restarting it, but I figured I would post this here just in case other people had similar issues.

    I haven't had any issues with the box in the 6 months it's been in use, so the timing suggests it was the update, but I have no proof until/if it does it again.

    Hardware config:
    Xeon E3-1220V3
    ASRock E3C224D2I (Intel C224 chipset)
    Dual i210 LAN onboard
    16 GB ECC DDR3 1333



  • What did you have to do to resolve the problem, more than a reboot?



  • @divsys:

    What did you have to do to resolve the problem, more than a reboot?

    I haven't done anything apart from reboot it.  It has been running for about 3 hours now without issue.

    I suppose it could be completely unrelated and random, but like I said the timing makes me suspicious.

    I have 2 more boxes with the exact same hardware config at remote sites and they are still knock on wood running fine, but this install is running more software and is on a larger site, so is much busier.



  • When I logged into the device this morning it prompted me to upload a crash report, the meat of which I've posted here: http://pastebin.com/pGb2uWcs

    The physical console doesn't show anything changing since I rebooted it yesterday, so I'm not sure if this is the crash dump from yesterday, or if it crashed again.



  • The date of that crash report's creation was Tue Jul 28 22:04:37 2015 (local time of that system).

    Crash report doesn't look familiar. Searching for similar crash reports via Google, and our submitted crash reports, came up empty. The OS level changes from 2.2.3 to 2.2.4 are very minimal. The only thing of any significance was AES-NI, and nothing in the backtrace is even close to anything relevant there. No other reports of crashes after upgrade, with upwards of 2000 systems upgraded in the first 36 hours or so (haven't checked count since).

    Given that, it seems likely it's a hardware issue. Where it's a software issue, generally you'll find other reports via Google of similar backtraces, or within our crash reports.



  • @cmb:

    Given that, it seems likely it's a hardware issue. Where it's a software issue, generally you'll find other reports via Google of similar backtraces, or within our crash reports.

    Yeah I would have expected hardware except this box has been a rock for 6 months now running pfSense, so I'm not sure what to think.  I would have used the cosmic ray excuse, but it's a Xeon E3 machine with ECC RAM.  It's behaving itself now so I'll just assume it wanted attention.  ;D