Fatal trap 12 kernel panic 2.4.2 random reboot {solved maybe} - turn off crypto



  • Not sure where to post this, but I think I found the source issue and a work around.

    Symptom : random reboot / reset of firewall , sometimes with error, sometimes not. time between reboot anywhere from 10 minutes to 2 day. When I did get a crash report, always the same error.

    Fatal trap 12: page fault while in kernel mode …....

    Tried everything, better cooling, different chipset NIC, disabling on board SATA , (bios :turning off hyper threading, turing off vt-d , turning off above 4g decoding ) , I did system tunables too, mbuf , some storm threshold, and I forget what else......

    My firewall was a custom made one (supermicro) , so I eventually gave up and ordered a brand new firewall from pfSense . Got the same exact problem , but I noticed it didn't pop up till after I noticed AES-NI was disabled and turned it on.  I put my old firewall back in, and turned off crypto acceleration , and I am now on day three without a reset !

    Can any one else confirm this experience with 2.4.x  ?    Incidentally all 8 of my other pfSense firewalls , had no issues on 2.3.x , and the new one is too new to install the 2.3.x .

    not sure if it is part of it, but I have VLANs and virtual IPs, and a lot of IPsec tunnels all configured for AES128-GCM and AES-XCBC

    Hardware is a Supermicro 5019S-ML  , Intel Xeon E3-1275  , 64GB ECC ram


  • Netgate Administrator

    We would need to see the actual crash report or at at least the backtrace to know more. Did you submit any of the reports? From the same IP you're posting from here?

    Steve



  • Seems to be a known issue;

    https://forum.pfsense.org/index.php?topic=139146.0

    https://redmine.pfsense.org/issues/8070

    pfSense Support told me they are aware of this issue…



  • yes, I did submit them (a bunch),  it should be from the same IP as this post.

    4 days now, no crash, so I am going to say turning off crypto is a genuine workaround.



  • Interesting, on the known issue. Initially I did not get errors either, whish I would have seen that post earlier….. but after playing with the bios I was eventually able to get the crash dump (unlike the poor fellow in that other post), which is where I focused my search. fatal trap 12.



  • Looks like there may be a patch available now !

    https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219356


  • Netgate Administrator

    That patch was added to FreeBSD and hence pfSense in June last year. It's in 2.4.2.
    https://github.com/pfsense/FreeBSD-src/commit/d8ee8035f662ebc549539a73c8d4d567ab467aa7#diff-180ff01618a0953b720ec87cf8169682

    I'd be interested to know where that information came from because I'm certainly not aware of that issue.  ???
    We use AES-GCM tunnels with AES-NI enabled internally and I've not seen any reports of this. If anyone has a support ticket number I can review I'll do that.

    Steve



  • We have a similar (probably the same) problem - plagueing us for a while now. We use IKEv2 EAP-Radius with aes256-gcm on an SG-8860 on a 1gb fiber uplink.

    When one of our users (he is on 100mbit fiber) tries e.g. speedtest.net while on the VPN, the pfsense box reliably crashes after a few seconds of upload (download works fine). When I try this at home on a 100mbit/40mbit DSL link, I can create all the traffic I'd like and can't get the box to crash.

    I now switched algorithms to AES-256 with SHA512 (still with AES-NI, I didn't disable that) and it seems the crashes have either gone away or we weren't yet able to reproduce them today.
    Kind regards,
    Lukas


 

© Copyright 2002 - 2018 Rubicon Communications, LLC | Privacy Policy