Reproducible kernel panic with pfSense 2.2 and IPSEC



  • @w0w:

    And you don't have "IP Random id generation" enabled?

    "IP Random id generation" is off. This setting I have never changed. Otherwise, I have only MSS clamping enabled and the Unity plugin disabled. "IPsec Mobile Client Support" is also turned off. All other settings are defaults.

    Apparently the kernel timesource is not the cause of the error. Could someone have another look at the dump? What information is still needed?



  • We need to use bugtracker to report the bug. Please report this issue here https://redmine.pfsense.org/
    Don't link the forum page, post your dumps and I'll add mine too.
    FYI I don't have changed MSS and unity settings.





  • Similar issue here. If I connect via VPN, PFSense stops and restarts - the IPSEC connection lasts between 30 seconds and 5 minutes. This issue is 100% reproducible - I've crashed my box about six times today.

    Fatal double fault:
    eip = 0xc12c62a8
    esp = 0xecf4cff8
    ebp = 0xecf4d000
    cpuid = 0; apic id = 00
    panic: double fault
    cpuid = 0
    KDB: enter: panic
    panic.txt0600001412471723700  7136 ustarrootwheeldouble faultversion.txt06000025112471723700  7614 ustarrootwheelFreeBSD 10.1-RELEASE-p4 #0 36d7dec(releng/10.1)-dirty: Thu Jan 22 15:12:38 CST 2015
        root@pfsense-22-i386-builder:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.10

    PFSense 2.2 - upgraded from 2.1.5

    Hardware:
    CPU: Intel(R) Atom(TM) CPU N270 @ 1.60GHz
    Mobo: KINO-945GSE
    Storage: 2 GB CF Card
    Dual LAN: Realtek PCIe 8111CP GbE controller

    IPSEC Details:
    Mobile Client
    No IP Compression
    Unity plugin disabled
    IKE v1
    Virtual IP Address Assigned to Clients
    IP Random ID Generation at default value (default is 0: sequential IP IDs)

    Interfaces:
    RE0: WAN: PPOE
    RE1: LAN/OPT1/OPT2 using VLAN tagging

    May try a clean install of v2.2 if you think there's any mileage in it.

    pfsenseCrashDump.txt



  • @afasoas: are you running 32 or 64 bit version? So far I think all the reports are coming from x86 versions. Is this reproducible under x64??



  • 32 bit.
    Well spotted.
    I will add my crash dump to the bug tracker shortly.



  • Upgraded my hardware so I could run the 64-bit version. No issues to report thus far. IPSEC seems solid and stable.



  • Great solution, but this is like cutting the head and sewing back a new one, more "brainful".
    It could be also driver Ethernet issue with physical low memory installed. I can only wait when somebody really smart will check our crash dumps to find out the reason of double triple crash and panic.



  • @w0w:

    Great solution, but this is like cutting the head and sewing back a new one, more "brainful".
    It could be also driver Ethernet issue with physical low memory installed. I can only wait when somebody really smart will check our crash dumps to find out the reason of double triple crash and panic.

    I appreciate that this isn't the most helpful solution. I just wanted to confirm that the problem went away using the same configuration with a 64-bit version.

    On Edit: I realise that you are using an Atom D2500 - you can run the 64-bit version of pfSense on it, if that helps?



  • I can, really, but what if I don't need it? My typical memory usage is less then 6% and CPU is mostly 90% in peak (300Mbit internet, three clients). I can buy some XEON based proliant G8 but why? :) 
    Maybe I'll move to x64 platform if we can't trust x86 anymore. But I need an answer for the question. DO we really need to move to x64 just because x86 is not supported or what?
    I think i'll wait for answer before buying some needless hardware. :)



  • And what about the ALIX boards? As far as I know they are all 32bit.


  • Banned

    @sh0gun:

    And what about the ALIX boards? As far as I know they are all 32bit.

    Not getting any IPsec panics on Alix. (Also, make sure you did not enable some stupid features, like the infamous "Insert a stronger id into IP header of packets passing through the filter.")



  • But your hardware is already 64-bit capable, at least as far as pfSense is concerned!
    Yes Intel don't provide 64-bit video drivers but seems to be a non-issue here.

    @w0w:

    My system is D2500CC mini-ITX motherboard from Intel, all embedded into it.



  • @doktornotor:

    @sh0gun:

    And what about the ALIX boards? As far as I know they are all 32bit.

    Not getting any IPsec panics on Alix. (Also, make sure you did not enable some stupid features, like the infamous "Insert a stronger id into IP header of packets passing through the filter.")

    I don't think that hiding your client OS unique ID behind firewall is so stupid as you think about it.



  • @afasoas:

    But your hardware is already 64-bit capable, at least as far as pfSense is concerned!
    Yes Intel don't provide 64-bit video drivers but seems to be a non-issue here.

    @w0w:

    My system is D2500CC mini-ITX motherboard from Intel, all embedded into it.

    Yep. But what is the point to use 64-bit OS with 2GB of RAM? It does not fix the problem in 32-bit version also :) There is some bug, that must be fixed and this is good, maybe, that it is pointed now to 32-bit version only, but next time it could be related to 64-bit only, so migrating between platforms is useless for me, until I read something like "64-bit freebsd is more secure and stable, don't use 32-bit anymore".



  • Migrating between platforms resolves your problem, for the time being.
    If your memory usage is at 6% then I figure there should not be a problem switching over to 64-bit.



  • @w0w:

    …so migrating between platforms is useless for me, until I read something like "64-bit freebsd is more secure and stable, don't use 32-bit anymore".

    Ahh, I think you mean this:
    "[_…64 bit is more widely used, what we test the most with, and what most of our development is done using.

    32 bit is a dying breed. FreeNAS and DragonflyBSD both just put out their last releases with 32 bit support. While we'll still continue to support 32 bit in 2.2.x releases and possibly beyond that, ending 32 bit support is certainly on the road map and will happen sooner than later.

    There is no reason to use 32 bit over 64 today, if your hardware is 64 bit capable, you should only be running 64 bit._](https://forum.pfsense.org/index.php?topic=84679.msg464432#msg464432)"

    Chris Buechler, November 27th, 2014


  • Rebel Alliance Developer Netgate



  • Ok… At least I'll give it a try. Later, next week maybe :)



  • Still, do we have any clues on the issue itself? It affects Alix boards and similar hardware to at least some degree (besides some other reports, I had to downgrade some production boxes myself due to random reboots which I am sure are related to all this). Unfortunately I couldn't find a specific trigger but there seem to be several crash dumps and reproducible configs available



  • https://redmine.pfsense.org/issues/4454 Looks like something moving forward, at least for me, thanks for Chris Buechler



  • Ok, installed amd64, no more kernel panic when "Insert strong ID…" option enabled. So yes, I can confirm that x86 platform is affected and amd64 not.



  • Can you try to set net.inet.ipsec.directdispatch to 0 and see if the panic goes away?



  • Update my pfsense in lab on 2.2.1 same behavior
    then i tried  to

    set net.inet.ipsec.directdispatch to 0

    looks good so far. stable since about 30 minutes (before after about 30 seconds i get a kernel panic)



  • Can you describe your WAN interface?



  • in our lab i used 3 pfsenses

    one so to say provider with an pppoe server (Version 2.2.1 Vmware)

    then i have one pfsense which stands for my company firewall (Version 2.1.5 Vmware)

    and i have another pfsense which stand for my home pfsense  (Version 2.2.1 Alix2d3)

    both pfsenses from company and home have a pppoe wan interface which is connected to my provider pfsense

    on my home pfsense i also have a vlan tag added like i have to do it at my real home pfsense.



  • Ok there is an open issue for this scenario already.

    Thank you for the information.



  • yes my scenario is like described befor in this thread.
    but set net.inet.ipsec.directdispatch to 0 seems to "workaround" the issue
    so there is may be hope for all 32 Bit Users  ;)



  • ermal, do you need my report too? :)
    Actually I am the man who reported the issue. But I moved my box on to amd64 version…
    I ask because I have troubles to restoring old x86 backup, so new installation take time... I can do it but only if it really needed.



  • Nope the scenario is clear.



  • i can reproduce the scenario in our lab any time. The VM's are set up already just need to put back the config into the alix board.
    If you need just let me know.



  • @flix87:

    Update my pfsense in lab on 2.2.1 same behavior
    then i tried  to

    set net.inet.ipsec.directdispatch to 0

    looks good so far. stable since about 30 minutes (before after about 30 seconds i get a kernel panic)

    Hello, had the same issues with reboot`s on 8 devices.

    Hardware ALIX.2 v0.99m tinyBIOS V1.4a.
    Pf.Version 2.2.2-RELEASE (i386) built on Mon Apr 13 20:10:33 CDT 2015.

    After the set of "net.inet.ipsec.directdispatch to 0" all systems works fine. No reboots, no systempanics, stable 3 days ago yet.  :)

    What is done with this adjustment ? Will this fix embedded as standard for further versions ?

    Best regards

    eeit



  • net.inet.ipsec.directdispatch=0

    fixes the issue on ALIX 2D13 (32bit) with IPSEC for me. Thank you very much! Marked as solved.



  • Can you please having issues with this confirm that you are running Proxy arps?
    If yes, can you try with the fix at https://redmine.pfsense.org/issues/4685 last comment.



  • @ermal:

    Can you please having issues with this confirm that you are running Proxy arps?
    If yes, can you try with the fix at https://redmine.pfsense.org/issues/4685 last comment.

    I'm having the same issue (see https://forum.pfsense.org/index.php?topic=94140.0) and afaik i'm not using proxy arps. None of the previous tunables (including net.inet.ipsec.directdispatch=0) did fix it!



  • I'm getting this error on my x86 pfsense box when booting up.

    How can I set  set net.inet.ipsec.directdispatch to 0?

    I've connected a monitor and keyboard, booted up my system
    Selected '3. Escape to loader prompt.'
    I type:: 'set net.inet.ipsec.directdispatch=0'
    Then type: 'boot'

    I get the same kernel panic? Am I doing this wrong? Can anyone help me? :)

    Thanks,
    Brad



  • @bab5470:

    I'm getting this error on my x86 pfsense box when booting up.

    Just make sure you're running 2.2.4, that's been fixed in 2.2.3 and newer. directdispatch is automatically set accordingly for 32 bit to avoid that crash. If you're already on 2.2.4, then you have a different problem, start a new thread detailing what you're seeing.



  • How can I upgrade if I can't even boot up the box though? I assume I could re-install from scratch but I don't really want to lose all my settings.



  • I tried booting from a live cd 2.2.4 (x86) and get the same KBD panic for what its worth. Does that mean this is a new/different issue?

    So frustrating that a simple upgrade has killed my pfsense box.

    Maybe the answer is I need to downgrade. What is the safest version that doesn't have this issue?



  • If I do show net.inet.ipsec.directdispatch it reports 0

    If I then do boot it panic's again.

    So I'm pretty sure this isn't helping in my case. Shall I open a new thread?


Log in to reply