PfSense 2.2 crashes (hard) when accessing Web Configurator from remote IPsec VPN



  • We have a stable pfSense 2.1.5 (i386) under VMWare ESXi 5.5.0 that I remotely manage through IPSec VPN. I'm normally able to access the pfSense web configurator from the remote IPSec network just fine and have no issues.

    However, under pfSense 2.2 with the new IPSec software, as soon as I access that remote pfSense 2.2 instance's web configurator, it crashes (hard, with dump which I submitted). If I remote desktop / VNC to different host on that remote network and then access the pfSense 2.2 instance (effectively the same as accessing it as if I'm local), it does not crash. So, that limits the crash behaviour to only when I remotely access the Web Configurator over the IPSec site-2-site VPN.

    If now had to roll back to pfSense 2.1.5 as the issue does not happen then. Below is some context about the configuration and "upgrade" procedure followed:

    _I've re-installed the VMWare Virtual Machine with pfSense 2.2 (i386) and restored the v2.1.5 config.xml. I do it this way, because prefer a clean install instead of an "Upgrade" install - surely this can't be an issue?

    After the restore of the configuration, the firewall reboots and spends a long time at "Configuring Firewall…" I've learnt from other posts that this is normal the first time, and after about 8 minutes it does indeed continue and also starts to restores the packages as per the config.xml.

    After package restore install is completed, I find that the Squid3 package did NOT install, however the SquidGuard3 package did install. I fix this by uninstalling SquidGuard3 and then manually installing Squid3 and then SquidGuard3 - the configuration information is still there after the post. We use Squid3 for a forward proxy which worked fine under 2.2, and also as a reverse proxy but this was not working despite the service running with the configuration the same as before. The logs didn't contain any error messages either. I haven't been able to figure out why yet, and the crash issue is taking priority.

    As for the IPSec VPN configuration, with pfSense 2.2 I change the VPN configuration from IKEv1 to use IKEv2 instead (on both ends) and I'm able to successfully establish the IPSec VPN again and send traffic between the two networks - all as expected.

    The crash happens the moment I browse to the web configurator over the VPN, and the browser never receives anything back before the crash reboot.

    I'm not sure if this is relevant, but I run the web configurator with on a different port (not 443) as the reverse proxy includes 443 servers with certs._

    Any ideas?



  • no clue what causes that issue … that's for the devs to find out.

    but why run i386? x64 is more stable and tested.



  • Squid3 package did NOT install, however the SquidGuard3 package did install.

    I bet it wouldn't crash if these were not installed.



  • @heper:

    no clue what causes that issue … that's for the devs to find out.

    but why run i386? x64 is more stable and tested.

    @heper - Unfortunately I have to run i386 because I cannot run a x64 guest in VMWare ESXI when the host PC does not supprt the VT-x and related XD/NX extensions (It's an old dual Intel Xeon 1U server).

    @kejianshi:

    Squid3 package did NOT install, however the SquidGuard3 package did install.

    I bet it wouldn't crash if these were not installed.

    @kejianshi OK Thanks - I've got a small maintenance window coming up, and can switch in a different virtual hard disk with the 2.2 build on. I'll try that to rule that in or out and post back when I know.



  • I removed the Squid3 and SquidGuard3 packages and rebooted.

    Unfortunately it still crashes the moment I access the web configurator via the IPSec VPN.

    I'll try a clean pfSense 2.2 install (i.e. without restoring any configuration) and then manually create a IPSec setup and test that scenario to try and isolate the conditions. I've run out of time available, so I'll post back here with the resuls, when I have managed to do that too.



  • Seems strange.  Just wondering…  Is there a reason you prefer IPSEC to Openvpn?



  • It looks like we are having similar issue here:
    Running i386 on alix2d13.

    Since an upgrade - connecting via IPSec VPN(mobile client)  and then tying to access the IP address of the pFsense box will core dump the box. SSH & HTTP are the same.
    ICMP requests are not affecting the box. The Hosts  on the remote network can be accessed with no problems.

    the same behavior  is observed on the box with upgraded image  as well as on the box with clean install+restore backup config

    Any recommendations ?


  • Banned

    @volandg:

    It looks like we are having similar issue here:
    Running i386 on alix2d13.

    Yeah, this thread is about ESXi; no idea why did you jump in here. As a quick guess, disable the glxsb crypto accelerator, might be completely toast. And move to your own thread.



  • @doktornotor:

    @volandg:

    It looks like we are having similar issue here:
    Running i386 on alix2d13.

    Yeah, this thread is about ESXi; no idea why did you jump in here. As a quick guess, disable the glxsb crypto accelerator, might be completely toast. And move to your own thread.

    the thread is about pFsense crashing upon using the IPSec VPN(site to site)  and accessing the the remote IP address of the box. That is the main problem. Isn't it?

    Due to the fact i can see the same behavior when doing mobile IPSec it looks like both cases are related somehow. don't you think?

    P.S  crypto accelerator is disabled



  • volandg (or anyone else who can replicate this), could you send me a backup of your config? Can be from status.php which removes keys, password hashes, etc. Can PM it to me here.

    I'd especially like to get yours volandg, since I have an ALIX, to test on the same hardware.



  • @cmb:

    volandg (or anyone else who can replicate this), could you send me a backup of your config? Can be from status.php which removes keys, password hashes, etc. Can PM it to me here.

    I'd especially like to get yours volandg, since I have an ALIX, to test on the same hardware.

    submitted via PM, happy to do any testing or provide any additional information if needed



  • @Jaans:

    it crashes (hard, with dump which I submitted).

    Just noticed this part. Could you PM me the IP that crash report was submitted from? I looked for anything from the IP you're accessing the forum from, but nothing recent that's even close to that IP.



  • @cmb:

    @Jaans:

    it crashes (hard, with dump which I submitted).

    Just noticed this part. Could you PM me the IP that crash report was submitted from? I looked for anything from the IP you're accessing the forum from, but nothing recent that's even close to that IP.

    Thanks!
    Hehe… yeah the IP that submitted the crash dumps is about 9000kms away from me ;-) Will PM you the IP blocks it would've originated from shortly.



  • Well, after two days of hard work to make my Windows 7 notebook connect to my pfSense 2.2 using IKEv2 and EAP-TLS I successfully connected just a few minutes ago for the first time.

    My first test was to access the pfSense webGUI over the IPsec connection which resulted in the same hard crash of my pfSense machine (net6501, nanoBSD). I had to power cycle to restart it.

    Regards,
    Peter



  • And if you need a fix while waiting 2.2.1, use openvpn.



  • @kejianshi:

    And if you need a fix while waiting 2.2.1, use openvpn.

    Yeah, OpenVPN is running since 2.0.1 without any issues and I like it much more because of the more straight forward concept. I just thought it would be nice to get familiar with IPsec as well. But all in all it's much harder to set up. And I have found another issue with CRL check, will open a thread soon. I strongly suppose, some of the IPsec problems are related to pfSense migrating from racoon to StrongSwan.

    Regards,
    Peter



  • I suspect they ALL are…  Gained some function but lost some stability.

    However, CMB pointed out, some of the functioning IPSEC configs out there never should have worked in the 2.15 - they just happened to.

    And so people with badly configured IPSEC on 2.15 that happen to work anyway break upon upgrade.

    Anyway - sounds like a reasonable explanation.

    I never found IPSEC that reliable compared to PPTP or openvpn and since PPTP isn't secure, the choice is clear for me.



  • @pvoigt:

    Well, after two days of hard work to make my Windows 7 notebook connect to my pfSense 2.2 using IKEv2 and EAP-TLS I successfully connected just a few minutes ago for the first time.

    My first test was to access the pfSense webGUI over the IPsec connection which resulted in the same hard crash of my pfSense machine (net6501, nanoBSD). I had to power cycle to restart it.

    Regards,
    Peter

    I'm having the exact same problem.

    It's a clean 2.2 install on soekris net6501-70

    I used this guide to setup ipsec. https://doc.pfsense.org/index.php/IKEv2_with_EAP-MSCHAPv2

    I'm able to browse other clients on my network but as soon as i try to access the pfsense GUI it chrashes.

    Regards.



  • What is the status of this issue? Is it recognized as a bug and being addressed ?



  • This is a VERY serious problem. Here's another topic which I bet is related: https://forum.pfsense.org/index.php?topic=88606

    I guess you can try what's suggested there. I'm also expericing crashes but couldn't find a specific trigger



  • I've seen the same issue on v2.2 over VMware (both ends virtual, IPSec between them), so it's not a hardware problem. Worked fine for years on v2.0x before, even on beta versions and on the same host hardware.
    In my case it SEEMS to be fixed by disabling IPcomp (IP Compression) on the Advanced tab under IPsec, and begins to fail again with a double fault as soon as I re-enable compression. Went from crashing more than once per hour to staying stable for more than 24h and counting.
    I already submitted one of the crash dumps for evaluation.



  • Should we report this as possible  bug on pfSense bugtracker and refer to this thread ?



  • Exact same issue here on a Sokeris net6501. Does anyone know if this is in the bug tracker? I had a look but couldn't see it.



  • Just completed the upgrade to 2.2.1.

    The issue is still there. Accessing the webUI while connected over IPSec VPN will crash the box.





  • Could you all test by setting net.inet.ipsec.directdispatch to 0 as ermal outlined in this post? I believe this is all the same issue



  • I think I ran into the same problem (2.2.1-RELEASE on AMD G-T40E) and submitted a few crash dumps.

    I set net.inet.ipsec.directdispatch to 0 and have not had another crash since then.



  • I have the exact same issue, my remote pfsense instantly crashes when I access it via remote ip. if I remote desktop to the server on the local network, I can access the box, but using ipsec and accessing the box from a different subnet the box crashes. I removed all squid and squid guard rebooted and It still crashes.



  • Nambi, did you try the fix that georgeman suggested??



  • I'm not too sure how to enter in that command.

    do i add this into this the "Execute Shell command"
    net.inet.ipsec.directdispatch=0

    I entered this in there, but when I remote connected it still crashed I probably didn't put it in properly.

    Thanks for the assistance.



  • I'm not too sure how to enter in that command.

    System - Advanced - System Tunables



  • Thank you, there was no entry so i added "net.inet.ipsec.directdispatch" and set it to 0 and this fixed the issue.

    Thanks for your patience.



  • Go to System, Advanced and enter it as a tunable. Reboot just in case and test again

    Are you on 32-bit??



  • yes we are running 32 bit, ,my system is capable for 64 bit, but I installed pfsesne a while back when 32 bit was the norm.



  • Alright, it is confirmed to be the same issue then, thanks



  • @georgeman:

    Alright, it is confirmed to be the same issue then, thanks

    thanks it works ! :)