PfSense Crashing



  • Hi,

    I was wondering if anyone could give me a bit of insight as to why pfSense is crashing. I have posted the crash report on pastebin.

    http://pastebin.com/4634WiH1

    Thank you very much,



  • Can you remove the sound card, or disable it in the BIOS if it's onboard? It's sharing an IRQ with two of your NICs, which looks like it could be related possibly.



  • Hi,

    Sorry for the late reply. It took awhile to actually be able to disable the sound card and in addition to that we removed any unused devices. I also updated to 2.2.4 just to be sure it didn't get fixed.

    Here is the new pastebin of the crash.

    http://pastebin.com/3F3XHbRS

    Thank you for looking at this.



  • https://www.freebsd.org/doc/en_US.ISO8859-1/books/arch-handbook/boot-bios.html
    The instruction pointer register (also known as the Program Counter) points to code to be executed by the processor.

    The below are identical in both reports, even though you have different versions with the 2nd report being from last month.Edit. My bad, the crash reports opened up on new tabs the other way round.

    Fatal trap 12: page fault while in kernel mode
    instruction pointer    = 0x20:0xc0d7cde0

    Are you running freenas on pfsense?

    https://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/advanced.html#idp60781776
    How can I make the most of the data I see when my kernel panics?
    While the instruction pointer value is important, it is also configuration dependent as it varies depending on the kernel image. If it is a GENERIC kernel image from one of the snapshots, it is possible for somebody else to track down the offending function, but for a custom kernel, only you can tell us where the fault occurred.

    The below might help you but if not follow the steps in the faq link above until you do.
    nm -n kernel.that.caused.the.panic | grep c0d7cde0
    no success then try
    nm -n kernel.that.caused.the.panic | grep c0d7cde
    no success then try
    nm -n kernel.that.caused.the.panic | grep c0d7cd

    Repeat until there is some sort of output.

    Theres also another procedure you can follow if you get no where with the above method which is the next part of the same link.
    https://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/advanced.html#idp60781776
    _However, the best way to track down the cause of a panic is by capturing a crash dump, then using kgdb(1) to generate a stack trace on the crash dump.

    In any case, the method is this:…..._

    Let us know what you find out.  ;)



  • It's either a hardware problem, or an issue with FreeBSD and that combination of hardware.



  • I am not running FreeNAS, Simple pfSense install with some NAT, DHCP, freeDNS and QoS.

    I am going to try swapping the hardware into something different tonight (hopefully). Right now its running off an old dell. I'll re-create the problem and see if it still persists on different hardware.



  • @kingpyro:

    I am not running FreeNAS, Simple pfSense install with some NAT, DHCP, freeDNS and QoS.

    I am going to try swapping the hardware into something different tonight (hopefully). Right now its running off an old dell. I'll re-create the problem and see if it still persists on different hardware.

    So what steps do you do that currently causes the crash? Wondering what part of the system it is, as that could be indicative.



  • @firewalluser:

    @kingpyro:

    I am not running FreeNAS, Simple pfSense install with some NAT, DHCP, freeDNS and QoS.

    I am going to try swapping the hardware into something different tonight (hopefully). Right now its running off an old dell. I'll re-create the problem and see if it still persists on different hardware.

    So what steps do you do that currently causes the crash? Wondering what part of the system it is, as that could be indicative.

    I swapped the config file and restored it into a Lenovo M58. A newer computer which should have been more compatible. Unfortunately it still crashed. The only component that was the same was the secondary NIC ( not the exact same component that was in the dell, but same model and part) Even more disappointing is that the computer did not come back online after crashing and I am unable to retrieve the new crash logs until tomorrow night GMT -5.

    As for how I am re-creating this problem is using Minecraft Server software Spigot. Any version of this software 1.8.3 or higher. Upon detonating a large amount of TnT causes pfSense to shoot a large bandwidth spike, freeze and crash.

    I know something simple like a minecraft server might be fairly trivial and perhaps a bit childish, but I assure you it is professionally run and i'm not sure why something as simple as that could create such an issue.

    Please let me know if you need any additional information.



  • Whats your pfsense setup? ie. what packages do you have running if any and how is it setup?

    When you see the massive traffic spike where do you see this, dashboard with traffic monitors or rrd graphs.

    You might be running out of resources which could be causing the crash in itself.



  • @firewalluser:

    Whats your pfsense setup? ie. what packages do you have running if any and how is it setup?

    When you see the massive traffic spike where do you see this, dashboard with traffic monitors or rrd graphs.

    You might be running out of resources which could be causing the crash in itself.

    I see the spike as I am RDP'd in I see a small glimpse of the traffic graph on the dashboard spike up to about 3megs then fall off. I attached a picture of the screen just as it was crashing while I was RDP'd in and lost connection. The picture was also taken on the old Dell box.

    I think the only package I do have installed is openVPN, however I do not use it and am unsure how to fully uninstall it as it is also not setup.




  • So you upgraded this from a previous version instead of a fresh installation which formats the hard drive?

    Have you tried backing up the config, then install a fresh copy from scratch in case anything has been left over from previous versions?

    OpenVPN is built in a bit, but there is/was a client generation package which could be installed as well. Packages can be uninstalled from the System, Packages menu, select installed packages tab. If you do a fresh install, I'd uninstall before any openvpn packages before backing up and then installing from scratch again. That eliminates anything left over causing a problem at least.

    Its also worth bearing in mind that newer versions also need more resources, but if you cant find anything in the logs, if you get a crash after installing from scratch, maybe a packet capture would help but you would need to direct the output to another device in case the tcpdump pcap files get wiped or not written properly on pfsense in the crucial seconds before a crash.



  • @firewalluser:

    So you upgraded this from a previous version instead of a fresh installation which formats the hard drive?

    Have you tried backing up the config, then install a fresh copy from scratch in case anything has been left over from previous versions?

    OpenVPN is built in a bit, but there is/was a client generation package which could be installed as well. Packages can be uninstalled from the System, Packages menu, select installed packages tab. If you do a fresh install, I'd uninstall before any openvpn packages before backing up and then installing from scratch again. That eliminates anything left over causing a problem at least.

    Its also worth bearing in mind that newer versions also need more resources, but if you cant find anything in the logs, if you get a crash after installing from scratch, maybe a packet capture would help but you would need to direct the output to another device in case the tcpdump pcap files get wiped or not written properly on pfsense in the crucial seconds before a crash.

    I did a fresh install on a different machine and just restored the old config (set interfaces and such). I think that the newer box has a sufficient amount of resources to handle the small amount of traffic that is generated. It's definatly not running out of RAM or HDD storage. as that is 2GB and 80GB respectively. I will try doing a fresh install again with default settings right off the hop and see what that bears. However that might take awhile to do as I do not always have direct access to the box.

    EDIT: The queue settings wouldn't happen to have any take into this would it? As the traffic that comes from the server is high priority, but maybe a packet that size/amount might disrupt something? I am unsure.



  • Quite possibly, although traffic shaping is not on by default so PRIQ is not enabled at least on what I have here.

    I'd look through all your logs if you can and see if anything shows up as I suspect its a lack of resource possibly behind the crash.

    I've noticed my 8Gb SSD machine is struggling to stream data from a radio station causing frequent stutters and hangs in the last two version of pfsense, which I've yet to track down, and I could get it to crash with a 2.42mbps ddos when I attempted to look at the RRD graphs.

    https://doc.pfsense.org/index.php/Traffic_Shaping_Guide#Priority_Queueing_.28PRIQ.29
    "Priority queuing is the simplest form of traffic shaping, and often the most effective. It performs prioritzation of traffic only, without regard for bandwidth."

    https://doc.pfsense.org/index.php/ALTQ_drivers

    Section 30.3.2. Enabling ALTQ for more info, but I dont think its on as a default option.
    https://www.freebsd.org/doc/handbook/firewalls-pf.html

    You nic's might not even support some or all of the ALTQ features either.