Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    Crash/kernel panic - reboot twice in short time

    General pfSense Questions
    5
    24
    5887
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      Michael Hovnivar last edited by

      Hello,

      we use pfsense in VMWARE ESXi for long time since 1.3 without any issues. One month ago we upgraded from 2.0.1 to 2.0.2. And pfsense crash and restart twice. First Tuesday, February 05, 2013 11:55:56 PM CET and second Friday, February 15, 2013 9:15:49 AM CET (http://www.timeanddate.com/time/map/ Central Europe - Prague). We click to send crash log to developers, our outgoing IP is 193.85.35.106. Is there any public system where we can track or view our crash report?
      Because we click to delete crash log, we have no log actually.

      Thanks for help.

      And many thanks for great product.

      1 Reply Last reply Reply Quote 0
      • C
        cmb last edited by

        Not anything really obvious to me in those crash reports. One thing that sticks out is you're using vlance/le NICs rather than e1000/em. Changing those to e1000 is highly preferable for performance and stability reasons. I can't recall ever seeing le causing kernel panics, but it's definitely something that should be changed on any production system.

        1 Reply Last reply Reply Quote 0
        • M
          Michael Hovnivar last edited by

          Thanks for reply. I will change NICs to recommended ones ASAP.

          1 Reply Last reply Reply Quote 0
          • M
            Michael Hovnivar last edited by

            Bad news :(
            After change NICs to em and 5 days running crash again. Crash report was send. Previous versions 2.0.1 nor 2.0 nor 1.3 never crashed. Hypervisor (licensed ESXi 4.1 - virtual hardware) hosts other systems without troubles.

            1 Reply Last reply Reply Quote 0
            • C
              cmb last edited by

              what happens if you uninstall open-vm-tools?

              1 Reply Last reply Reply Quote 0
              • H
                heper last edited by

                -check the esxi log/console
                -enable or disable openvm tools
                -check hardware raid for any issues (i've had a faulty raid card once that caused all kinds of issue's)
                -check faulty ram
                -try updating to ESXi 5.x
                -…

                it is unlikely that pfsense itself is causing problems.
                it can however seem that the symptoms only show up with FreeBSD/pfSense because of how esxi deals with that type of VM.

                there are thousands of 2.0.2/2.0.3 systems running on ESXi without any issues.

                1 Reply Last reply Reply Quote 0
                • M
                  Michael Hovnivar last edited by

                  Thanks for reply.
                  Open-Vm-Tools unistall and reboot for sure done.
                  We can't stress test this hardware now, as i said in previous post, other virtual machines (Linuxes) on the same hypervisor are without any issues.
                  Update to ESXi5 is not possible due our planned migration to KVM - licensed Vmware is too expensive for us. We have prepared second similar hardware with KVM. This unexpected issue change our priorities and after another crash we will start migration to KVM.

                  1 Reply Last reply Reply Quote 0
                  • M
                    Michael Hovnivar last edited by

                    Bad news: crash (auto reboot) again, crash logs was sent.
                    Update 3/8/13: crash (auto reboot) again, crash logs was sent. We'll start new install 2.0.2 in KVM (host Ubuntu 12.04) during weekend 3/9-10/13.

                    1 Reply Last reply Reply Quote 0
                    • M
                      Michael Hovnivar last edited by

                      Now we are running pfSense in KVM.

                      1 Reply Last reply Reply Quote 0
                      • M
                        Michael Hovnivar last edited by

                        Ouch, crash again, crash dumps was sent.

                        • running clean install 2.0.2 in KVM, only config was migrated
                        • different hardware than previous

                        In next post i'll show our config and graphs without sensitive information (i hope).

                        1 Reply Last reply Reply Quote 0
                        • M
                          Michael Hovnivar last edited by

                          Crash happen at approx  8:40 AM local time.

                          Physical hardware is IBM x3550M2, Intel network cards (same configuration was with VMWARE, we have two identical servers)
                          VM has 2CPUs, 2GB RAM.

                          config
                          http://ompldr.org/vaHY2Yw

                          rrd graphs
                          system http://ompldr.org/vaHY2ag
                          traffic http://ompldr.org/vaHY2aw
                          packets http://ompldr.org/vaHY2ZA
                          quality http://ompldr.org/vaHY2ZQ
                          queues wan http://ompldr.org/vaHY2Zg
                          queues lan http://ompldr.org/vaHY2Zw
                          queuesdrops wan http://ompldr.org/vaHY2aA
                          queuesdrops lan http://ompldr.org/vaHY2bA

                          1 Reply Last reply Reply Quote 0
                          • M
                            Michael Hovnivar last edited by

                            We are trying blindly to set:
                            [2.0.2-RELEASE][root@vostok.forrest.local]/root: cat /boot/loader.conf.local
                            kern.ipc.nmbclusters="131072"
                            hw.em.num_queues=1
                            http://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards#Intel_igb.284.29_and_em.284.29_Cards

                            1 Reply Last reply Reply Quote 0
                            • M
                              Michael Hovnivar last edited by

                              Ooops, crash again (2013-04-17 11:18 CEST), crash dump sent. In next days we try update to 2.0.3.

                              1 Reply Last reply Reply Quote 0
                              • M
                                Michael Hovnivar last edited by

                                After few weeks with 2.0.3 crash again. In next days we downgrade to 2.0.1.

                                1 Reply Last reply Reply Quote 0
                                • Gertjan
                                  Gertjan last edited by

                                  Hi there.
                                  What about loading pfSense in a stand alone 'PC' ?
                                  This will rule out the VM and hardware factor.

                                  No "help me" PM's please. Use the forum.

                                  1 Reply Last reply Reply Quote 0
                                  • M
                                    Michael Hovnivar last edited by

                                    Real HW is not possible for us. In VM we use Pfsense for few years without any issue - until 2.0.2 and 2.0.3.

                                    1 Reply Last reply Reply Quote 0
                                    • C
                                      cmb last edited by

                                      there is no real OS difference between any of the 2.0.x versions so it's unlikely to be any difference between 2.0.1 and 2.0.2 or 2.0.3.

                                      1 Reply Last reply Reply Quote 0
                                      • M
                                        Michael Hovnivar last edited by

                                        2.0.1 crashes again, in next days we try new install 2.1 RC (i386) with config migration.

                                        1 Reply Last reply Reply Quote 0
                                        • M
                                          Michael Hovnivar last edited by

                                          We had Pfsense 2.1 RC1 installed for 3 days and crash reboot again :(
                                          No option to send debug logs.
                                          Now we are trying to delete Traffic Shapper.

                                          Next step (after another crash reboot) will be load KVM Virtio support.

                                          We have only one additional package innstalled: Zabbix package
                                          We have a few IPSEC tunnels.
                                          We use URL table aliases.
                                          We use some firewall rules with "Advaced settings": different values in "Maximum new connections / per second(s)" - also now disabled.

                                          1 Reply Last reply Reply Quote 0
                                          • M
                                            Michael Hovnivar last edited by

                                            We have two physical identical servers with KVM (Ubuntu 12.04 x64, default package qemu 1.0+noroms-0ubuntu14.7). Both recently sucessfully tested (more then 70 passes) with Memtest. Pfsense 2.1 RC i386 after few weeks got into strange state PAUSED not usual RUNNING (few days ago). This is different behavior than previous automatically reboot. We rebooted Pfsense by hand and updated it to current 2.1.
                                            After next unexpected behavior we plan clean install x64 Pfsense 2.1
                                            … to be continued

                                            1 Reply Last reply Reply Quote 0
                                            • K
                                              kejianshi last edited by

                                              I agree with above.  Go straight hardware if feasible. 
                                              Is your pfsense 64 bits?  How many WAN interfaces or WAN IPs do you have?
                                              How much RAM is allocated to pfsense?

                                              1 Reply Last reply Reply Quote 0
                                              • M
                                                Michael Hovnivar last edited by

                                                @kejianshi:

                                                I agree with above.  Go straight hardware if feasible. 
                                                Is your pfsense 64 bits?  How many WAN interfaces or WAN IPs do you have?
                                                How much RAM is allocated to pfsense?

                                                Direct hardware install is not possible for us. Our pfSense is i386 with 2Gigs of RAM (a lot of RAM is free). We have one WAN with one IP.
                                                After deleting all rule's "Advanced settings - Maximum new connections / per second" (approx. by 15 rules) we have only one reboot and one freeze. Before that, it happened more often.

                                                After next reboot/freeze we will install x64bit version. Virtual host is also x64bit architecture (Linux KVM).

                                                1 Reply Last reply Reply Quote 0
                                                • K
                                                  kejianshi last edited by

                                                  This may seem like a counter-intuitive solution, but - Is 2.1 bringing you something you need that 2.03 wasn't?  You could always use 2.03 until 2.1 gets a patch that makes it stable for you?

                                                  1 Reply Last reply Reply Quote 0
                                                  • M
                                                    Michael Hovnivar last edited by

                                                    Troubles solved.  :)

                                                    When virtualization host has heavy I/O load (due other virtual guest), pfsense on IDE virtual controller has troubles and fall into reboot or other unexpected state. After we load VirtIO drivers https://doc.pfsense.org/index.php/VirtIO_Driver_Support, pfsense is happy and we too.
                                                    But don't allow all VirtIO drivers! VTNET in our case slown down net traffic after few days rapidly.

                                                    Working configs
                                                    virtual guest pfsense:
                                                    pfSense 2.1-RELEASE-pfSense (amd64)
                                                    cat /boot/loader.conf.local
                                                    virtio_load="YES"
                                                    virtio_pci_load="YES"
                                                    #if_vtnet_load="YES"
                                                    virtio_balloon_load="YES"
                                                    virtio_blk_load="YES"
                                                    virtualization host Ubuntu 12.04 64bit:
                                                    Linux xxx 3.2.0-54-generic #82-Ubuntu SMP Tue Sep 10 20:08:42 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
                                                    qemu 1.0+noroms-0ubuntu14.11

                                                    1 Reply Last reply Reply Quote 0
                                                    • First post
                                                      Last post