Crash/kernel panic - reboot twice in short time
-
Hello,
we use pfsense in VMWARE ESXi for long time since 1.3 without any issues. One month ago we upgraded from 2.0.1 to 2.0.2. And pfsense crash and restart twice. First Tuesday, February 05, 2013 11:55:56 PM CET and second Friday, February 15, 2013 9:15:49 AM CET (http://www.timeanddate.com/time/map/ Central Europe - Prague). We click to send crash log to developers, our outgoing IP is 193.85.35.106. Is there any public system where we can track or view our crash report?
Because we click to delete crash log, we have no log actually.Thanks for help.
And many thanks for great product.
-
Not anything really obvious to me in those crash reports. One thing that sticks out is you're using vlance/le NICs rather than e1000/em. Changing those to e1000 is highly preferable for performance and stability reasons. I can't recall ever seeing le causing kernel panics, but it's definitely something that should be changed on any production system.
-
Thanks for reply. I will change NICs to recommended ones ASAP.
-
Bad news :(
After change NICs to em and 5 days running crash again. Crash report was send. Previous versions 2.0.1 nor 2.0 nor 1.3 never crashed. Hypervisor (licensed ESXi 4.1 - virtual hardware) hosts other systems without troubles. -
what happens if you uninstall open-vm-tools?
-
-check the esxi log/console
-enable or disable openvm tools
-check hardware raid for any issues (i've had a faulty raid card once that caused all kinds of issue's)
-check faulty ram
-try updating to ESXi 5.x
-…it is unlikely that pfsense itself is causing problems.
it can however seem that the symptoms only show up with FreeBSD/pfSense because of how esxi deals with that type of VM.there are thousands of 2.0.2/2.0.3 systems running on ESXi without any issues.
-
Thanks for reply.
Open-Vm-Tools unistall and reboot for sure done.
We can't stress test this hardware now, as i said in previous post, other virtual machines (Linuxes) on the same hypervisor are without any issues.
Update to ESXi5 is not possible due our planned migration to KVM - licensed Vmware is too expensive for us. We have prepared second similar hardware with KVM. This unexpected issue change our priorities and after another crash we will start migration to KVM. -
Bad news: crash (auto reboot) again, crash logs was sent.
Update 3/8/13: crash (auto reboot) again, crash logs was sent. We'll start new install 2.0.2 in KVM (host Ubuntu 12.04) during weekend 3/9-10/13. -
Now we are running pfSense in KVM.
-
Ouch, crash again, crash dumps was sent.
- running clean install 2.0.2 in KVM, only config was migrated
- different hardware than previous
In next post i'll show our config and graphs without sensitive information (i hope).
-
Crash happen at approx 8:40 AM local time.
Physical hardware is IBM x3550M2, Intel network cards (same configuration was with VMWARE, we have two identical servers)
VM has 2CPUs, 2GB RAM.config
http://ompldr.org/vaHY2Ywrrd graphs
system http://ompldr.org/vaHY2ag
traffic http://ompldr.org/vaHY2aw
packets http://ompldr.org/vaHY2ZA
quality http://ompldr.org/vaHY2ZQ
queues wan http://ompldr.org/vaHY2Zg
queues lan http://ompldr.org/vaHY2Zw
queuesdrops wan http://ompldr.org/vaHY2aA
queuesdrops lan http://ompldr.org/vaHY2bA -
We are trying blindly to set:
[2.0.2-RELEASE][root@vostok.forrest.local]/root: cat /boot/loader.conf.local
kern.ipc.nmbclusters="131072"
hw.em.num_queues=1
http://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards#Intel_igb.284.29_and_em.284.29_Cards -
Ooops, crash again (2013-04-17 11:18 CEST), crash dump sent. In next days we try update to 2.0.3.
-
After few weeks with 2.0.3 crash again. In next days we downgrade to 2.0.1.
-
Hi there.
What about loading pfSense in a stand alone 'PC' ?
This will rule out the VM and hardware factor. -
Real HW is not possible for us. In VM we use Pfsense for few years without any issue - until 2.0.2 and 2.0.3.
-
there is no real OS difference between any of the 2.0.x versions so it's unlikely to be any difference between 2.0.1 and 2.0.2 or 2.0.3.
-
2.0.1 crashes again, in next days we try new install 2.1 RC (i386) with config migration.
-
We had Pfsense 2.1 RC1 installed for 3 days and crash reboot again :(
No option to send debug logs.
Now we are trying to delete Traffic Shapper.Next step (after another crash reboot) will be load KVM Virtio support.
We have only one additional package innstalled: Zabbix package
We have a few IPSEC tunnels.
We use URL table aliases.
We use some firewall rules with "Advaced settings": different values in "Maximum new connections / per second(s)" - also now disabled. -
We have two physical identical servers with KVM (Ubuntu 12.04 x64, default package qemu 1.0+noroms-0ubuntu14.7). Both recently sucessfully tested (more then 70 passes) with Memtest. Pfsense 2.1 RC i386 after few weeks got into strange state PAUSED not usual RUNNING (few days ago). This is different behavior than previous automatically reboot. We rebooted Pfsense by hand and updated it to current 2.1.
After next unexpected behavior we plan clean install x64 Pfsense 2.1
… to be continued -
I agree with above. Go straight hardware if feasible.
Is your pfsense 64 bits? How many WAN interfaces or WAN IPs do you have?
How much RAM is allocated to pfsense? -
I agree with above. Go straight hardware if feasible.
Is your pfsense 64 bits? How many WAN interfaces or WAN IPs do you have?
How much RAM is allocated to pfsense?Direct hardware install is not possible for us. Our pfSense is i386 with 2Gigs of RAM (a lot of RAM is free). We have one WAN with one IP.
After deleting all rule's "Advanced settings - Maximum new connections / per second" (approx. by 15 rules) we have only one reboot and one freeze. Before that, it happened more often.After next reboot/freeze we will install x64bit version. Virtual host is also x64bit architecture (Linux KVM).
-
This may seem like a counter-intuitive solution, but - Is 2.1 bringing you something you need that 2.03 wasn't? You could always use 2.03 until 2.1 gets a patch that makes it stable for you?
-
Troubles solved. :)
When virtualization host has heavy I/O load (due other virtual guest), pfsense on IDE virtual controller has troubles and fall into reboot or other unexpected state. After we load VirtIO drivers https://doc.pfsense.org/index.php/VirtIO_Driver_Support, pfsense is happy and we too.
But don't allow all VirtIO drivers! VTNET in our case slown down net traffic after few days rapidly.Working configs
virtual guest pfsense:
pfSense 2.1-RELEASE-pfSense (amd64)
cat /boot/loader.conf.local
virtio_load="YES"
virtio_pci_load="YES"
#if_vtnet_load="YES"
virtio_balloon_load="YES"
virtio_blk_load="YES"
virtualization host Ubuntu 12.04 64bit:
Linux xxx 3.2.0-54-generic #82-Ubuntu SMP Tue Sep 10 20:08:42 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
qemu 1.0+noroms-0ubuntu14.11
