Crash/kernel panic - reboot twice in short time

Michael Hovnivar

Ouch, crash again, crash dumps was sent.

running clean install 2.0.2 in KVM, only config was migrated
different hardware than previous

In next post i'll show our config and graphs without sensitive information (i hope).

Michael Hovnivar

Crash happen at approx 8:40 AM local time.

Physical hardware is IBM x3550M2, Intel network cards (same configuration was with VMWARE, we have two identical servers)
VM has 2CPUs, 2GB RAM.

config
http://ompldr.org/vaHY2Yw

rrd graphs
system http://ompldr.org/vaHY2ag
traffic http://ompldr.org/vaHY2aw
packets http://ompldr.org/vaHY2ZA
quality http://ompldr.org/vaHY2ZQ
queues wan http://ompldr.org/vaHY2Zg
queues lan http://ompldr.org/vaHY2Zw
queuesdrops wan http://ompldr.org/vaHY2aA
queuesdrops lan http://ompldr.org/vaHY2bA

Michael Hovnivar

We are trying blindly to set:
[2.0.2-RELEASE][root@vostok.forrest.local]/root: cat /boot/loader.conf.local
kern.ipc.nmbclusters="131072"
hw.em.num_queues=1
http://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards#Intel_igb.284.29_and_em.284.29_Cards

Michael Hovnivar

Ooops, crash again (2013-04-17 11:18 CEST), crash dump sent. In next days we try update to 2.0.3.

Michael Hovnivar

After few weeks with 2.0.3 crash again. In next days we downgrade to 2.0.1.

Gertjan

Hi there.
What about loading pfSense in a stand alone 'PC' ?
This will rule out the VM and hardware factor.

Michael Hovnivar

Real HW is not possible for us. In VM we use Pfsense for few years without any issue - until 2.0.2 and 2.0.3.

cmb

there is no real OS difference between any of the 2.0.x versions so it's unlikely to be any difference between 2.0.1 and 2.0.2 or 2.0.3.

Michael Hovnivar

2.0.1 crashes again, in next days we try new install 2.1 RC (i386) with config migration.

Michael Hovnivar

We had Pfsense 2.1 RC1 installed for 3 days and crash reboot again :(
No option to send debug logs.
Now we are trying to delete Traffic Shapper.

Next step (after another crash reboot) will be load KVM Virtio support.

We have only one additional package innstalled: Zabbix package
We have a few IPSEC tunnels.
We use URL table aliases.
We use some firewall rules with "Advaced settings": different values in "Maximum new connections / per second(s)" - also now disabled.

Michael Hovnivar

We have two physical identical servers with KVM (Ubuntu 12.04 x64, default package qemu 1.0+noroms-0ubuntu14.7). Both recently sucessfully tested (more then 70 passes) with Memtest. Pfsense 2.1 RC i386 after few weeks got into strange state PAUSED not usual RUNNING (few days ago). This is different behavior than previous automatically reboot. We rebooted Pfsense by hand and updated it to current 2.1.
After next unexpected behavior we plan clean install x64 Pfsense 2.1
… to be continued

kejianshi

I agree with above. Go straight hardware if feasible.
Is your pfsense 64 bits? How many WAN interfaces or WAN IPs do you have?
How much RAM is allocated to pfsense?

Michael Hovnivar

@kejianshi:

I agree with above. Go straight hardware if feasible.
Is your pfsense 64 bits? How many WAN interfaces or WAN IPs do you have?
How much RAM is allocated to pfsense?

Direct hardware install is not possible for us. Our pfSense is i386 with 2Gigs of RAM (a lot of RAM is free). We have one WAN with one IP.
After deleting all rule's "Advanced settings - Maximum new connections / per second" (approx. by 15 rules) we have only one reboot and one freeze. Before that, it happened more often.

After next reboot/freeze we will install x64bit version. Virtual host is also x64bit architecture (Linux KVM).

kejianshi

This may seem like a counter-intuitive solution, but - Is 2.1 bringing you something you need that 2.03 wasn't? You could always use 2.03 until 2.1 gets a patch that makes it stable for you?

Michael Hovnivar

Troubles solved. :)

When virtualization host has heavy I/O load (due other virtual guest), pfsense on IDE virtual controller has troubles and fall into reboot or other unexpected state. After we load VirtIO drivers https://doc.pfsense.org/index.php/VirtIO_Driver_Support, pfsense is happy and we too.
But don't allow all VirtIO drivers! VTNET in our case slown down net traffic after few days rapidly.

Working configs
virtual guest pfsense:
pfSense 2.1-RELEASE-pfSense (amd64)
cat /boot/loader.conf.local
virtio_load="YES"
virtio_pci_load="YES"
#if_vtnet_load="YES"
virtio_balloon_load="YES"
virtio_blk_load="YES"
virtualization host Ubuntu 12.04 64bit:
Linux xxx 3.2.0-54-generic #82-Ubuntu SMP Tue Sep 10 20:08:42 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
qemu 1.0+noroms-0ubuntu14.11