Kernel Panic

jimp · Feb 14, 2011, 12:33 PM

Make sure both of your firewalls are on the latest snapshot and try again. Those panics should be cleared up.

disa · Feb 14, 2011, 2:30 PM

Upgraded the slave to 2.0-RC1 (amd64) built on Sun Feb 13 23:53:14 EST 2011.
Crashed after installing autobackup package on master (sync started and slave crashed): I'm really scared to upgrade the master node to the EC1 as I've already put the cluster in production…is the commercial support going to support me even if I'm running on 2.0? thanks

jimp · Feb 14, 2011, 2:35 PM

Yes, commercial support will assist you even if it's 2.0. Your panic is different than the previous posted in this thread though.

disa · Feb 14, 2011, 3:12 PM

Ok,
I'll try to upgrade the master node in two hours. Thanks

eri-- · Feb 14, 2011, 6:59 PM

Can you describe please your setup?

EDIT: I put even more safety belts so your carp panic should not happen on new snapshots.

disa · Feb 14, 2011, 10:31 PM

Hi, I'll open a ticket tomorrow with all the details on my setup. By the way, I have the 2 nodes running 2.0-RC1 (amd64) built on Sun Feb 13 23:53:14 EST 2011 for a few hours without problems. I didn't work on any new carp ips, only on openvpn: the master was synced to the slave many times without problems. I'll try to play more with carp tomorrow and let you know :-)
thanks

disa · Feb 15, 2011, 4:34 PM

(Ticket #KZZ-134399 opened)
I have the cluster in production for 1 day without any problem right now.
Here is one strange thing I've seen in the logs (on the slave) the other day, what happened?

Feb 13 22:14:50 kernel: vip7: link state changed to DOWN
Feb 13 22:14:50 kernel: vip7: MASTER -> BACKUP (more frequent advertisement received)
Feb 13 22:14:50 kernel: vip8: link state changed to DOWN
Feb 13 22:14:50 kernel: vip9: link state changed to DOWN
Feb 13 22:14:50 kernel: nt received)
Feb 13 22:14:50 kernel:
Feb 13 22:14:50 kernel: e<5m>Ne
Feb 13 22:14:50 kernel: <d6o>Wis
Feb 13 22:14:50 kernel: e<5r>to
Feb 13 22:14:50 kernel: d< 6>tv
Feb 13 22:14:50 kernel: <5a>dge
Feb 13 22:14:50 kernel: h<6a>nt
Feb 13 22:14:50 kernel: q<u5>en c
Feb 13 22:14:50 kernel: t<a6t>ere
Feb 13 22:14:50 kernel: e< 5f>s
Feb 13 22:14:50 kernel: <k6>m or
Feb 13 22:14:50 kernel: P<5 >(n
Feb 13 22:14:50 kernel: :< 6l>iU
Feb 13 22:14:50 kernel: B<a5c>K1
Feb 13 22:14:50 kernel: i<6p>
Feb 13 22:14:50 kernel: -<5>>v
Feb 13 22:14:50 kernel: vip8: MASTER
Feb 13 22:14:50 kernel: vip10: link state changed to DOWN
Feb 13 22:14:50 kernel: vip11: link state changed to DOWN
Feb 13 22:14:50 kernel: vip9: MASTER -> BACKUP (more frequent advertisement received)
Feb 13 22:14:50 kernel: vip1: MASTER -> BACKUP (more frequent advertisement received)
Feb 13 22:14:50 kernel: : MASTER -> BACKUP (more frequent advertisement received)
Feb 13 22:14:50 kernel:
Feb 13 22:14:50 kernel: p<150>N
Feb 13 22:14:50 kernel: o <d6o>Wvi
Feb 13 22:14:50 kernel: vip12: link state changed t
Feb 13 22:14:50 kernel: vip11: MASTER -> BACKUP (more frequent advertisement received)
Feb 13 22:14:50 kernel: vip12: MASTER -> BACKUP (more frequent advertisement received)
Feb 13 22:14:49 dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Feb 13 22:14:49 dhcpd: All rights reserved.
Feb 13 22:14:49 dhcpd: Copyright 2004-2010 Internet Systems Consortium.
Feb 13 22:14:49 dhcpd: Internet Systems Consortium DHCP Server 4.1.1-P1</d6o></a5c></k6></a6t></u5></d6o>

jimp · Feb 15, 2011, 4:40 PM

The next snap (building now) should have more carp panic fixes.

Not sure about the crazy kernel message there, though it looks like maybe it was two messages overlapping, but a bit more corrupt than that usually is.

LostInIgnorance · Feb 15, 2011, 5:42 PM

I would like to inform that since its updating on Sunday (when the snapshot server was available again), the Dell P4 with the integrated em gigE, I have not recieved any panics and the machine is running well. Thank you all for all the hard work fixing the issue! :D

spacelui · Feb 15, 2011, 9:47 PM

Hi,

Like cyber7, I'm getting kernel panics (pf_state_tree_id_RB_REMOVE_COLOR, pfpurge is always the current process) since two weeks (approx. 2 to 3 times a week) on a VMWare setup. Here's a screenshot :

jimp · Feb 15, 2011, 9:57 PM

spacelui,

amd64 or i386? snapshot date? What type of setup, CARP? Multi-WAN? anything special going on with FTP/PPTP? Any more detail might help. Posting the same panic that someone else had doesn't help much (unless you also have the bt output to go with it) but if we can track down what about your setup might be related to the panics, that would be more helpful. What we need is to find some commonality between the people still getting them.

cyber7 · Feb 16, 2011, 4:49 AM

Hi Guys.
I am still getting the same panic, just this time with a cmpl error. I am busy updating to the new snapshot, but why is it that after months of running fine, I am starting to see this panic?

Kind regards
Aubrey Kloppers
ps - I am updating to the following snapshot:

Auto Update Download Status
–--------------------------------------------------
Current Version : 2.0-RC1
Latest Version : Tue Feb 15 16:36:07 EST 2011

jimp · Feb 16, 2011, 5:15 AM

If it's a different error, it's not the same panic. We need the full text of the panic and the backtrace (bt at the db> prompt) in order to say anything.

But don't report anything until you've replicated it on the new snapshot you're updating to.

cyber7 · Feb 16, 2011, 5:30 AM

Thank you Jim.
ps - It was nice to chat to you yesterday. Do you not ever sleep :)

Kind regards
Aubrey Kloppers

spacelui · Feb 16, 2011, 8:21 AM

Sorry, I cut the details…
It's a 2.0-RC1 of yesterday running on an i386.

No multiwan, no carp and I have indeed some incoming pptp traffic going to a nated server on the lan. VMWare ESX setup. I do an upgrade every week, it began to happen on the 28th of january. I tried to look in the repo to see commits between 21st and 28th, to see if something relevant has changed, but a lot of things happened... plus, I'm not sure that couldn't happen before.

Next time it happens, I'll post the backtrace.

Edit : I reverted to the last backup I had before upgrading on the 28th, it's a Fri Jan 7 15:25:33 EST 2011 snapshot…

cyber7 · Feb 17, 2011, 11:13 AM

Hi Guys

look at my panic: http://forum.pfsense.org/index.php/topic,33403.0.html
and see if this does not solve your problems

Kind regards
Aubrey

acherman · Feb 17, 2011, 2:43 PM

Hey Aubrey, changing my mtu's didn't fix anything. In fact it caused a panic on both boxes when trying to apply the changes. I don't think I will try that again.

jimp · Feb 17, 2011, 2:53 PM

@acherman:

Hey Aubrey, changing my mtu's didn't fix anything. In fact it caused a panic on both boxes when trying to apply the changes. I don't think I will try that again.

Did you get the panic/crash info when it did that? Also, are you on i386 or amd64? What snapshot date?

acherman · Feb 17, 2011, 2:58 PM

No, I didn't. Sorry. But I'm sure it will happen again soon and I have the camera ready. I am running i386 and build Mon Jan 31 07:16:37 EST 2011 right now (saw the same issues with Mon Feb 14 02:12:45 EST 2011 and I can't remember which one from the 15th).

jimp · Feb 17, 2011, 3:03 PM

Update to the most current snap ASAP and then test again. Testing on old snaps isn't likely to provide any useful feedback for this. There were patches after those dates to help prevent panics in other situations.