Kernel Panic

jnorell

Just another "me too" .. panics in devd on the carp backup host. Just got them while adding firewall aliases on the primary (no recent changes to VIP addrs). I'll report if tomorrow's snapshot doesn't stop them.

jimp

vito and others with em failing, can you look at the output of this:

sysctl -a | grep 'dev.em.*fail'

And report any non-zero values. Specifically with dev.em.0.mbuf_alloc_fail and dev.em.0.tx_dma_fail

vito

No non zero's

here is the output
dev.em.0.mbuf_alloc_fail: 0
dev.em.0.cluster_alloc_fail: 0
dev.em.0.tx_dma_fail: 0
dev.em.0.mac_stats.tso_ctx_fail: 0
dev.em.1.mbuf_alloc_fail: 0
dev.em.1.cluster_alloc_fail: 0
dev.em.1.tx_dma_fail: 0
dev.em.1.mac_stats.tso_ctx_fail: 0

LostInIgnorance

Nope, over TCP still did this:

Kernel page fault with the following non-sleepable locks held:
exclusive sleep mutex vr1 (network driver) r = 0 (0xc3640aec) locked @ /usr/pfSensesrc/src/sys/dev/vr/if_vr.c:1675
KDB: stack backtrace:
X_db_sym_numargs(c0c4e35f,d5341a88,c092ce85,68b,0,...) at X_db_sym_numargs+0x146
kdb_backtrace(68b,0,ffffffff,c11b7554,d5341ac0,...) at kdb_backtrace+0x29
witness_display_spinlock(c0c50877,d5341ad4,4,1,0,...) at witness_display_spinlock+0x75
witness_warn(5,0,c0c823d7,c1981a94,c3556aa0,...) at witness_warn+0x20d
trap(d5341b60) at trap+0x172
alltraps(c39bf700,dedeadc0,c39bf700,c39bf700,d5341be8,...) at alltraps+0x1b
m_tag_delete_chain(c39bf700,0,c092cc2b,0,0,...) at m_tag_delete_chain+0x3f
m_pkthdr_init(c39bf700,100,0,c092cc2b,c0c3acf7,...) at m_pkthdr_init+0x8b5
uma_zfree_arg(c1981a80,c39bf700,0,c3640000,d5341c70,...) at uma_zfree_arg+0x29
m_freem(c39bf700,4,c0c3acf7,5a3,0,...) at m_freem+0x43
ucom_attach(c3640aec,0,c0c3acf7,68b,c3640aec,...) at ucom_attach+0x88f5
ucom_attach(c3640000,d5341cc8,c08d8a54,c107b5c0,c3554238,...) at ucom_attach+0xaa17
intr_event_execute_handlers(c3556aa0,c3554200,c0c4616c,533,c3554270,...) at intr_event_execute_handlers+0x125
intr_event_add_handler(c3644b60,d5341d38,c0c45ecc,344,c3556aa0,...) at intr_event_add_handler+0x42f
fork_exit(c08c1b70,c3644b60,d5341d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xd5341d70, ebp = 0 ---

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0xdedeadc0
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc094b038
stack pointer           = 0x28:0xd5341ba0
frame pointer           = 0x28:0xd5341bb0
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 11 (irq5: vr1)
[thread]
Stopped at      m_tag_delete+0x48:      movl    0(%ecx),%eax
db> bt
Tracing pid 11 tid 64025 td 0xc358d780
m_tag_delete(c39bf700,dedeadc0,c39bf700,c39bf700,d5341be8,...) at m_tag_delete+0x48
m_tag_delete_chain(c39bf700,0,c092cc2b,0,0,...) at m_tag_delete_chain+0x3f
m_pkthdr_init(c39bf700,100,0,c092cc2b,c0c3acf7,...) at m_pkthdr_init+0x8b5
uma_zfree_arg(c1981a80,c39bf700,0,c3640000,d5341c70,...) at uma_zfree_arg+0x29
m_freem(c39bf700,4,c0c3acf7,5a3,0,...) at m_freem+0x43
ucom_attach(c3640aec,0,c0c3acf7,68b,c3640aec,...) at ucom_attach+0x88f5
ucom_attach(c3640000,d5341cc8,c08d8a54,c107b5c0,c3554238,...) at ucom_attach+0xaa17
intr_event_execute_handlers(c3556aa0,c3554200,c0c4616c,533,c3554270,...) at intr_event_execute_handlers+0x125
intr_event_add_handler(c3644b60,d5341d38,c0c45ecc,344,c3556aa0,...) at intr_event_add_handler+0x42f
fork_exit(c08c1b70,c3644b60,d5341d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xd5341d70, ebp = 0 ---
db>
[/thread]

jimp

It was worth trying/checking…

eri--

Can anyone of you please test with this kernel http://files.pfsense.org/kernel.gz
Just copy it under /boot/kernel/kernel.gz on pfsense and reboot.

LostInIgnorance

@jimp:

For those wanting to debug on ALIX/other embedded devices…
/etc/rc.conf_mount_rw
fetch http://pingle.org/files/kernel_wrap_Dev.gz
tar xzpf kernel_wrap_Dev.gz -C /boot/
And then reboot. It works on my ALIX.

The next snapshot after the one building now should have them in there as well, but not the one building now.

Should I use the above process, just changing the file to http://files.pfsense.org/kernel.gz?

jimp

That kernel is probably for full installs only though, not embedded.

LostInIgnorance

so, not for me? Just for the old dell p4 with the built in em card and everyone else with gig cards.

jimp

Yeah, for the em and not for the Soekris.

eri--

Please try with tomorrows snapshots to see if it is fixed.

vito

@ermal:

Can anyone of you please test with this kernel http://files.pfsense.org/kernel.gz
Just copy it under /boot/kernel/kernel.gz on pfsense and reboot.

ermal,
I installed the kernel and it seems to work.
I copied about 2gb of data with no problems.

vito

may have spoke to soon,
No panic but firewall threw this error and lost all connects.
hard to hard reset the box

pf_test: pf_get_mtag returned null

Slaygon

Hi,

My setup consists of two boxes, both having quad intel nic's ("card=0x10a48086 chip=0x10a48086 rev=0x06" from pciconf -lvb).
Both of the boxes run 2.0-BETA5 (amd64) built on Wed Jan 26 01:53:43 EST 2011.

I have had three different types of errors occuring on these;
1 - Box spontaneously reboots
2 - Fatal trap 12 type error
3 - Box freezes

I only have the OpenOSPFD and Postfix Forwarder packages installed. No VLANs. Current uptime is 50 and 33 minutes. I am expecting one of these boxes to do #1, 2 or 3 above within the next 10 or so minutes. They've been doing that since the upgrade.
(while writing this post, one of them did indeed die by freezing)

Prior to upgrading these two, I had no panics, though I did have other errors, such as connections being reset, which was what prompted me to upgrade.

Output of pciconf -lvb:

em0@pci0:3:0:0: class=0x020000 card=0x10a48086 chip=0x10a48086 rev=0x06 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xec120000, size 131072, enabled
bar [14] = type Memory, range 32, base 0xec100000, size 131072, enabled
bar [18] = type I/O Port, range 32, base 0x2000, size 32, enabled
em1@pci0:3:0:1: class=0x020000 card=0x10a48086 chip=0x10a48086 rev=0x06 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xec160000, size 131072, enabled
bar [14] = type Memory, range 32, base 0xec140000, size 131072, enabled
bar [18] = type I/O Port, range 32, base 0x2020, size 32, enabled
em2@pci0:4:0:0: class=0x020000 card=0x10a48086 chip=0x10a48086 rev=0x06 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xec220000, size 131072, enabled
bar [14] = type Memory, range 32, base 0xec200000, size 131072, enabled
bar [18] = type I/O Port, range 32, base 0x3000, size 32, enabled
em3@pci0:4:0:1: class=0x020000 card=0x10a48086 chip=0x10a48086 rev=0x06 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xec260000, size 131072, enabled
bar [14] = type Memory, range 32, base 0xec240000, size 131072, enabled
bar [18] = type I/O Port, range 32, base 0x3020, size 32, enabled
bge0@pci0:14:0:0: class=0x020000 card=0x7051103c chip=0x165a14e4 rev=0x00 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 64, base 0xed000000, size 65536, enabled

The trap 12 error is as follows:

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 0
fault virtual address = 0xa8
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff801fe02f
stack pointer = 0x20:0xffffff8058b56b10
frame pointer = 0x20:0xffffff8058b56b28
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 8 (pfpurge)

jimp

@Slaygon:

current process = 8 (pfpurge)

Slaygon

@jimp:

@Slaygon:

current process = 8 (pfpurge)

toomeek

Hello,

I've just upgraded pfSense on my virtual machine to version:
2.0-BETA5 (i386) built on Tue Jan 25 22:26:25 EST 2011
and first time - kernel panic happend - never seen before.
Updated from.. I don't know earilier version, but from January.
Screenshot attached.

UPDATE: just few hours after reboot, it hanged again without kernel error messages :(

pfSense_last_snapshot_kernel_panic.png_thumb

vito

tested again to be sure and the error did happen again.
Otherwise, i was connected much longer and is looking promising.

pf_test: pf_get_mtag returned null

jimp

Can you type "bt" at that db prompt and post the backtrace output?

clarknova

2.0-BETA5 (amd64)
built on Tue Jan 25 07:56:16 EST 2011

In the ultimate of ironies, I installed the dev kernel about 5 hours ago and rebooted. I checked 5 minutes ago and pfsense was in panic. Unfortunately when I tried to save the console output my serial terminal program froze, probably because my nfs share had disappeared.

Fortunately, within 2 minutes of having pfsense back up again, and while composing this post, it panicked again. I've attached two files for your reading pleasure, but I'm back to the SMP kernel for now.

panic.txt
backtrace.txt