Kernel Panic

LostInIgnorance · Jan 24, 2011, 6:46 PM

So that should fix the gig card issue, but what can I do to get the full backtrace on my embedded Soekris board?

jimp · Jan 24, 2011, 6:48 PM

We don't have an embedded debug kernel, not sure how hard it might be to make one. If it doesn't work properly with the full install's debug kernel, it may be a pain to do.

vito · Jan 24, 2011, 6:50 PM

Jimp,
I will let you know after the next full update snap.
from another post, are the snaps building fine now? (or the server, what ever the problem was)

Thanks again!!

jimp · Jan 24, 2011, 6:53 PM

@vito:

I will let you know after the next full update snap.
from another post, are the snaps building fine now? (or the server, what ever the problem was)

The build won't be restarted for a couple more hours (waiting on patches for other issues to go in, too) so it probably won't be uploaded until tomorrow AM.

The snapshots were building fine all weekend, but at one point the snapshot web server (where they are copied after being built) ran out of space. It's OK now.

LostInIgnorance · Jan 24, 2011, 6:58 PM

Yeah I tried the full install from the cd on the hd, but the board wouldn't boot. Had to stick with the embedded build.

clarknova · Jan 24, 2011, 7:00 PM

Soekris won't boot from USB. You have to put your boot media in another machine to do the install. Once you have the drive back in the soekris you will probably have to manually enter your rootfs device at the prompt, then update /etc/fstab.

LostInIgnorance · Jan 24, 2011, 7:02 PM

that is exactly what I did, using vm and an external usb-sata adapter. Just wish I could get the developer to boot on the board. I should have specified I used a usb dvd since I don't have one on my netbook.

clarknova · Jan 24, 2011, 7:07 PM

I have installed 1.2.3 and 2.0 on a net5501-70 a few times, never had a problem other than changing the root device. Post specifics if you want help troubleshooting it.

Have you tried installing the full developer kernel on the embedded install?

LostInIgnorance · Jan 24, 2011, 7:26 PM

last time I tried to do what PJ2 outlined at the begining of this thread, it locked my system up and wouldn't boot. Help is always greatly appreciated if I can get it installed and get the panic.

EDIT: Would actually prefer having the developer kernel on this embedded device. I am using the 5501-70; sata wd 80g; running full embedded; vr0-wan, vr1-lan, vr2-wrls (friendly wifi for visitors), vr3-dmz; packages squid, lightsquid, nut, havp, nmap, snort; anything else?

jimp · Jan 24, 2011, 9:35 PM

I just added support to the builder to make an embedded kernel with debug options. I've got a test build going on my box, if one cranks out I'll upload it somewhere this evening or tomorrow. Failing that, the main snapshots should include it from here on. Not the next snapshot, but the one after it, should have them.

LostInIgnorance · Jan 24, 2011, 9:37 PM

OMG, THANKS JIMP!!! I am excited to know if it works!

jimp · Jan 24, 2011, 11:15 PM

For those wanting to debug on ALIX/other embedded devices…

/etc/rc.conf_mount_rw
fetch http://pingle.org/files/kernel_wrap_Dev.gz
tar xzpf kernel_wrap_Dev.gz -C /boot/

And then reboot. It works on my ALIX.

The next snapshot after the one building now should have them in there as well, but not the one building now.

LostInIgnorance · Jan 25, 2011, 1:10 AM

JimP, the above mentioned code worked great on my Soekris net-5501-70 board. Thanks for that, I hope to drive down the road and use my neighbors wifi to see if I can crash it now.

LostInIgnorance · Jan 25, 2011, 2:46 AM

Grabbed the panic from the above mentioned Soekris board.

Kernel page fault with the following non-sleepable locks held:
exclusive sleep mutex vr1 (network driver) r = 0 (0xc3640aec) locked @ /usr/pfSensesrc/src/sys/dev/vr/if_vr.c:1675
KDB: stack backtrace:
X_db_sym_numargs(c0c4e35f,d5341a88,c092ce85,68b,0,...) at X_db_sym_numargs+0x146
kdb_backtrace(68b,0,ffffffff,c11b73f4,d5341ac0,...) at kdb_backtrace+0x29
witness_display_spinlock(c0c50877,d5341ad4,4,1,0,...) at witness_display_spinlock+0x75
witness_warn(5,0,c0c823d7,c1981a94,c3556aa0,...) at witness_warn+0x20d
trap(d5341b60) at trap+0x172
alltraps(c3899300,dedeadc0,c3899300,c3899300,d5341be8,...) at alltraps+0x1b
m_tag_delete_chain(c3899300,0,c092cc2b,0,0,...) at m_tag_delete_chain+0x3f
m_pkthdr_init(c3899300,100,0,c092cc2b,c0c3acf7,...) at m_pkthdr_init+0x8b5
uma_zfree_arg(c1981a80,c3899300,0,c3640000,d5341c70,...) at uma_zfree_arg+0x29
m_freem(c3899300,4,c0c3acf7,5a3,0,...) at m_freem+0x43
ucom_attach(c3640aec,0,c0c3acf7,68b,c3640aec,...) at ucom_attach+0x88f5
ucom_attach(c3640000,d5341cc8,c08d8a54,c107b5c0,c3554238,...) at ucom_attach+0xaa17
intr_event_execute_handlers(c3556aa0,c3554200,c0c4616c,533,c3554270,...) at intr_event_execute_handlers+0x125
intr_event_add_handler(c3644b60,d5341d38,c0c45ecc,344,c3556aa0,...) at intr_event_add_handler+0x42f
fork_exit(c08c1b70,c3644b60,d5341d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xd5341d70, ebp = 0 ---

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address= 0xdedeadc0
fault code= supervisor read, page not present
instruction pointer= 0x20:0xc094b038
stack pointer        = 0x28:0xd5341ba0
frame pointer        = 0x28:0xd5341bb0
code segment= base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process= 11 (irq5: vr1)
[thread]
Stopped at      m_tag_delete+0x48:      movl    0(%ecx),%eax
db> bt
Tracing pid 11 tid 64025 td 0xc358d780
m_tag_delete(c3899300,dedeadc0,c3899300,c3899300,d5341be8,...) at m_tag_delete+0x48
m_tag_delete_chain(c3899300,0,c092cc2b,0,0,...) at m_tag_delete_chain+0x3f
m_pkthdr_init(c3899300,100,0,c092cc2b,c0c3acf7,...) at m_pkthdr_init+0x8b5
uma_zfree_arg(c1981a80,c3899300,0,c3640000,d5341c70,...) at uma_zfree_arg+0x29
m_freem(c3899300,4,c0c3acf7,5a3,0,...) at m_freem+0x43
ucom_attach(c3640aec,0,c0c3acf7,68b,c3640aec,...) at ucom_attach+0x88f5
ucom_attach(c3640000,d5341cc8,c08d8a54,c107b5c0,c3554238,...) at ucom_attach+0xaa17
intr_event_execute_handlers(c3556aa0,c3554200,c0c4616c,533,c3554270,...) at intr_event_execute_handlers+0x125
intr_event_add_handler(c3644b60,d5341d38,c0c45ecc,344,c3556aa0,...) at intr_event_add_handler+0x42f
fork_exit(c08c1b70,c3644b60,d5341d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xd5341d70, ebp = 0 ---
db> 

EDIT: Deleted crap from HAVP to reflect just panic[/thread]

jimp · Jan 25, 2011, 1:54 AM

Is vr1 bridged to anything?

LostInIgnorance · Jan 25, 2011, 1:58 AM

not that I am aware of, I do force all traffic through the OpenVPN interface (Force all client generated traffic through the tunnel)

LostInIgnorance · Jan 25, 2011, 2:30 AM

I did notice this on the console after having to uninstall and reinstall HAVP.

lock order reversal:
 1st 0xc4279df4 ufs (ufs) @ /usr/pfSensesrc/src/sys/kern/vfs_mount.c:1204
 2nd 0xc474e6a0 syncer (syncer) @ /usr/pfSensesrc/src/sys/kern/vfs_subr.c:2203
KDB: stack backtrace:
X_db_sym_numargs(c0c4e35f,d6704a3c,c092ce85,c091d9bb,c0c512cb,...) at X_db_sym_numargs+0x146
kdb_backtrace(c091d9bb,c0c512cb,c3516ee8,c3517020,d6704a98,...) at kdb_backtrace+0x29
witness_display_spinlock(c0c512cb,c474e6a0,c0c5872e,c3517020,c0c585a4,...) at witness_display_spinlock+0x75
witness_checkorder(c474e6a0,9,c0c585a4,89b,0,...) at witness_checkorder+0x839
__lockmgr_args(c474e6a0,80100,c474e6bc,0,0,...) at __lockmgr_args+0x7f5
vop_stdlock(d6704bb4,3,c0c585a4,80100,c474e648,...) at vop_stdlock+0x62
VOP_LOCK1_APV(c1032b00,d6704bb4,c08d9223,c10560a0,c474e648,...) at VOP_LOCK1_APV+0xb5
_vn_lock(c474e648,80100,c0c585a4,89b,0,...) at _vn_lock+0x5e
insmntque(d6704c58,c097342e,c474e648,0,c0c57db8,...) at insmntque+0x288
vrele(c474e648,0,c0c57db8,4f9,80,...) at vrele+0x10
dounmount(c37b2000,8080000,c3b3a780,47e,fdf65b4a,...) at dounmount+0x3ce
unmount(c3b3a780,d6704cf8,c3b3a780,d6704d2c,206,...) at unmount+0x2bf
syscall(d6704d38) at syscall+0x1da
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (22, FreeBSD ELF32, unmount), eip = 0x280dfa9f, esp = 0xbfbfe61c, ebp = 0xbfbfe6e8 ---

jimp · Jan 25, 2011, 2:44 AM

Don't worry about LORs, they're mostly harmless.

LostInIgnorance · Jan 25, 2011, 2:51 AM

Did notice this from both panics though.

Panic from old P4 computer:

Kernel page fault with the following non-sleepable locks held:
exclusive sleep mutex em0 (EM TX Lock) r = 0 (0xc2f52580) locked @ /usr/pfSensesrc/src/sys/dev/e1000/if_lem.c:1350

Panic from Soekris board:

Kernel page fault with the following non-sleepable locks held:
exclusive sleep mutex vr1 (network driver) r = 0 (0xc3640aec) locked @ /usr/pfSensesrc/src/sys/dev/vr/if_vr.c:1675

Is it just coincidence?

disa · Jan 25, 2011, 8:31 AM

Hi, I've reinstalled the secondary machine, it's now running 2.0-BETA5 (amd64) built on Mon Jan 17 22:14:04 EST 2011 (the primary has 2.0-BETA5 (amd64) built on Fri Jan 21 23:51:34 EST 2011).

I disabled the sync, created all the remaining carp vips on the primary (I've 12 of them right now), and re-enabled the sync. The secondary didn't crash.

What shall I do now? I'm a bit scared of upgrading it :-) thanks