Kernel panic 4-5 Nov (i386)

FisherKing

Yeah - thanks for looking into this ermal. It's hard to to much other testing when we can't do anything w/ the GUI. :)

Let us know if you need any other information. I can probably arrange remote access to the box if you can't reproduce this on your hardware.

mbartosch

Hi,

unhappy camper as well. Using the snapshot from 2010-11-18 I tried to configure an interface group, after this I got trapped in a kernel trap - reboot loop.

Running on a Soekris 5501.

FreeBSD/i386 bootstrap loader, Revision 1.1
(sullrich@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org, Thu Nov 18 07:57:55 EST 20)
…
vlan7: changing name to 'lagg0_vlan2920'
vlan8: changing name to 'lagg0_vlan2921'
vlan9: changing name to 'lagg0_vlan2930'
vlan10: changing name to 'lagg0_vlan2931'

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x14
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc095d504
stack pointer = 0x28:0xc33fbb48
frame pointer = 0x28:0xc33fbb78
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 11 (swi4: clock)
trap number = 12
panic: page fault
cpuid = 0
Uptime: 29s

eri--

Give a shot at tomorrows snapshots.

FisherKing

Wahoo! Thanks ermal!

bEsTiAn

coming from my own thread :
http://forum.pfsense.org/index.php/topic,30207.msg157592.html
Unfortunately, using latest available snaphost, the crash repeats, after 1 or 2 minutes uptime

FisherKing

Yeah - I installed the 2010-11-25 update and experienced the same thing.

Ermal - do you need a backtrace from the current snapshot, or is what you have already sufficient?

mbartosch

Hi,

doesn't look good yet, using the snapshot from yesterday (sullrich@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org, Thu Nov 25 10:39:16 EST 20).

I just created the first LAGG interface from vr2 and vr3 on my Soekris 5501. Immediately after clicking the apply button in the web interface:

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0xc
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc09bbc2d
stack pointer = 0x28:0xe5dbec04
frame pointer = 0x28:0xe5dbec24
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 11 (irq9: vr2)
trap number = 12
panic: page fault
cpuid = 0
Uptime: 17h58m38s
Cannot dump. Device not defined or unavailable.

eri--

Hmmm sorry i thought i had pushed the patch to repo!
Just did though try with next snaps.

jlepthien

Is this probably the same issue that hits me here: http://forum.pfsense.org/index.php/topic,30465.0.html

mbartosch

Hi,

@ermal:

Hmmm sorry i thought i had pushed the patch to repo!
Just did though try with next snaps.

just to give a quick head-up: I have tried the latest snapshot and it seems to have fixed the problem. No kernel panic yes with extensive VLAN on LAGG interfaces config fiddling (but no actual traffic over these yet).

Thanks for fixing this,

Martin

bEsTiAn

same here
uptime 2 hours and increasing :)

thanks a lot once more for the hard and continuous work

jlepthien

Still not possible for me to add my second wireless interface…

See http://forum.pfsense.org/index.php/topic,30465.0.html

Still rebooting then...

eri--

Go over to your thread jlepthien do not hijack this one.

jlepthien

Well, doesn't this look kinda the same problem? And since you are the one fixing this I thought you might be interesting. If you do not want the info on bugs I'll happily install 1.2.3 again and leave the testing up to you…

CryoGenID

Unfortunately, I am still fighting with the kernel panic :'( :'( :'( :'(

I had hopefully upgraded to "Sat Nov 27 03:13:22 EST 2010 "-Build via USB-Stick, which worked pretty fine.
Today, I tried to use the built-in update feature and… BANG! Kernel-Panic (Fatal Trap 12), see attached screenshot...
(the screenshot is from an older build, but the error is still the same)

ermal, are you still fighting this problem? :-)

Hopefully there will be a solution soon :)

Best regards,

Christian

pfSense_ERROR.jpg_thumb

eri--

without a backtrace i cannot say that this is related to this.

FisherKing

I tested both the 27th and 29th snapshots, doing an upgrade to the 27th and then a clean install for the 29th. Kernel panic still pops up for me under the same conditions as before. Oddly, the panic doesn't happen until after the update / install has been in place for a while (1-4 hours?) If I install an update, both GUI and fetch and will work for a bit, then a couple hours later, I run fetch from the console or log into the GUI again and it bonks almost immediately.

I'll get a current backtrace for you in the next hour or two.

FisherKing

Ok - here's the panic and back trace from the 11/29 build.


FreeBSD/i386 (xxxxxx.xxx) (ttyu0)

login: root
Password:
# fetch http://snapshots.pfsense.org/FreeBSD_RELENG_8_1/i386/pfSense_HEAD/update
s/pfSense-Full-Update-2.0-BETA4-20101129-2231.tgz
Kernel page fault with the following non-sleepable locks held:
exclusive sleep mutex fxp0 (network driver) r = 0 (0xc36de018) locked @ /usr/pfS
ensesrc/src/sys/kern/kern_mutex.c:147
KDB: stack backtrace:
X_db_sym_numargs(c0ea7a7e,c3304a4c,c0a33fb5,93,0,...) at X_db_sym_numargs+0x146
kdb_backtrace(93,0,ffffffff,c144e8a4,c3304a84,...) at kdb_backtrace+0x29
witness_display_spinlock(c0ea9f96,c3304a98,4,1,0,...) at witness_display_spinloc
k+0x75
witness_warn(5,0,c0ee832e,1,c35ab7f8,...) at witness_warn+0x20d
trap(c3304b24) at trap+0x19e
alltraps(c36e6100,dedeadc0,c36e6100,c36e6100,c3304bac,...) at alltraps+0x1b
m_tag_delete_chain(c36e6100,0,df,0,c36de000,...) at m_tag_delete_chain+0x3f
reallocf(c36e6100,100,0,9e3,c0ea29ae,...) at reallocf+0x8a5
uma_zfree_arg(c1d7e380,c36e6100,0,c36df1f0,c3304c20,...) at uma_zfree_arg+0x29
m_freem(c36e6100,c36e4d40,8,c36de000,c36cc800,...) at m_freem+0x43
fwohci_init(c36de018,4,c0e63eff,82a,c36de018,...) at fwohci_init+0x545c
fwohci_init(c36de000,1,c0ea5a5d,189,c130de98,...) at fwohci_init+0x7a25
softclock(c130de60,c3304cc8,c09dfff4,c1311c00,c35d45b8,...) at softclock+0x24a
intr_event_execute_handlers(c35ab7f8,c35d4580,c0e9f7b8,533,c35d45f0,...) at intr
_event_execute_handlers+0x125
intr_event_add_handler(c35aa110,c3304d38,c0e9f51d,344,c35ab7f8,...) at intr_even
t_add_handler+0x42f
fork_exit(c09c8da0,c35aa110,c3304d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3304d70, ebp = 0 ---

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0xdedeadc0
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc0a53258
stack pointer           = 0x28:0xc3304b64
frame pointer           = 0x28:0xc3304b74
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 12 (swi4: clock)
[thread]
Stopped at      m_tag_delete+0x48:      movl    0(%ecx),%eax
db> lock order reversal: (Giant after non-sleepable)
 1st 0xc36de018 fxp0 (network driver) @ /usr/pfSensesrc/src/sys/kern/kern_mutex.
c:147
 2nd 0xc130d110 Giant (Giant) @ /usr/pfSensesrc/src/sys/dev/usb/input/ukbd.c:170
4
KDB: stack backtrace:
X_db_sym_numargs(c0ea7a7e,c330489c,c0a33fb5,c0a24b9b,c0eaa9ea,...) at X_db_sym_n
umargs+0x146
kdb_backtrace(c0a24b9b,c0eaa9ea,c355f040,c355e1a0,c33048f8,...) at kdb_backtrace
+0x29
witness_display_spinlock(c0eaa9ea,c130d110,c0eceae9,c355e1a0,c0e93752,...) at wi
tness_display_spinlock+0x75
witness_checkorder(c130d110,9,c0e93752,6a8,0,...) at witness_checkorder+0x839
_mtx_lock_flags(c130d110,0,c0e93752,6a8,c392ea80,...) at _mtx_lock_flags+0xc4
ucom_attach(c3c4b000,1,c130bd68,c1309f20,c3304978,...) at ucom_attach+0x1c08
ixgbe_init_fdir_perfect_82599(c35b2c00,1,c1309fa4,c130bd68,1,...) at ixgbe_init_
fdir_perfect_82599+0x6456
sc_attach_unit(c122daa0,78,c3304990,c09b0166,c33049b0,...) at sc_attach_unit+0x5
23
cncheckc(c33049b0,c0526fe5,c0e3bd0d,c0528290,c33049ac,...) at cncheckc+0x48
cngetc(c0e3bd0d,c0528290,c33049ac,c33049e8,1,...) at cngetc+0x16
db_readline(c12dad60,78,c33049cc,c0525c26,c0e3bd0d,...) at db_readline+0x75
db_read_line(c0e3bd0d,c3304a20,c0527add,c0ee41e5,0,...) at db_read_line+0x1a
db_command_loop(c0ee41e5,0,c33049f4,c0e0cf9d,0,...) at db_command_loo3304b24,...
) at 0xed
kdb_trap(c,0,c3304
db>
db> bt
Tracing pid 12 tid 64007 td 0xc35ad280
m_tag_delete(c36e6100,dedeadc0,c36e6100,c36e6100,c3304bac,...) at m_tag_delete+0
x48
m_tag_delete_chain(c36e6100,0,df,0,c36de000,...) at m_tag_delete_chain+0x3f
reallocf(c36e6100,100,0,9e3,c0ea29ae,...) at reallocf+0x8a5
uma_zfree_arg(c1d7e380,c36e6100,0,c36df1f0,c3304c20,...) at uma_zfree_arg+0x29
m_freem(c36e6100,c36e4d40,8,c36de000,c36cc800,...) at m_freem+0x43
fwohci_init(c36de018,4,c0e63eff,82a,c36de018,...) at fwohci_init+0x545c
fwohci_init(c36de000,1,c0ea5a5d,189,c130de98,...) at fwohci_init+0x7a25
softclock(c130de60,c3304cc8,c09dfff4,c1311c00,c35d45b8,...) at softclock+0x24a
intr_event_execute_handlers(c35ab7f8,c35d4580,c0e9f7b8,533,c35d45f0,...) at intr
_event_execute_handlers+0x125
intr_event_add_handler(c35aa110,c3304d38,c0e9f51d,344,c35ab7f8,...) at intr_even
t_add_handler+0x42f
fork_exit(c09c8da0,c35aa110,c3304d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3304d70, ebp = 0 ---
db>

I noticed some KDB: stack backtrace messages in the boot up sequence also, so I've included those as well.

[code]
Configuring firewall.....lock order reversal:
 1st 0xc12d8a0c pf task mtx (pf task mtx) @ /usr/pfSensesrc/src/sys/contrib/pf/n
et/pf_ioctl.c:1397
 2nd 0xc147a864 ifnet_rw (ifnet_rw) @ /usr/pfSensesrc/src/sys/net/if.c:2055
KDB: stack backtrace:
X_db_sym_numargs(c0ea7a7e,d43de9c0,c0a33fb5,c0a24b9b,c0eaa9ea,...) at X_db_sym_n
umargs+0x146
kdb_backtrace(c0a24b9b,c0eaa9ea,c3565638,c35627e8,d43dea1c,...) at kdb_backtrace
+0x29
witness_display_spinlock(c0eaa9ea,c147a864,c0eb3e25,c35627e8,c0eb3e04,...) at wi
tness_display_spinlock+0x75
witness_checkorder(c147a864,1,c0eb3e04,807,0,...) at witness_checkorder+0x839
_rw_rlock(c147a864,c0eb3e04,807,c392e930,3,...) at _rw_rlock+0x9c
ifunit(c392e930,0,c0e3ab59,575,c0eb2f6d,...) at ifunit+0x27
pfioctl(c3923500,c0104414,c392e930,3,c3c37780,...) at pfioctl+0x2724
dev2udev(c3c62888,c0104414,c392e930,c35a8400,c3c37780,...) at dev2udev+0x898
kern_ioctl(c3c37780,8,c0104414,c392e930,1a2d860,...) at kern_ioctl+0x1fd
ioctl(c3c37780,d43decf8,c0ee8261,c0ec0326,c3c3ad48,...) at ioctl+0x134
syscall(d43ded38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (54, FreeBSD ELF32, ioctl), eip = 0x281e0533, esp = 0xbfbfdd9c, ebp
= 0xbfbfddd8 ---
.done.
Starting PFLOG...done.
Setting up gateway monitors...No matching processes were found
done.
Synchronizing user settings...lock order reversal:
 1st 0xcc04c260 bufwait (bufwait) @ /usr/pfSensesrc/src/sys/kern/vfs_bio.c:2559
 2nd 0xc3c27000 dirhash (dirhash) @ /usr/pfSensesrc/src/sys/ufs/ufs/ufs_dirhash.
c:285
KDB: stack backtrace:
X_db_sym_numargs(c0ea7a7e,d44187c0,c0a33fb5,c0a24b9b,c0eaa9ea,...) at X_db_sym_n
umargs+0x146
kdb_backtrace(c0a24b9b,c0eaa9ea,c3561fc8,c3565ec0,d441881c,...) at kdb_backtrace
+0x29
witness_display_spinlock(c0eaa9ea,c3c27000,c0ed4887,c3565ec0,c0ed4515,...) at wi
tness_display_spinlock+0x75
witness_checkorder(c3c27000,9,c0ed4515,11d,0,...) at witness_checkorder+0x839
_sx_xlock(c3c27000,0,c0ed4515,11d,c3c59d24,...) at _sx_xlock+0x85
ufsdirhash_enduseful(cc04c200,d4418944,34,cc1833e8,d44188ec,...) at ufsdirhash_e
nduseful+0x2f5
ufsdirhash_add(c3c59d24,d4418944,3e8,d44188d8,d44188dc,...) at ufsdirhash_add+0x
13
ufs_direnter(c3957648,c3e55c90,d4418944,d4418bd4,0,...) at ufs_direnter+0x729
ufs_itimes(d4418bd4,0,d4418b30,d4418a8c,c0dba8d5,...) at ufs_itimes+0x14bd
ufs_itimes(d4418b30,d4418b48,0,0,d4418ba8,...) at ufs_itimes+0x17b0
VOP_CREATE_APV(c12bd740,d4418b30,d4418bd4,d4418ac8,0,...) at VOP_CREATE_APV+0xa5

vn_open_cred(d4418ba8,d4418c5c,1a4,0,c35a8400,...) at vn_open_cred+0x215
vn_open(d4418ba8,d4418c5c,1a4,c3c63818,281df000,...) at vn_open+0x3b
kern_openat(c3d31280,ffffff9c,bfbfd2f8,0,603,...) at kern_openat+0x125
kern_open(c3d31280,bfbfd2f8,0,602,1a4,...) at kern_open+0x35
open(c3d31280,d4418cf8,c,c3d31280,c3c3cd48,...) at open+0x30
syscall(d4418d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (5, FreeBSD ELF32, open), eip = 0x281a7a83, esp = 0xbfbfd2bc, ebp =
0xbfbfd708 ---
done.
Starting webConfigurator...done.
Configuring CRON...done.
Starting OpenNTP time client...done.
lock order reversal:
 1st 0xc3e5edf4 ufs (ufs) @ /usr/pfSensesrc/src/sys/kern/vfs_mount.c:1058
 2nd 0xc3e848b8 devfs (devfs) @ /usr/pfSensesrc/src/sys/kern/vfs_subr.c:2090
KDB: stack backtrace:
X_db_sym_numargs(c0ea7a7e,d4455800,c0a33fb5,c0a24b9b,c0eaa9ea,...) at X_db_sym_n
umargs+0x146
kdb_backtrace(c0a24b9b,c0eaa9ea,c3565e58,c3565d88,d445585c,...) at kdb_backtrace
+0x29
witness_display_spinlock(c0eaa9ea,c3e848b8,c0e96afc,c3565d88,c0eb1da5,...) at wi
tness_display_spinlock+0x75
witness_checkorder(c3e848b8,9,c0eb1da5,82a,0,...) at witness_checkorder+0x839
__lockmgr_args(c3e848b8,80100,c3e848d4,0,0,...) at __lockmgr_args+0x7f5
vop_stdlock(d4455978,c0a33d5b,c0e96d43,80100,c3e84860,...) at vop_stdlock+0x62
VOP_LOCK1_APV(c1292060,d4455978,c3dea5a4,c12d68e0,c3e84860,...) at VOP_LOCK1_APV
+0xb5
_vn_lock(c3e84860,80100,c0eb1da5,82a,8,...) at _vn_lock+0x5e
vget(c3e84860,80100,c3dea500,15e,c0e96c5a,...) at vget+0xb9
devfs_allocv(c3ddf880,c3c4f284,d4455a10,9d,c147a63c,...) at devfs_allocv+0x102
devfs_rules_apply(c3c4f284,80000,d4455c30,430,0,...) at devfs_rules_apply+0x14a
vfs_donmount(c3dea500,0,c3ddf980,c3ddf980,bfbfde09,...) at vfs_donmount+0x14c2
nmount(c3dea500,d4455cf8,c,c3dea500,c3e817f8,...) at nmount+0x75
syscall(d4455d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (378, FreeBSD ELF32, nmount), eip = 0x280ee5db, esp = 0xbfbfdddc, eb
p = 0xbfbfe338 ---
Starting DHCP service...done.
Starting DNS forwarder...done.
Starting captive portal... load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched PRIO loaded
done
Configuring firewall......done.
Generating RRD graphs...done.
Starting CRON... done.
Executing rc.d items...
 Starting /usr/local/etc/rc.d/*.sh...done.
Bootup complete
[/code]

[/thread]

eri--

Can you please test the new snapshots and see if this is fixed?

FisherKing

Will do. Thanks ermal!

Do you believe the update made it into the current builder run (looks like it started 10:40am EST), or do I need to wait for the build after this?