RESOLVED: Kernel panic w/ Captive Portal enabled (i386 full)

[FisherKing]

This is sort of an extension / out growth of the previous kernel panic that Ermal was working on.
http://forum.pfsense.org/index.php/topic,29839.0.html

Running i386 snapshots.  Currently installed build is Sat Dec 4 01:05:18 EST 2010.
I've got 2 dual port intel NICs (total of 4 ports)
fxp0 - wan (Public IP assigned via PPPoE)
fxp1 - DMZ (Option1 - public subnet)
fxp2 - lan (172.16.xxx.xxx)
fxp3 - guest lan (Option2 - 172.16.yyy.yyy)

When captive portal is enabled (on guest lan / fxp3, all settings at default, or w/ settings configured)
I am able to trigger a panic doing any of the following:

• running "fetch" from the console
• using option 13 (upgrade), then option 1 from the console menu
• installing a new package from System:Package Manager
• FTP from the console to a remote site (Initial ftp login doesn't trigger it, but by the time I logout it does panic)

The following does NOT seem to trigger the panic:

• ping -i 0.001 remote.ip.address
• repeated "dig" commands

It appears to me that it is only TCP traffic over the WAN that triggers the panic, and only after a certain (small) amount has passed.

When I disable CP, the panic no longer happens.

 pfSense console setup
***************************
 0)  Logout (SSH only)
 1)  Assign Interfaces
 2)  Set interface(s) IP address
 3)  Reset webConfigurator password
 4)  Reset to factory defaults
 5)  Reboot system
 6)  Halt system
 7)  Ping host
 8)  Shell
 9)  PFtop
10)  Filter Logs
11)  Restart webConfigurator
12)  pfSense Developer Shell
13)  Upgrade from console
14)  Disable Secure Shell (sshd)

Enter an option: 8

[2.0-BETA4][root@fw1.xxxxxx.org]/root(1): ~/test
pfSense-Full-Update-2.0-BETA4-20101203-2137.tg  0% of   76 MB    0  Bps

Kernel page fault with the following non-sleepable locks held:
exclusive sleep mutex fxp0 (network driver) r = 0 (0xc36de018) locked @ /usr/pfSensesrc/src/sys/dev/fxp/if_fxp.c:1288
KDB: stack backtrace:
X_db_sym_numargs(c0ea7b7e,c3307888,c0a33fd5,508,0,...) at X_db_sym_numargs+0x146

kdb_backtrace(508,0,ffffffff,c144e9a4,c33078c0,...) at kdb_backtrace+0x29
witness_display_spinlock(c0eaa096,c33078d4,4,1,0,...) at witness_display_spinlock+0x75
witness_warn(5,0,c0ee842e,c33078ee,c35ab550,...) at witness_warn+0x20d
trap(c3307960) at trap+0x19e
alltraps(c39a8200,c39a923d,c39a8200,c39a8200,c33079ec,...) at alltraps+0x1b
m_tag_delete_chain(c39a8200,0,df,0,c36de000,...) at m_tag_delete_chain+0x3f
reallocf(c39a8200,100,0,9e3,c0a33d7b,...) at reallocf+0x8a5
uma_zfree_arg(c1d7e380,c39a8200,0,c36df430,c3307a60,...) at uma_zfree_arg+0x29
m_freem(c39a8200,c36e4440,8,c36cc800,c36de018,...) at m_freem+0x43
fwohci_init(c36de018,4,c0e63fff,519,c3307ab4,...) at fwohci_init+0x545c
fwohci_init(c36de018,0,c0e63fff,508,c36cc800,...) at fwohci_init+0x6613
fwohci_init(c36cc800,c3307c0c,c0aa27ef,c36cc800,0,...) at fwohci_init+0x733b
if_start(c36cc800,0,c0eb3f04,d1d,2,...) at if_start+0x12
if_handoff(c36cc800,c4077d00,0,0) at if_handoff+0x25f
ether_output_frame(c36cc800,c4077d00,c0e9a61c,1,c3da4b40,...) at ether_output_frame+0x65
ng_car_q_event(c3da9980,c3da4b40,c0ebd64e,c0e9a61c,3,...) at ng_car_q_event+0x2e2b
ng_rmnode(c391eb50,0,c0ebd64e,d2c,0,...) at ng_rmnode+0x2e4
ng_rmnode(0,c3307d38,c0e9f61d,344,c35ab550,...) at ng_rmnode+0x16a1
fork_exit(c0b0fd90,0,c3307d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3307d70, ebp = 0 ---

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0xdedeadc0
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xdedeadc0
stack pointer           = 0x28:0xc33079a0
frame pointer           = 0x28:0xc33079b4
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 13 (ng_queue0)
[thread]
Stopped at      0xdedeadc0:     *** error reading from address dedeadc0 ***
db> lock order reversal: (Giant after non-sleepable)
 1st 0xc36de018 fxp0 (network driver) @ /usr/pfSensesrc/src/sys/dev/fxp/if_fxp.c:1288
 2nd 0xc130d210 Giant (Giant) @ /usr/pfSensesrc/src/sys/dev/usb/input/ukbd.c:1704
KDB: stack backtrace:
X_db_sym_numargs(c0ea7b7e,c33076d8,c0a33fd5,c0a24bbb,c0eaaaea,...) at X_db_sym_numargs+0x146
kdb_backtrace(c0a24bbb,c0eaaaea,c355f040,c355e1a0,c3307734,...) at kdb_backtrace+0x29
witness_display_spinlock(c0eaaaea,c130d210,c0ecebe9,c355e1a0,c0e93852,...) at witness_display_spinlock+0x75
witness_checkorder(c130d210,9,c0e93852,6a8,0,...) at witness_checkorder+0x839
_mtx_lock_flags(c130d210,0,c0e93852,6a8,c392eb50,...) at _mtx_lock_flags+0xc4
ucom_attach(c3c4c000,1,c130be68,c130a020,c33077b4,...) at ucom_attach+0x1c08
ixgbe_init_fdir_perfect_82599(c35b2c00,1,c130a0a4,c130be68,1,...) at ixgbe_init_fdir_perfect_82599+0x6456
sc_attach_unit(c122dba0,78,c33077cc,c09b0186,c33077ec,...) at sc_attach_unit+0x523
cncheckc(c33077ec,c0527005,c0e3be0d,c05282b0,c33077e8,...) at cncheckc+0x48
cngetc(c0e3be0d,c05282b0,c33077e8,c3307824,1,...) at cngetc+0x16
db_readline(c12dae60,78,c3307808,c0525c46,c0e3be0d,...) at db_readline+0x75
db_read_line(c0e3be0d,c330785c,c0527afd,c0ee42e5,0,...) at db_read_line+0x1a
db_command_loop(c0ee42e5,0,c3307830,c0e0d09d,0,...) at db_command_loop+0x46
X_db_sym_numargs(c,0,0,28,c3307960,...) at X_db_sym_numargs+0xed
kdb_trap(c,0,c3307960,1,1,...) at kdb_trap+0x96
dblfault_handler() at dblfault_handler+0x38f
--- trap 0x17, eip = 0, esp = 0, ebp = 0 ---

db> bt
Tracing pid 13 tid 64008 td 0xc35ad000
m_tag_delete(c39a8200,c39a923d,c39a8200,c39a8200,c33079ec,...) at m_tag_delete+0x52
m_tag_delete_chain(c39a8200,0,df,0,c36de000,...) at m_tag_delete_chain+0x3f
reallocf(c39a8200,100,0,9e3,c0a33d7b,...) at reallocf+0x8a5
uma_zfree_arg(c1d7e380,c39a8200,0,c36df430,c3307a60,...) at uma_zfree_arg+0x29
m_freem(c39a8200,c36e4440,8,c36cc800,c36de018,...) at m_freem+0x43
fwohci_init(c36de018,4,c0e63fff,519,c3307ab4,...) at fwohci_init+0x545c
fwohci_init(c36de018,0,c0e63fff,508,c36cc800,...) at fwohci_init+0x6613
fwohci_init(c36cc800,c3307c0c,c0aa27ef,c36cc800,0,...) at fwohci_init+0x733b
if_start(c36cc800,0,c0eb3f04,d1d,2,...) at if_start+0x12
if_handoff(c36cc800,c4077d00,0,0) at if_handoff+0x25f
ether_output_frame(c36cc800,c4077d00,c0e9a61c,1,c3da4b40,...) at ether_output_frame+0x65
ng_car_q_event(c3da9980,c3da4b40,c0ebd64e,c0e9a61c,3,...) at ng_car_q_event+0x2e2b
ng_rmnode(c391eb50,0,c0ebd64e,d2c,0,...) at ng_rmnode+0x2e4
ng_rmnode(0,c3307d38,c0e9f61d,344,c35ab550,...) at ng_rmnode+0x16a1
fork_exit(c0b0fd90,0,c3307d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3307d70, ebp = 0 ---
db> reboot

[/thread]

[FisherKing]

Don't know exactly when the problem was resolved (it was in the last week or so), but running 2.0-BETA5 (i386) built on Wed Feb 9 16:14:43 EST 2011 I am no longer experiencing the panic.

[jimp]

The mbuf tagging patch was backed out yesterday. That's probably what was causing the panic you had.