Kernel panic 4-5 Nov (i386)

FisherKing

Are you running captive portal?

CryoGenID

PJ2,

thanks for your reply.
Actually yes, I have a CP set up… Do I have to deactivate it?

Best regards,

Christian

FisherKing

turning off CP does the trick for me.

What network drivers are you using? fxp?

cmb

@CryoGenID:

Actually yes, I have a CP set up… Do I have to deactivate it?

That's something to try, it may narrow down the issue considerably.

CryoGenID

@PJ2:
I have these interfaces:
vr0 <- on-Board
ste <- 4-port 100Mbit Card (D-Link)

@cmb:
I have now switched off CP and just clicked on "Auto Update". This time pfSense was able to check if there is a new version without a restart / kernel panic.
Tomorrow will be interesting, as then there'll be an update available.

I will report back, if the update went OK tomorrow, or if I still got a kernel panic :-)

Thanks for your help anyway so far!!

Best regards,

Chris

FisherKing

Thanks Chris -

I've been trying to sort out what it is besides CP that makes the difference. I was hoping that switching NICs would solve the problem for me, but that doesn't seem to be the case. :-/

CryoGenID

Hm… Well if CP really makes the difference, then I am sure that we can sort this, together with the dev, out somehow ;-)

They would be having the same issues if they turn on CP as well, wouldn't they? Or is it indeed coupled with the NIC-type?

I am eager to see what happens tomorrow ;-)

Best regards,

Chris

CryoGenID

Update:
Tried the update today and… it worked ;D

So the problem really seems to be that I get a kernel panic with the update function if there is the CP enabled...

Strange isn't it?

Best regards

Christian

Cino

I was using a ver from the 2nd week of Dec. I would get a kernel panic every time I would start heavy downloading from the internet.. I updated the box to 16th ver and the kernel panics went away. Been updating the box every couple of days after and it seems stable. Only using Snort, no CP.

FisherKing

For a long time I thought I was the only one with this issue. Glad I'm not totally crazy.

Cino

@PJ2:

For a long time I thought I was the only one with this issue. Glad I'm not totally crazy.

I was thinking the same thing until I saw this topic. I had a good reason tho. I swapped my case back to mini-box M300 and started to mess around getting the picolcd to work again. Figured it was that at first but my panic had somthing to do with the nic irqs.

gori

I don't think it's a hardware issue. 2.0-BETA4 (i386) built on Tue Nov 2 14:53:54 EDT 2010 is perfectly stable with Captive Portal active.

Automatic update -> success -> restart -> several minutes running -> kernel panic :-/
disconnect WAN iface cable -> manually update back to the Nov 2 version

I tried these:
pfSense-Full-Update-2.0-BETA4-20101107-0244.tgz
pfSense-Full-Update-2.0-BETA4-20101110-0504.tgz
pfSense-Full-Update-2.0-BETA4-20101115-1340.tgz
and auto-update from Dec 9, Dec 17, Dec 21

all af them crashing in the same way.

Anybody know what has changed with CP between Nov 2 and Nov 7 ?

jimp

Update to the most current snapshot and try again. Many fixes have been checked in, some in the last couple days. It should be working again now without panics. (hopefully)

FisherKing

Just downloaded this mornings update (2.0-BETA4 (i386) built on Thu Dec 23 03:37:06 EST 2010)
Enabled captive portal, logged into the console, tried to fetch, and it panicked.

I see a patch related to PPPoE that ermal checked in last night that wouldn't have made it into the current snap. I'll try again when a new snap comes out. If it still panics then, I'll get the back trace and post it.

CyroGenID - I'm using PPPoE to get my WAN's IP. How about you? Does your system still panic with the 12/23 3am build?

jimp

It would be in that snap, I started building it at around 1am on the i386 builder.

FisherKing

Hmm - ok. I'll try a clean install & post the backtrace if it panics on that.

jimp

Sounds good. Normally I wouldn't know the build times quite that well but I had to kick that off by hand after rebuilding all of the ports on the box. :-)

CryoGenID

@PJ2:
I will update today to the newest snapshot and try an update (with enabled CP) tomorrow.
And yes, I am also getting my 2 WAN-IPs via PPPoE ;-)

Best regards,

Chris

CryoGenID

OK, here comes the update:
It …. didn't work :-(

I still get the panic when CP is enabled... As soon as I disable CP, the updating works perfectly :-(

Best regards,

Chris

FisherKing

Updated to 2.0-BETA5 (i386) built on Sun Dec 26 01:43:40 EST 2010

Same result as CryoGenID - panic when CP is enabled.
Also like CryoGenID - using PPPoE to pick up WAN IP.

Backtrack (+ a little more) from the panic follows.


load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched PRIO loaded

FreeBSD/i386 (xxx.xxxx.xxxx) (ttyu0)

*** Welcome to pfSense 2.0-BETA5-pfSense (i386) on xxx ***

  WAN (wan)                 -> pppoe0     -> xxx.xxx.xxx.xxx (PPPoE)
  LAN (lan)                 -> fxp2       -> yyy.yyy.yyy.yyy
  SERVER_LAN (opt1)         -> fxp1       -> xxx.xxx.xxx.xxx
  GUEST_LAN (opt2)          -> fxp3       -> zzz.zzz.zzz.zzz
  WAN_PHYSICAL (opt3)       -> fxp0       -> NONE (DHCP)

 0) Logout (SSH only)                  8) Shell
 1) Assign Interfaces                  9) pfTop
 2) Set interface(s) IP address       10) Filter Logs
 3) Reset webConfigurator password    11) Restart webConfigurator
 4) Reset to factory defaults         12) pfSense Developer Shell
 5) Reboot system                     13) Upgrade from console
 6) Halt system                       14) Disable Secure Shell (sshd)
 7) Ping host

Enter an option: 8

[2.0-BETA5][root@xxx.xxx.xxx]/root(1): ~/test
pfSense-Full-Update-2.0-BETA4-20101203-2137.tg  0% of   76 MB    0  Bps
Kernel page fault with the following non-sleepable locks held:
exclusive sleep mutex fxp0 (network driver) r = 0 (0xc36e8018) locked @ /usr/pfS
ensesrc/src/sys/dev/fxp/if_fxp.c:1288
KDB: stack backtrace:
X_db_sym_numargs(c0eb56e6,c3307888,c0a403d5,508,0,...) at X_db_sym_numargs+0x146

kdb_backtrace(508,0,ffffffff,c145cabc,c33078c0,...) at kdb_backtrace+0x29
witness_display_spinlock(c0eb7bfe,c33078d4,4,1,0,...) at witness_display_spinloc
k+0x75
witness_warn(5,0,c0ef5fad,2,c35ad550,...) at witness_warn+0x20d
trap(c3307960) at trap+0x19e
alltraps(c39bd100,c3f2ed3d,c39bd100,c39bd100,c33079ec,...) at alltraps+0x1b
m_tag_delete_chain(c39bd100,0,df,0,c36e8000,...) at m_tag_delete_chain+0x3f
reallocf(c39bd100,100,0,9e3,c0a4017b,...) at reallocf+0x8a5
uma_zfree_arg(c1d7e380,c39bd100,0,c36e91a0,c3307a60,...) at uma_zfree_arg+0x29
m_freem(c39bd100,c36eba40,8,c36ce400,c36e8018,...) at m_freem+0x43
fwohci_init(c36e8018,4,c0e70676,519,c3307ab4,...) at fwohci_init+0x545c
fwohci_init(c36e8018,0,c0e70676,508,c36ce400,...) at fwohci_init+0x6613
fwohci_init(c36ce400,c3307c0c,c0aaebef,c36ce400,0,...) at fwohci_init+0x733b
if_start(c36ce400,0,c0ec1a6c,d1d,2,...) at if_start+0x12
if_handoff(c36ce400,c3f29400,0,0) at if_handoff+0x25f
ether_output_frame(c36ce400,c3f29400,c0ea83b9,1,c3db9a80,...) at ether_output_fr
ame+0x65
ng_car_q_event(c3db4400,c3db9a80,c0ecb1af,c0ea83b9,3,...) at ng_car_q_event+0x2e
2b
ng_rmnode(c39c1450,0,c0ecb1af,d2c,0,...) at ng_rmnode+0x2e4
ng_rmnode(0,c3307d38,c0ead185,344,c35ad550,...) at ng_rmnode+0x16a1
fork_exit(c0b1c190,0,c3307d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3307d70, ebp = 0 ---

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0xdedeadc0
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xdedeadc0
stack pointer           = 0x28:0xc33079a0
frame pointer           = 0x28:0xc33079b4
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 13 (ng_queue0)
[thread]
Stopped at      0xdedeadc0:     *** error reading from address dedeadc0 ***
db> lock order reversal: (Giant after non-sleepable)
 1st 0xc36e8018 fxp0 (network driver) @ /usr/pfSensesrc/src/sys/dev/fxp/if_fxp.c
:1288
 2nd 0xc131b2d0 Giant (Giant) @ /usr/pfSensesrc/src/sys/dev/usb/input/ukbd.c:170
4
KDB: stack backtrace:
X_db_sym_numargs(c0eb56e6,c33076d8,c0a403d5,c0a30fbb,c0eb8652,...) at X_db_sym_n
umargs+0x146
kdb_backtrace(c0a30fbb,c0eb8652,c3561040,c35601a0,c3307734,...) at kdb_backtrace
+0x29
witness_display_spinlock(c0eb8652,c131b2d0,c0edc770,c35601a0,c0e9fec2,...) at wi
tness_display_spinlock+0x75
witness_checkorder(c131b2d0,9,c0e9fec2,6a8,0,...) at witness_checkorder+0x839
_mtx_lock_flags(c131b2d0,0,c0e9fec2,6a8,c3c2b6b0,...) at _mtx_lock_flags+0xc4
ucom_attach(c3c57000,1,c1319f28,c13180a0,c33077b4,...) at ucom_attach+0x1c08
ixgbe_init_fdir_perfect_82599(c35b4c00,1,c1318124,c1319f28,1,...) at ixgbe_init_
fdir_perfect_82599+0x6456
sc_attach_unit(c123b7a0,78,c33077cc,c09bc586,c33077ec,...) at sc_attach_unit+0x5
23
cncheckc(c33077ec,c0527655,c0e4846d,c0528900,c33077e8,...) at cncheckc+0x48
cngetc(c0e4846d,c0528900,c33077e8,c3307824,1,...) at cngetc+0x16
db_readline(c12e8ee0,78,c3307808,c0526296,c0e4846d,...) at db_readline+0x75
db_read_line(c0e4846d,c330785c,c052814d,c0ef1e64,0,...) at db_read_line+0x1a
db_command_loop(c0ef1e64,0,c3307830,c0e196fd,0,...) at db_command_loop+0x46
X_db_sym_numargs(c,0,0,28,c3307960,...) at X_db_sym_numargs+0xed
kdb_trap(c,0,c3307960,1,1,...) at kdb_trap+0x96
dblfault_handler() at dblfault_handler+0x38f
--- trap 0x17, eip = 0, esp = 0, ebp = 0 ---

db> bt
Tracing pid 13 tid 64008 td 0xc35af000
m_tag_delete(c39bd100,c3f2ed3d,c39bd100,c39bd100,c33079ec,...) at m_tag_delete+0
x52
m_tag_delete_chain(c39bd100,0,df,0,c36e8000,...) at m_tag_delete_chain+0x3f
reallocf(c39bd100,100,0,9e3,c0a4017b,...) at reallocf+0x8a5
uma_zfree_arg(c1d7e380,c39bd100,0,c36e91a0,c3307a60,...) at uma_zfree_arg+0x29
m_freem(c39bd100,c36eba40,8,c36ce400,c36e8018,...) at m_freem+0x43
fwohci_init(c36e8018,4,c0e70676,519,c3307ab4,...) at fwohci_init+0x545c
fwohci_init(c36e8018,0,c0e70676,508,c36ce400,...) at fwohci_init+0x6613
fwohci_init(c36ce400,c3307c0c,c0aaebef,c36ce400,0,...) at fwohci_init+0x733b
if_start(c36ce400,0,c0ec1a6c,d1d,2,...) at if_start+0x12
if_handoff(c36ce400,c3f29400,0,0) at if_handoff+0x25f
ether_output_frame(c36ce400,c3f29400,c0ea83b9,1,c3db9a80,...) at ether_output_fr
ame+0x65
ng_car_q_event(c3db4400,c3db9a80,c0ecb1af,c0ea83b9,3,...) at ng_car_q_event+0x2e
2b
ng_rmnode(c39c1450,0,c0ecb1af,d2c,0,...) at ng_rmnode+0x2e4
ng_rmnode(0,c3307d38,c0ead185,344,c35ad550,...) at ng_rmnode+0x16a1
fork_exit(c0b1c190,0,c3307d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3307d70, ebp = 0 ---
db>
Tracing pid 13 tid 64008 td 0xc35af000
m_tag_delete(c39bd100,c3f2ed3d,c39bd100,c39bd100,c33079ec,...) at m_tag_delete+0
x52
m_tag_delete_chain(c39bd100,0,df,0,c36e8000,...) at m_tag_delete_chain+0x3f
reallocf(c39bd100,100,0,9e3,c0a4017b,...) at reallocf+0x8a5
uma_zfree_arg(c1d7e380,c39bd100,0,c36e91a0,c3307a60,...) at uma_zfree_arg+0x29
m_freem(c39bd100,c36eba40,8,c36ce400,c36e8018,...) at m_freem+0x43
fwohci_init(c36e8018,4,c0e70676,519,c3307ab4,...) at fwohci_init+0x545c
fwohci_init(c36e8018,0,c0e70676,508,c36ce400,...) at fwohci_init+0x6613
fwohci_init(c36ce400,c3307c0c,c0aaebef,c36ce400,0,...) at fwohci_init+0x733b
if_start(c36ce400,0,c0ec1a6c,d1d,2,...) at if_start+0x12
if_handoff(c36ce400,c3f29400,0,0) at if_handoff+0x25f
ether_output_frame(c36ce400,c3f29400,c0ea83b9,1,c3db9a80,...) at ether_output_fr
ame+0x65
ng_car_q_event(c3db4400,c3db9a80,c0ecb1af,c0ea83b9,3,...) at ng_car_q_event+0x2e
2b
ng_rmnode(c39c1450,0,c0ecb1af,d2c,0,...) at ng_rmnode+0x2e4
ng_rmnode(0,c3307d38,c0ead185,344,c35ad550,...) at ng_rmnode+0x16a1
fork_exit(c0b1c190,0,c3307d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3307d70, ebp = 0 ---
db> reboot

[/thread]