Kernel panic 4-5 Nov (i386)

FisherKing

I've found another way to produce the kernel panic and it seems to be related.

I've noticed that when I log into the GUI that pfSense doesn't panic until the automatic update check has completed. Related to that, if I ssh into the console and select option 13 (upgrade from console), and then option 1 (Update from a URL), the system will panic when it starts to download the file from the snapshot server.

Even stranger, if I ssh into the console, choose option 8 (shell), then then type something like "fetch http://snapshots.pfsense.org/FreeBSD_RELENG_8_1/i386/pfSense_HEAD/updates/pfSense-Full-Update-2.0-BETA4-20101115-1340.tgz" the kernel will panic then also.

My "WAN" port is configured via PPPoE. I've also set the physical port that the WAN is on as Opt 3.

Still having this panic on snapshot dated "Nov 15 16:00:39 EST 2010".

FisherKing

Ok, I figured out how to setup the serial console on the full version of pfSense.

Having done that, I've captured a couple of kernel panics and the back traces from that. I've noticed that the panics appear to be related to my WAN port. My WAN's IP is received via PPPoE.

The others who are seeing this panic - are you also getting your WAN IP via PPPoE? Or maybe PPTP? What driver is your WAN using?

Here are the kernel panics & back traces.

–After running fetch from an SSH terminal session


# fetch http://snapshots.pfsense.org/FreeBSD_RELENG_8_1/i386/pfSense_HEAD/update
s/pfSense-Full-Update-2.0-BETA4-20101115-1340.tgz

pfSense-Full-Update-2.0-BETA4-20101115-1340.tg  0% of   75 MB    0  Bps

Kernel page fault with the following non-sleepable locks held:
exclusive sleep mutex fxp0 (network driver) r = 0 (0xc36c2018) locked @ /usr/pfS
ensesrc/src/sys/dev/fxp/if_fxp.c:1288
KDB: stack backtrace:
X_db_sym_numargs(c0ea6373,c330788c,c0a32ac5,508,0,...) at X_db_sym_numargs+0x146

kdb_backtrace(508,0,ffffffff,c144d77c,c33078c4,...) at kdb_backtrace+0x29
witness_display_spinlock(c0ea888b,c33078d8,4,1,0,...) at witness_display_spinloc
k+0x75
witness_warn(5,0,c0ee6c23,c144d778,c3590550,...) at witness_warn+0x20d
trap(c3307964) at trap+0x19e
alltraps(c36d1b00,dedeadc0,c36d1b00,c36d1b00,c33079ec,...) at alltraps+0x1b
m_tag_delete_chain(c36d1b00,0,df,0,c36c2000,...) at m_tag_delete_chain+0x3f
reallocf(c36d1b00,100,0,9e3,3,...) at reallocf+0x8a5
uma_zfree_arg(c1d7e380,c36d1b00,0,c36c3020,c3307a60,...) at uma_zfree_arg+0x29
m_freem(c36d1b00,c36ca5c0,8,c36b9800,c36c2018,...) at m_freem+0x43
fwohci_init(c36c2018,4,c0e62817,519,c3307ab4,...) at fwohci_init+0x545c
fwohci_init(c36c2018,0,c0e62817,508,c36b9800,...) at fwohci_init+0x6613
fwohci_init(c36b9800,c3307c0c,c0aa12bf,c36b9800,0,...) at fwohci_init+0x733b
if_start(c36b9800,0,c0eb26f9,d1d,2,...) at if_start+0x12
if_handoff(c36b9800,c39ad800,0,0) at if_handoff+0x25f
ether_output_frame(c36b9800,c39ad800,c0e98e11,1,c3dac380,...) at ether_output_fr
ame+0x65
ng_car_q_event(c3db1080,c3dac380,c0ebbe43,c0e98e11,3,...) at ng_car_q_event+0x2e
2b
ng_rmnode(c3903bd0,0,c0ebbe43,d2c,0,...) at ng_rmnode+0x2e4
ng_rmnode(0,c3307d38,c0e9de12,344,c3590550,...) at ng_rmnode+0x16a1
fork_exit(c0b0e840,0,c3307d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3307d70, ebp = 0 ---

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0xdedeadc0
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc0a51d58
stack pointer           = 0x28:0xc33079a4
frame pointer           = 0x28:0xc33079b4
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 13 (ng_queue0)
[thread]
Stopped at      m_tag_delete+0x48:      movl    0(%ecx),%eax
db>
db>
db>
db>
db> bt
Tracing pid 13 tid 64008 td 0xc3592000
m_tag_delete(c36d1b00,dedeadc0,c36d1b00,c36d1b00,c33079ec,...) at m_tag_delete+0
x48
m_tag_delete_chain(c36d1b00,0,df,0,c36c2000,...) at m_tag_delete_chain+0x3f
reallocf(c36d1b00,100,0,9e3,3,...) at reallocf+0x8a5
uma_zfree_arg(c1d7e380,c36d1b00,0,c36c3020,c3307a60,...) at uma_zfree_arg+0x29
m_freem(c36d1b00,c36ca5c0,8,c36b9800,c36c2018,...) at m_freem+0x43
fwohci_init(c36c2018,4,c0e62817,519,c3307ab4,...) at fwohci_init+0x545c
fwohci_init(c36c2018,0,c0e62817,508,c36b9800,...) at fwohci_init+0x6613
fwohci_init(c36b9800,c3307c0c,c0aa12bf,c36b9800,0,...) at fwohci_init+0x733b
if_start(c36b9800,0,c0eb26f9,d1d,2,...) at if_start+0x12
if_handoff(c36b9800,c39ad800,0,0) at if_handoff+0x25f
ether_output_frame(c36b9800,c39ad800,c0e98e11,1,c3dac380,...) at ether_output_fr
ame+0x65
ng_car_q_event(c3db1080,c3dac380,c0ebbe43,c0e98e11,3,...) at ng_car_q_event+0x2e
2b
ng_rmnode(c3903bd0,0,c0ebbe43,d2c,0,...) at ng_rmnode+0x2e4
ng_rmnode(0,c3307d38,c0e9de12,344,c3590550,...) at ng_rmnode+0x16a1
fork_exit(c0b0e840,0,c3307d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3307d70, ebp = 0 ---
db>

After logging into the GUI - the dashboard just completed it's automatic update check
[code]
Kernel page fault with the following non-sleepable locks held:
exclusive sleep mutex fxp0 (network driver) r = 0 (0xc36c2018) locked @ /usr/pfS
ensesrc/src/sys/kern/kern_mutex.c:147
KDB: stack backtrace:
X_db_sym_numargs(c0ea6373,c3304a4c,c0a32ac5,93,0,...) at X_db_sym_numargs+0x146
kdb_backtrace(93,0,ffffffff,c144d82c,c3304a84,...) at kdb_backtrace+0x29
witness_display_spinlock(c0ea888b,c3304a98,4,1,0,...) at witness_display_spinloc
k+0x75
witness_warn(5,0,c0ee6c23,c144d828,c35907f8,...) at witness_warn+0x20d
trap(c3304b24) at trap+0x19e
alltraps(c39af800,dedeadc0,c39af800,c39af800,c3304bac,...) at alltraps+0x1b
m_tag_delete_chain(c39af800,0,df,0,c36c2000,...) at m_tag_delete_chain+0x3f
reallocf(c39af800,100,0,9e3,3,...) at reallocf+0x8a5
uma_zfree_arg(c1d7e380,c39af800,0,c36c32c0,c3304c20,...) at uma_zfree_arg+0x29
m_freem(c39af800,c36c9a00,8,c36c2000,c36b9800,...) at m_freem+0x43
fwohci_init(c36c2018,4,c0e62817,82a,c36c2018,...) at fwohci_init+0x545c
fwohci_init(c36c2000,1,c0ea4352,189,c130ccf8,...) at fwohci_init+0x7a25
softclock(c130ccc0,c3304cc8,c09deb04,c1310a80,c35b95b8,...) at softclock+0x24a
intr_event_execute_handlers(c35907f8,c35b9580,c0e9e0ad,533,c35b95f0,...) at intr
_event_execute_handlers+0x125
intr_event_add_handler(c358f110,c3304d38,c0e9de12,344,c35907f8,...) at intr_even
t_add_handler+0x42f
fork_exit(c09c78b0,c358f110,c3304d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3304d70, ebp = 0 ---

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0xdedeadc0
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc0a51d58
stack pointer           = 0x28:0xc3304b64
frame pointer           = 0x28:0xc3304b74
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 12 (swi4: clock)
[thread]
Stopped at      m_tag_delete+0x48:      movl    0(%ecx),%eax
db>
db>
db> bt
Tracing pid 12 tid 64007 td 0xc3592280
m_tag_delete(c39af800,dedeadc0,c39af800,c39af800,c3304bac,...) at m_tag_delete+0
x48
m_tag_delete_chain(c39af800,0,df,0,c36c2000,...) at m_tag_delete_chain+0x3f
reallocf(c39af800,100,0,9e3,3,...) at reallocf+0x8a5
uma_zfree_arg(c1d7e380,c39af800,0,c36c32c0,c3304c20,...) at uma_zfree_arg+0x29
m_freem(c39af800,c36c9a00,8,c36c2000,c36b9800,...) at m_freem+0x43
fwohci_init(c36c2018,4,c0e62817,82a,c36c2018,...) at fwohci_init+0x545c
fwohci_init(c36c2000,1,c0ea4352,189,c130ccf8,...) at fwohci_init+0x7a25
softclock(c130ccc0,c3304cc8,c09deb04,c1310a80,c35b95b8,...) at softclock+0x24a
intr_event_execute_handlers(c35907f8,c35b9580,c0e9e0ad,533,c35b95f0,...) at intr
_event_execute_handlers+0x125
intr_event_add_handler(c358f110,c3304d38,c0e9de12,344,c35907f8,...) at intr_even
t_add_handler+0x42f
fork_exit(c09c78b0,c358f110,c3304d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3304d70, ebp = 0 ---
db>
[/thread][/code][/thread]

singerie

my wan is DHCP.

my pfsense was rebooting while downloading lightsquid.

it seems to crash only when the box itself is getting something from the internet.

slow dsl, or fast cable connection, same issue here i think.

FisherKing

Hi singerie - So it probably isn't PPPoE then. Thanks.

are you running either Captive Portal or Traffic shaping?
What network driver is your WAN using?

singerie

i'm not using captive portal or traffic shaping, but i use multi wan, but only as a failover.

My network card is a Intel pci-e.

wallabybob

I've seen similar panics in a build of 9-Nov. More in http://forum.pfsense.org/index.php/topic,29927.0.html

Beerman

Same panic here: http://forum.pfsense.org/index.php/topic,30048.0.html

FisherKing

Beerman, Wallabybob, I'm curious, what nic is assigned to your WAN? Or what driver is your WAN using?

wallabybob

I'm using rl0 for my WAN interface. I could easily swap to vr0 or a wireless link or a USB NIC if you thought it worthwhile to gather some more data. I can reproduce my problem fairly easily.

Efonnes

My Alix 2c3 has this happen when updating and randomly at some other times when accessing the web gui. WAN is vr1 and uses only DHCP, LAN is vr0, and I have ath0 for an access point on OPT1.

PJ2: Some good information there. It looks like it is crashing in the same function at the same line of code every time, called from the same code path.

wallabybob

I swapped interfaces so vr0 was my WAN and on three restarts I've hit the same panic: syncache: mbuf too small

FisherKing

I had wondered if it was just an issue w/ intel nics. Sounds like it's not that specific. Thanks Wallabybob.

Beerman

@PJ2:

Beerman, Wallabybob, I'm curious, what nic is assigned to your WAN? Or what driver is your WAN using?

My WAN is on vr2_vlan7. (ALIX Board)

CryoGenID

Any updates on this already?

We are also facing this problem here…
We stumbled over this "bug" (or whatever it is) a few weeks ago, when we tried to update to the newest snapshot...
Normally we used to update every few days and then suddenly we got this "Fatal Trap 12"-Error during the update...

It seems to be related to the update-system of pfSense, as it occurs either when checking for a new FW or trying to
download a new FW (using the Web-Interface or the Shell doesn't matter, the Trap comes anyway).
(I have attached a screenshot of our system)

We are using pfSense on >10 boxes, all with the same HW.
We have already switched the complete HW, but the problem stays the same (so I doubt that it is related to HW).

WAN1: ste0
WAN2: ste1
LAN: vr0

Hopefully there will be a fix soon (as we want to be able to upgrade again, also because of the now fixed PPTP-Bug)...

Thanks a lot and best regards,

Christian

pfSense_ERROR.jpg_thumb

CryoGenID

Hello everybody,

are there already any news on this?
As I can on other threads here, the "kernel panic"-problem seems to be happening to a lot of people…

Is this a bug in FreeBSD itself oder pfSense?

Thanks and best regards,

Christian

eri--

I am looking into it and will post when a solution is checked in.

CryoGenID

Perfect, ermal,

thank you :)

Best regards,

Chris

FisherKing

Yeah - thanks for looking into this ermal. It's hard to to much other testing when we can't do anything w/ the GUI. :)

Let us know if you need any other information. I can probably arrange remote access to the box if you can't reproduce this on your hardware.

mbartosch

Hi,

unhappy camper as well. Using the snapshot from 2010-11-18 I tried to configure an interface group, after this I got trapped in a kernel trap - reboot loop.

Running on a Soekris 5501.

FreeBSD/i386 bootstrap loader, Revision 1.1
(sullrich@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org, Thu Nov 18 07:57:55 EST 20)
…
vlan7: changing name to 'lagg0_vlan2920'
vlan8: changing name to 'lagg0_vlan2921'
vlan9: changing name to 'lagg0_vlan2930'
vlan10: changing name to 'lagg0_vlan2931'

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x14
fault code = supervisor read, page not present
instruction pointer = 0x20:0xc095d504
stack pointer = 0x28:0xc33fbb48
frame pointer = 0x28:0xc33fbb78
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 11 (swi4: clock)
trap number = 12
panic: page fault
cpuid = 0
Uptime: 29s

eri--

Give a shot at tomorrows snapshots.