Kernel Panic
-
may have spoke to soon,
No panic but firewall threw this error and lost all connects.
hard to hard reset the boxpf_test: pf_get_mtag returned null
-
Hi,
My setup consists of two boxes, both having quad intel nic's ("card=0x10a48086 chip=0x10a48086 rev=0x06" from pciconf -lvb).
Both of the boxes run 2.0-BETA5 (amd64) built on Wed Jan 26 01:53:43 EST 2011.I have had three different types of errors occuring on these;
1 - Box spontaneously reboots
2 - Fatal trap 12 type error
3 - Box freezesI only have the OpenOSPFD and Postfix Forwarder packages installed. No VLANs. Current uptime is 50 and 33 minutes. I am expecting one of these boxes to do #1, 2 or 3 above within the next 10 or so minutes. They've been doing that since the upgrade.
(while writing this post, one of them did indeed die by freezing)Prior to upgrading these two, I had no panics, though I did have other errors, such as connections being reset, which was what prompted me to upgrade.
Output of pciconf -lvb:
em0@pci0:3:0:0: class=0x020000 card=0x10a48086 chip=0x10a48086 rev=0x06 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xec120000, size 131072, enabled
bar [14] = type Memory, range 32, base 0xec100000, size 131072, enabled
bar [18] = type I/O Port, range 32, base 0x2000, size 32, enabled
em1@pci0:3:0:1: class=0x020000 card=0x10a48086 chip=0x10a48086 rev=0x06 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xec160000, size 131072, enabled
bar [14] = type Memory, range 32, base 0xec140000, size 131072, enabled
bar [18] = type I/O Port, range 32, base 0x2020, size 32, enabled
em2@pci0:4:0:0: class=0x020000 card=0x10a48086 chip=0x10a48086 rev=0x06 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xec220000, size 131072, enabled
bar [14] = type Memory, range 32, base 0xec200000, size 131072, enabled
bar [18] = type I/O Port, range 32, base 0x3000, size 32, enabled
em3@pci0:4:0:1: class=0x020000 card=0x10a48086 chip=0x10a48086 rev=0x06 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xec260000, size 131072, enabled
bar [14] = type Memory, range 32, base 0xec240000, size 131072, enabled
bar [18] = type I/O Port, range 32, base 0x3020, size 32, enabled
bge0@pci0:14:0:0: class=0x020000 card=0x7051103c chip=0x165a14e4 rev=0x00 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 64, base 0xed000000, size 65536, enabledThe trap 12 error is as follows:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 0
fault virtual address = 0xa8
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff801fe02f
stack pointer = 0x20:0xffffff8058b56b10
frame pointer = 0x20:0xffffff8058b56b28
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 8 (pfpurge) -
current process = 8 (pfpurge)
-
-
Hello,
I've just upgraded pfSense on my virtual machine to version:
2.0-BETA5 (i386) built on Tue Jan 25 22:26:25 EST 2011
and first time - kernel panic happend - never seen before.
Updated from.. I don't know earilier version, but from January.
Screenshot attached.UPDATE: just few hours after reboot, it hanged again without kernel error messages :(
-
tested again to be sure and the error did happen again.
Otherwise, i was connected much longer and is looking promising.pf_test: pf_get_mtag returned null
-
Can you type "bt" at that db prompt and post the backtrace output?
-
2.0-BETA5 (amd64)
built on Tue Jan 25 07:56:16 EST 2011In the ultimate of ironies, I installed the dev kernel about 5 hours ago and rebooted. I checked 5 minutes ago and pfsense was in panic. Unfortunately when I tried to save the console output my serial terminal program froze, probably because my nfs share had disappeared.
Fortunately, within 2 minutes of having pfsense back up again, and while composing this post, it panicked again. I've attached two files for your reading pleasure, but I'm back to the SMP kernel for now.
-
huh…just tested again but with a about 500mb
Did two transfers of this size during the same openvpn session and everything was fine.
Will keep an eye on it.
update: just happened again. I will try to get to the console.
-
Yeah this one is definitely new. When ermal comes online he should hopefully be able to spot the problem.
-
I understand the em is mainly a driver issue, but what about the soekris vr? Is that just a driver issue too?
-
Not sure. It may be a more general bug that certain driver code is more likely to hit. Ermal had some ideas based on the testing done here with that custom kernel yesterday, but he was pretty overloaded with other stuff so I'm not sure if that specific fix made it in last night. Plenty of other changes went in though.
-
After 2 hours 40 minutes, one of the firewalls panicked again. This time it didn't freeze though, so I managed to get a backtrace. Here's what it says:
Stopped at pf_state_tree_id_RB_REMOVE_COLOR+0x90: cmpl $0x1,0xa (rest is not visible on the photo)
db> bt
Tracing pid 8 tid 64074 td 0xffffff00025a3000
pf_state_tree_id_RB_REMOVE_COLOR() at pf_state_tree_id_RB_REMOVE_COLOR+0x (something)
pf_state_tree_id_RB_REMOVE() at pf_state_tree_id_RB_REMOVE+0x126
pf_unlink_state() at pf_unlink_state+0x5e
pf_purge_expired_states() at pf_purge_expired_states+0xed
pf_purge_thread() at pf_purge_thread+0xb4
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
–- trap 0, rip = 0, rsp = 0xffffff801f73ed30, rbp = 0 ---[might be an error or two, I typed that off of a photo]
Cheers.
-
You have any ftp/pptp traffic going in?
-
@ermal:
You have any ftp/pptp traffic going in?
Only very little ftp, but that doesn't seem to trigger it.
I do, however, have quite a bit of http traffic (~300-600mbit) from a million or so clients. -
Nope, still a problem. I updated the carp backup to the latest snapshot (Jan 26, with the master still on Jan 24), added a new VIP addr on the master, and the backup immediately panic'd.
-
Hi! this is my first post on this forum.
I have been using pfsense for one year, I have installed it on a alix 2d3 an on a jetway nc9c-550lf. since 1 month pfsense rebooting itself on both installation.
The snapshot is 2.0-BETA5 (i386) built on Wed Jan 26 09:44:03 EST 2011
On screen I see this messages:
Fatal trap 12: page fault while in kernel mode
cpuid= 1; apic id= 01
fault virtual address = 0x0
fault code = supervisor read, page not present
instruction pointer = 0x20:0x0
stack pointer = 0x28:8xc5610b80
frame pointer = 0x28:8xc5610b8c
code segment = base 0x0, limit 0xfffff, type0x1b
= DPL 0, pres 1, def32 1, gran1
processor eflags= interrupt enable, resume,IOPL = 0
current process= 12(swi5: + ) -
Without the output of "bt" at that debug prompt, it's hard to say. Yours doesn't look like the other panics in this thread.
-
the next time I will try the bt command and i will post the screen.
thx
-
the next time I will try the bt command and i will post the screen.
…after you install the dev kernel. Instructions are on the wiki (for full installs) or about 2 screens back for embedded.
