2.1rc on kvm on linux-x86-64 won't boot with "Fatal trap 12"

wallabybob

At the

db>

prompt type

bt

then hit the Enter key to get a "backtrace" show the function call tree. This might give some clues to narrow down the problem.

speculatrix

thanks for coming back to me.

–edit-- captured a better backtrace by fixing up serial port during boot


 �������������������������������������������
 �          Welcome to pfSense!            �                             
 �  1\. Boot pfSense [default]              �          _____/    f   \       
 �������������������������������������������

KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.3-RELEASE-p8 #0: Sun Jun  9 21:59:09 EDT 2013
    root@snapshots-8_3-amd64.builders.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8 amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: QEMU Virtual CPU version 1.3.0 (2494.44-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x623  Family = 6  Model = 2  Stepping = 3
  Features=0x783fbfd <fpu,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,mmx,fxsr,sse,sse2>Features2=0x80802001 <sse3,cx16,popcnt,hv>AMD Features=0x20100800 <syscall,nx,lm>AMD Features2=0x1 <lahf>real memory  = 536870912 (512 MB)
avail memory = 491675648 (468 MB)
ACPI APIC Table: <bochs  bxpcapic="">
ioapic0 <version 1.1=""> irqs 0-23 on motherboard
wlan: mac acl policy registered
ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff804a5250, 0) error 1
ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff804a52f0, 0) error 1
ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff804a5390, 0) error 1
kbd1 at kbdmux0
cryptosoft0: <software crypto=""> on motherboard
padlock0: No ACE support.
acpi0: <bochs bxpcrsdt=""> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
cpu0: <acpi cpu=""> on acpi0
pcib0: <acpi host-pci="" bridge=""> port 0xcf8-0xcff on acpi0
pci0: <acpi pci="" bus=""> on pcib0
isab0: <pci-isa bridge=""> at device 1.0 on pci0
isa0: <isa bus=""> on isab0
atapci0: <intel piix3="" wdma2="" controller=""> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc240-0xc24f at device 1.1 on pci0
ata0: <ata channel=""> at channel 0 on atapci0
ata0: [ITHREAD]
ata1: <ata channel=""> at channel 1 on atapci0
ata1: [ITHREAD]
uhci0: <intel 82371sb="" (piix3)="" usb="" controller=""> port 0xc200-0xc21f irq 11 at device 1.2 on pci0
uhci0: [ITHREAD]
usbus0: controller did not stop
usbus0: <intel 82371sb="" (piix3)="" usb="" controller=""> on uhci0
pci0: <bridge> at device 1.3 (no driver attached)
vgapci0: <vga-compatible display=""> mem 0xfd000000-0xfdffffff,0xfebf0000-0xfebf0fff at device 2.0 on pci0
re0: <realtek 10="" 8139c+="" 100basetx=""> port 0xc000-0xc0ff mem 0xfebf1000-0xfebf10ff irq 11 at device 3.0 on pci0
re0: Chip rev. 0x74800000
re0: MAC rev. 0x00000000
miibus0: <mii bus=""> on re0
rlphy0: <realtek internal="" media="" interface=""> PHY 0 on miibus0
rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
re0: [FILTER]
re1: <realtek 10="" 8139c+="" 100basetx=""> port 0xc100-0xc1ff mem 0xfebf2000-0xfebf20ff irq 11 at device 4.0 on pci0
re1: Chip rev. 0x74800000
re1: MAC rev. 0x00000000
miibus1: <mii bus=""> on re1
rlphy1: <realtek internal="" media="" interface=""> PHY 0 on miibus1
rlphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
re1: [FILTER]
pci0: <old> at device 5.0 (no driver attached)
acpi_hpet0: <high precision="" event="" timer=""> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 100000000 Hz quality 900
atrtc0: <at realtime="" clock=""> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
atkbdc0: <keyboard controller="" (i8042)=""> port 0x60,0x64 irq 1 on acpi0
atkbd0: <at keyboard=""> irq 1 on atkbdc0
kernel trap 12 with interrupts disabled

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0xffffff8000012000
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80b42fff
stack pointer           = 0x28:0xffffffff8166b6a0
frame pointer           = 0x28:0xffffffff8166b6d0
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = resume, IOPL = 0
current process         = 0 (swapper)
[thread]
Stopped at      x86bios_emu_rdw+0x2f:   movzwl  (%rbx),%eax
db> bt
Tracing pid 0 tid 100000 td 0xffffffff81386b90
x86bios_emu_rdw() at x86bios_emu_rdw+0x2f
fetch_word() at fetch_word+0x13
fetch_data_word() at fetch_data_word+0x27
decode_and_fetch_word_disp() at decode_and_fetch_word_disp+0x40
decode_and_fetch_word() at decode_and_fetch_word+0xb
x86emu_exec() at x86emu_exec+0x387e
x86bios_intr() at x86bios_intr+0xbe
get_typematic() at get_typematic+0x55
atkbd_init() at atkbd_init+0x2f9
atkbd_attach_unit() at atkbd_attach_unit+0x91
atkbdattach() at atkbdattach+0x78
device_attach() at device_attach+0x69
bus_generic_attach() at bus_generic_attach+0x1a
atkbdc_isa_attach() at atkbdc_isa_attach+0x155
device_attach() at device_attach+0x69
bus_generic_attach() at bus_generic_attach+0x1a
acpi_attach() at acpi_attach+0xab6
device_attach() at device_attach+0x69
bus_generic_attach() at bus_generic_attach+0x1a
nexus_acpi_attach() at nexus_acpi_attach+0x69
device_attach() at device_attach+0x69
bus_generic_new_pass() at bus_generic_new_pass+0xd6
bus_set_pass() at bus_set_pass+0x7a
configure() at configure+0xa
mi_startup() at mi_startup+0x77
btext() at btext+0x2c
db> 

I will try booting an older kernel and see if that helps.

--edit-- this hasn't been too successful so far.[/thread]</at></keyboard></at></high></old></realtek></mii></realtek></realtek></mii></realtek></vga-compatible></bridge></intel></intel></ata></ata></intel></isa></pci-isa></acpi></acpi></acpi></bochs></software></version></bochs ></lahf></syscall,nx,lm></sse3,cx16,popcnt,hv></fpu,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,mmx,fxsr,sse,sse2>

speculatrix

I've also tried changing the virtual display from cirrus to vga, drives from IDE to virtio etc, as much as I can. Even tried the memstick serial one and remove video!

speculatrix

OK, I can boot pfsense 2.1rc on my old linux install*, kernel 3.4.6-x86-64, with the following packages:

kvm-1.2.prerc0-10.2.x86_64
libvirt-0.9.13-11.1.x86_64
libvirt-client-0.9.13-11.1.x86_64
libvirt-python-0.9.13-11.1.x86_64
opensuse-kvm_en-pdf-12.1-12.3.2.noarch
python-virtinst-0.600.3-3.1.x86_64
qemu-tools-1.2.0rc0-14.1.x86_64
virt-manager-0.9.4-8.1.x86_64
virt-utils-1.1.7-3.2.1.x86_64
virt-viewer-0.4.1-3.3.1.x86_64

so now I need to test newer linux install with older kernel.

my laptop is triple boot; a small Win7 install (a fairly minimal working enviroment for the very rare programs for which there's no linux substitute), and two linux installs. So I can try different linux distros, or, simply ensure I have a bootable good snapshot before doing major system upgrades!

speculatrix

still haven't been able to boot the kernel from the old release, but I did try installing the default kernel and use that instead of the desktop kernel, no difference.

foonus

Same end error I am seeing on a direct hardware install http://forum.pfsense.org/index.php/topic,63484.0.html

Been this way since any snapshots after June 12, reverting back to it works just fine. Shame the June 12 version has been removed from the current snapshot list.
Sucks that there is a problem but at least its not an isolated one!

*edit
For curiosity sake… do you have any Intel NICs installed in this system??

speculatrix

I am trying to boot a pfsense disk image as a virtual machine, and it doesn't matter what kind of NIC is in use, what kind of CPU I set it to, whether it's a "thin" virtio drive or a fully virtualised IDE controller, I still get the same problem.

I set up serial console for when pfsense boots and removed the virtual video card, no luck.

speculatrix

I even just tried removing the IDE controller and setting the disk type to virtio, setting virtual network devices to virtio.

All that's left is virtual USB which I can't remove!

foonus

We identified that something changed after the 12-13 that causes this "Fatal Trap 12" on certain setups.. unfortunately we still need somebody to look into a resolution.
http://forum.pfsense.org/index.php/topic,63484.0.html

wallabybob

@foonus:

We identified that something changed after the 12-13 that causes this "Fatal Trap 12" on certain setups. . . . .
http://forum.pfsense.org/index.php/topic,63484.0.html

This crash occurs in attach processing of the keyboard driver.The crash in the referenced topic occurs in detach processing of the bge driver.
It is not obvious these are related other than having a similar crash cause: "Fatal Trap 12".