Pfsense crashed at 8am. Can anyone help me figure out what caused it?

corvey

Can anyone determine what caused the crash; a hardware or software package add-on perhaps? This morning at 8am the internet was gone while I surfing previously the moment before, but I discovered pfsense was frozen 99.99% (still a 8 second delay in caps lock and num lock) and when I tried use reboot in the console I couldn't access the keys, so I hit the reset button and it said NOT READY, so I waited for like 10 minutes, still not ready it says. I said screw it, I need internet and forced a power down. I booted it back up and pfsense reboot itself once and then booted up fine the second time. Then when I went into the Pfsense url, it gave me the crash report.

–--------------------------------------------------------------------------------
Crash report begins. Anonymous machine information:

amd64
8.1-RELEASE-p6
FreeBSD 8.1-RELEASE-p6 #0: Mon Dec 12 18:15:35 EST 2011 root@FreeBSD_8.0_pfSense_2.0-AMD64.snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8

Crash report details:

Filename: /var/crash/bounds
1

Filename: /var/crash/info.0
Dump header from device /dev/da0s1b
Architecture: amd64
Architecture Version: 1
Dump Length: 62976B (0 MB)
Blocksize: 512
Dumptime: Thu Sep 6 12:07:27 2012
Hostname: brick.localdomain
Magic: FreeBSD Text Dump
Version String: FreeBSD 8.1-RELEASE-p6 #0: Mon Dec 12 18:15:35 EST 2011
root@FreeBSD_8.0_pfSense_2.0-AMD64.snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8
Panic String:
Dump Parity: 3615442783
Bounds: 0
Dump Status: good

Filename: /var/crash/minfree
2048

Filename: /var/crash/textdump.tar.0
ddb.txt��0600��0��0��114346��12022110777� 7107� ��ustar��root��wheel��db:0:kdb.enter.default> run lockinfo
db:1:lockinfo> show locks
No such command
db:1:locks> show alllocks
No such command
db:1:alllocks> show lockedvnods
Locked vnodes
db:0:kdb.enter.default> show pcpu
cpuid = 2
dynamic pcpu = 0xffffff807ef11300
curthread = 0xffffff009d49bba0: pid 24640 "syslogd"
curpcb = 0xffffff803fe24d40
fpcurthread = none
idlethread = 0xffffff00023583e0: pid 11 "idle: cpu2"
curpmap = 0
tssp = 0xffffffff811dabd0
commontssp = 0xffffffff811dabd0
rsp0 = 0xffffff803fe24d40
gs32p = 0xffffffff811d9a08
ldt = 0xffffffff811d9a48
tss = 0xffffffff811d9a38
db:0:kdb.enter.default> bt
Tracing pid 24640 tid 64115 td 0xffffff009d49bba0
vm_page_flag_set() at vm_page_flag_set+0x1
vm_fault() at vm_fault+0x14dc
trap_pfault() at trap_pfault+0x132
trap() at trap+0x4dc
calltrap() at calltrap+0x8
--- trap 0xc, rip = 0x80050d8ee, rsp = 0x7fffffffd980, rbp = 0x800539000 ---
db:0:kdb.enter.default> ps
pid ppid pgrp uid state wmesg wchan cmd
30454 262 24 0 LL+ *vm page 0xffffff0003602780 sh
24640 24073 24640 0 RLs CPU 2 syslogd
24328 1 24 0 S+ piperd 0xffffff009d482b60 logger
24255 1 24 0 RL+ CPU 0 tcpdump
24073 1 24 0 S+ wait 0xffffff009d4df000 initial thread
17648 1 17648 65 Ss select 0xffffff009d40bb40 dhclient
8668 1 8668 0 Ss select 0xffffff009d2e9bc0 dhclient
270 1 270 0 Ss select 0xffffff009d2eb3c0 devd
262 24 24 0 S+ piperd 0xffffff009d3442d8 initial thread
259 257 257 0 S kqread 0xffffff009d2e6000 check_reload_status
257 1 257 0 Ss kqread 0xffffff009d2e6700 check_reload_status
39 0 0 0 SL mdwait 0xffffff0004fad800 [md0]
24 1 24 0 Ss+ wait 0xffffff009d27e460 sh
23 0 0 0 SL sdflush 0xffffffff811a0498 [softdepflush]
22 0 0 0 SL syncer 0xffffffff8117dbc0 [syncer]
21 0 0 0 SL vlruwt 0xffffff009d261460 [vnlru]
20 0 0 0 SL psleep 0xffffffff8117d6e8 [bufdaemon]
19 0 0 0 SL pollid 0xffffffff811587c8 [idlepoll]
18 0 0 0 SL pgzero 0xffffffff811a1f2c [pagezero]
17 0 0 0 SL psleep 0xffffffff811a12c8 [vmdaemon]
16 0 0 0 SL psleep 0xffffffff811a128c [pagedaemon]
9 0 0 0 SL ccb_scan 0xffffffff8111f760 [xpt_thrd]
8 0 0 0 SL pftm 0xffffffff80204b30 [pfpurge]
7 0 0 0 SL waiting_ 0xffffffff81189b80 [sctp_iterator]
15 0 0 0 SL (threaded) usb
64058 D - 0xffffff80003d9dd0 [usbus4]
64057 D - 0xffffff80003d9d78 [usbus4]
64056 D - 0xffffff80003d9d20 [usbus4]
64055 D - 0xffffff80003d9cc8 [usbus4]
64054 D - 0xffffff80003d0ef0 [usbus3]
64053 D - 0xffffff80003d0e98 [usbus3]
64052 D - 0xffffff80003d0e40 [usbus3]
64051 D - 0xffffff80003d0de8 [usbus3]
64049 D - 0xffffff80003c7ef0 [usbus2]
64048 D - 0xffffff80003c7e98 [usbus2]
64047 D - 0xffffff80003c7e40 [usbus2]
64046 D - 0xffffff80003c7de8 [usbus2]
64044 D - 0xffffff80003beef0 [usbus1]
64043 D - 0xffffff80003bee98 [usbus1]
64042 D - 0xffffff80003bee40 [usbus1]
64041 D - 0xffffff80003bede8 [usbus1]
64039 D - 0xffffff80003b5ef0 [usbus0]
64038 D - 0xffffff80003b5e98 [usbus0]
64037 D - 0xffffff80003b5e40 [usbus0]
64036 D - 0xffffff80003b5de8 [usbus0]
14 0 0 0 SL - 0xffffffff81159684 [yarrow]
6 0 0 0 SL crypto_r 0xffffffff8119f3c0 [crypto returns]
5 0 0 0 SL crypto_w 0xffffffff8119f380 [crypto]
4 0 0 0 SL - 0xffffffff81155468 [g_down]
3 0 0 0 SL - 0xffffffff81155460 [g_up]
2 0 0 0 SL - 0xffffffff81155450 [g_event]
13 0 0 0 SL (threaded) ng_queue
64017 D sleep 0xffffffff810ce850 [ng_queue3]
64016 D sleep 0xffffffff810ce850 [ng_queue2]
64015 D sleep 0xffffffff810ce850 [ng_queue1]
64014 D sleep 0xffffffff810ce850 [ng_queue0]
12 0 0 0 WL (threaded) intr
64063 I [irq1: atkbd0]
64050 I [irq16: uhci3]
64045 I [irq18: uhci2]
64040 I [irq19: uhci1+]
64035 I [irq23: cas2 uhci0+]
64034 I [irq9: acpi0]
64033 I [swi5: +]
64028 I [swi2: cambio]
64027 I [swi6: task queue]
64026 I [swi6: Giant taskq]
64013 I [swi4: clock]
64012 I [swi4: clock]
64011 I [swi4: clock]
64010 I [swi4: clock]
64009 I [swi1: netisr 0]
64008 I [swi3: vm]
11 0 0 0 RL (threaded) idle
64007 CanRun [idle: cpu0]
64006 Run CPU 1 [idle: cpu1]
64005 CanRun [idle: cpu2]
64004 Run CPU 3 [idle: cpu3]
1 0 1 0 SLs wait 0xffffff00023558c0 [init]
10 0 0 0 SL audit_wo 0xffffffff8119f8f0 [audit]
0 0 0 0 SLs (threaded) kernel
64062 D - 0xffffff0004004d80 [cas3 taskq]
64061 D - 0xffffff0004004480 [cas2 taskq]
64060 D - 0xffffff0004005700 [cas1 taskq]
64059 D - 0xffffff0003b2b280 [cas0 taskq]
64032 D - 0xffffff000249db80 [kqueue taskq]
64031 D - 0xffffff000249dc00 [acpi_task_2]
64030 D - 0xffffff000249dc00 [acpi_task_1]
64029 D - 0xffffff000249dc00 [acpi_task_0]
64025 D - 0xffffff0002441500

corvey

PART 2 of crash:

options GEOM_MIRROR
options DDB
options KDB
options USB_DEBUG
options AH_SUPPORT_AR5416
options IEEE80211_SUPPORT_MESH
options IEEE80211_AMPDU_AGE
options IEEE80211_DEBUG
options AHD_REG_PRETTY_PRINT
options AHC_REG_PRETTY_PRINT
options ATA_STATIC_ID
options SMP
options INCLUDE_CONFIG_FILE
options AUDIT
options HWPMC_HOOKS
options KBD_INSTALL_CDEV
options PRINTF_BUFR_SIZE=128
options _KPOSIX_PRIORITY_SCHEDULING
options P1003_1B_SEMAPHORES
options SYSVSEM
options SYSVMSG
options SYSVSHM
options STACK
options KTRACE
options SCSI_DELAY=5000
options COMPAT_FREEBSD7
options COMPAT_FREEBSD32
options GEOM_LABEL
options GEOM_PART_GPT
options PSEUDOFS
options PROCFS
options CD9660
options MSDOSFS
options NFS_ROOT
options NFSLOCKD
options NFSSERVER
options NFSCLIENT
options MD_ROOT
options UFS_GJOURNAL
options UFS_DIRHASH
options UFS_ACL
options SOFTUPDATES
options FFS
options SCTP
options INET6
options INET
options PREEMPTION
options SCHED_ULE
options GEOM_PART_MBR
options GEOM_PART_EBR_COMPAT
options GEOM_PART_EBR
options GEOM_PART_BSD
device isa
device mem
device io
device uart_ns8250
device cpufreq
device acpi
device pci
device fdc
device ata
device atadisk
device ataraid
device atapicd
device atapifd
device atapist
device ahc
device ahd
device amd
device hptiop
device isp
device mpt
device sym
device trm
device adv
device adw
device aic
device bt
device scbus
device ch
device da
device sa
device cd
device pass
device ses
device amr
device arcmsr
device ciss
device dpt
device hptmv
device hptrr
device iir
device ips
device mly
device twa
device aac
device aacp
device ida
device mfi
device mlx
device twe
device atkbdc
device atkbd
device psm
device kbdmux
device vga
device splash
device sc
device agp
device cbb
device pccard
device cardbus
device uart
device ppc
device ppbus
device lpt
device plip
device ppi
device de
device em
device igb
device ixgbe
device le
device ti
device txp
device vx
device miibus
device ae
device age
device alc
device ale
device bce
device bfe
device bge
device dc
device et
device fxp
device jme
device lge
device msk
device nfe
device nge
device pcn
device re
device rl
device sf
device sge
device sis
device sk
device ste
device stge
device tl
device tx
device vge
device vr
device wb
device xl
device cs
device ed
device ex
device ep
device fe
device sn
device xe
device wlan
device wlan_wep
device wlan_ccmp
device wlan_tkip
device wlan_amrr
device an
device ath
device ath_hal
device ath_rate_sample
device ral
device wi
device loop
device random
device ether
device vlan
device tun
device pty
device md
device gif
device faith
device firmware
device bpf
device uhci
device ohci
device ehci
device usb
device uhid
device ukbd
device ulpt
device umass
device ums
device urio
device uark
device ubsa
device uftdi
device uipaq
device uplcom
device uslcom
device uvisor
device uvscom
device aue
device axe
device cdce
device cue
device kue
device rue
device udav
device rum
device uath
device ural
device zyd
device firewire
device fwe
device fwip
device dcons
device dcons_crom
device enc
device wlan_xauth
device wlan_acl
device iwi
device ipw
device ipwfw
device wpi
device wpifw
device iwn
device iwnfw
device ralfw
device urtw
device mwl
device mwlfw
device malo
device upgt
device bwi
device siba_bwn
device bwn
device run
device carp
device pf
device pflog
device pfsync
device gre
device ufoma
device ucom
device umct
device umodem
device u3g
device if_bridge
device tap
device hme
device speaker
device lagg
device bktr
device safe
device mxge
device cxgb
device cas
device crypto
device cryptodev
device hifn
device ubsec
device padlock
��msgbuf.txt��0600��0��0��36045��12022110777� 7560� ��ustar��root��wheel��Copyright 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-RELEASE-p6 #0: Mon Dec 12 18:15:35 EST 2011
root@FreeBSD_8.0_pfSense_2.0-AMD64.snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8 amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Atom(TM) CPU D510 @ 1.66GHz (1682.64-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x106ca Family = 6 Model = 1c Stepping = 10
Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x40e31d <sse3,dtes64,mon,ds_cpl,tm2,ssse3,cx16,xtpr,pdcm,movbe>AMD Features=0x20100800 <syscall,nx,lm>AMD Features2=0x1 <lahf>TSC: P-state invariant
real memory = 4294967296 (4096 MB)
avail memory = 4085641216 (3896 MB)
ACPI APIC Table: <intel mopnv10n="">FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 HTT threads
cpu0 (BSP): APIC ID: 0
cpu1 (AP/HT): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP/HT): APIC ID: 3
ioapic0: Changing APIC ID to 8
ioapic0 <version 2.0="">irqs 0-23 on motherboard
lapic0: Forcing LINT1 to edge trigger
netisr_init: forcing maxthreads to 1 and bindthreads to 0 for device polling
wlan: mac acl policy registered
ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80470e10, 0) error 1
wpi: You need to read the LICENSE file in /usr/share/doc/legal/intel_wpi/.
wpi: If you agree with the license, set legal.intel_wpi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (wpi_fw, 0xffffffff806548d0, 0) error 1
ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff80470cd0, 0) error 1
ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80470d70, 0) error 1
kbd1 at kbdmux0
cryptosoft0: <software crypto="">on motherboard
padlock0: No ACE support.
acpi0: <intel mopnv10n="">on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
cpu0: <acpi cpu="">on acpi0
cpu1: <acpi cpu="">on acpi0
cpu2: <acpi cpu="">on acpi0
cpu3: <acpi cpu="">on acpi0
acpi_hpet0: <high precision="" event="" timer="">iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
acpi_button0: <sleep button="">on acpi0
pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
pci0: <acpi pci="" bus="">on pcib0
vgapci0: <vga-compatible display="">port 0x20c0-0x20c7 mem 0xf0a00000-0xf0a7ffff,0xe0000000-0xefffffff,0xf0900000-0xf09fffff irq 16 at device 2.0 on pci0
agp0: <intel pineview="" svga="" controller="">on vgapci0
agp0: detected 8188k stolen memory
agp0: aperture size is 256M
pcib1: <acpi pci-pci="" bridge="">at device 28.0 on pci0
pci1: <acpi pci="" bus="">on pcib1
re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" pcie="" gigabit="" ethernet="">port 0x1000-0x10ff mem 0xf0804000-0xf0804fff,0xf0800000-0xf0803fff irq 16 at device 0.0 on pci1
re0: Using 1 MSI messages
re0: Chip rev. 0x28000000
re0: MAC rev. 0x00000000
miibus0: <mii bus="">on re0
rgephy0: <rtl8169s 8110s="" 8211b="" media="" interface="">PHY 1 on miibus0
rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
re0: [FILTER]
pcib2: <acpi pci-pci="" bridge="">at device 28.1 on pci0
pci2: <acpi pci="" bus="">on pcib2
pcib3: <acpi pci-pci="" bridge="">at device 28.2 on pci0
pci3: <acpi pci="" bus="">on pcib3
pcib4: <acpi pci-pci="" bridge="">at device 28.3 on pci0
pci4: <acpi pci="" bus="">on pcib4
uhci0: <intel 82801g="" (ich7)="" usb="" controller="" usb-a="">port 0x2080-0x209f irq 23 at device 29.0 on pci0
uhci0: [ITHREAD]
uhci0: LegSup = 0x2f00
usbus0: <intel 82801g="" (ich7)="" usb="" controller="" usb-a="">on uhci0
uhci1: <intel 82801g="" (ich7)="" usb="" controller="" usb-b="">port 0x2060-0x207f irq 19 at device 29.1 on pci0
uhci1: [ITHREAD]
uhci1: LegSup = 0x2f00
usbus1: <intel 82801g="" (ich7)="" usb="" controller="" usb-b="">on uhci1
uhci2: <intel 82801g="" (ich7)="" usb="" controller="" usb-c="">port 0x2040-0x205f irq 18 at device 29.2 on pci0
uhci2: [ITHREAD]
uhci2: LegSup = 0x2f00
usbus2: <intel 82801g="" (ich7)="" usb="" controller="" usb-c="">on uhci2
uhci3: <intel 82801g="" (ich7)="" usb="" controller="" usb-d="">port 0x2020-0x203f irq 16 at device 29.3 on pci0
uhci3: [ITHREAD]
uhci3: LegSup = 0x2f00
usbus3: <intel 82801g="" (ich7)="" usb="" controller="" usb-d="">on uhci3
ehci0: <intel 82801gb="" r="" (ich7)="" usb="" 2.0="" controller="">mem 0xf0a80400-0xf0a807ff irq 23 at device 29.7 on pci0
ehci0: [ITHREAD]
usbus4: EHCI version 1.0
usbus4: <intel 82801gb="" r="" (ich7)="" usb="" 2.0="" controller="">on ehci0
pcib5: <acpi pci-pci="" bridge="">at device 30.0 on pci0
pci5: <acpi pci="" bus="">on pcib5
pcib6: <pci-pci bridge="">at device 0.0 on pci5
pci6: <pci bus="">on pcib6
cas0: <ns dp83065="" saturn="" gigabit="" ethernet="">mem 0xf0600000-0xf07fffff irq 21 at device 0.0 on pci6
miibus1: <mii bus="">on cas0
nsgphy0: <dp83865 10="" 100="" 1000="" media="" interface="">PHY 1 on miibus1
nsgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
cas0: 16kB RX FIFO, 9kB TX FIFO
cas0: [FILTER]
cas1: <ns dp83065="" saturn="" gigabit="" ethernet="">mem 0xf0400000-0xf05fffff irq 22 at device 1.0 on pci6
miibus2: <mii bus="">on cas1
nsgphy1: <dp83865 10="" 100="" 1000="" media="" interface="">PHY 1 on miibus2
nsgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
cas1: 16kB RX FIFO, 9kB TX FIFO
cas1: [FILTER]
cas2: <ns dp83065="" saturn="" gigabit="" ethernet="">mem 0xf0200000-0xf03fffff irq 23 at device 2.0 on pci6
miibus3: <mii bus="">on cas2
nsgphy2: <dp83865 10="" 100="" 1000="" media="" interface="">PHY 1 on miibus3
nsgphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
cas2: 16kB RX FIFO, 9kB TX FIFO
cas2: [FILTER]
cas3: <ns dp83065="" saturn="" gigabit="" ethernet="">mem 0xf0000000-0xf01fffff irq 20 at device 3.0 on pci6
miibus4: <mii bus="">on cas3
nsgphy3: <dp83865 10="" 100="" 1000="" media="" interface="">PHY 1 on miibus4
nsgphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
cas3: 16kB RX FIFO, 9kB TX FIFO
cas3: [FILTER]
isab0: <pci-isa bridge="">at device 31.0 on pci0
isa0: <isa bus="">on isab0
atapci0: <intel ich7="" sata300="" controller="">port 0x20b8-0x20bf,0x20cc-0x20cf,0x20b0-0x20b7,0x20c8-0x20cb,0x20a0-0x20af mem 0xf0a80000-0xf0a803ff irq 19 at device 31.2 on pci0
atapci0: [ITHREAD]
atapci0: AHCI called from vendor specific driver
atapci0: AHCI v1.10 controller with 4 3Gbps ports, PM not supported
ata2: <ata 0="" channel="">on atapci0
ata2: [ITHREAD]
ata3: <ata 1="" channel="">on atapci0
ata3: [ITHREAD]
pci0: <serial bus,="" smbus="">at device 31.3 (no driver attached)
atrtc0: <at realtime="" clock="">port 0x70-0x71,0x74-0x77 irq 8 on acpi0
atkbdc0: <keyboard controller="" (i8042)="">port 0x60,0x64 irq 1 on acpi0
atkbd0: <at keyboard="">irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
sc0: <system console="">at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
p4tcc0: <cpu frequency="" thermal="" control="">on cpu0
p4tcc1: <cpu frequency="" thermal="" control="">on cpu1
p4tcc2: <cpu frequency="" thermal="" control="">on cpu2
p4tcc3: <cpu frequency="" thermal="" control="">on cpu3
RTC BIOS diagnostic error 80 <clock_battery>Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 480Mbps High Speed USB v2.0
ugen0.1: <intel>at usbus0
uhub0: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus0
ugen1.1: <intel>at usbus1
uhub1: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus1
ugen2.1: <intel>at usbus2
uhub2: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus2
ugen3.1: <intel>at usbus3
uhub3: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus3
ugen4.1: <intel>at usbus4
uhub4: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus4
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub4: 8 ports with 8 removable, self powered
ugen4.2: <lexar media="">at usbus4
umass0: <lexar 0="" 2="" media="" jd="" firefly,="" class="" 0,="" rev="" 2.00="" 30.00,="" addr="">on usbus4
umass0: SCSI over Bulk-Only; quirks = 0x0000
umass0:0:0 Attached to scbus0
da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
da0: <lexar 3000="" jd="" firefly="">Removable Direct Access SCSI-0 device
da0: 40.000MB/s transfers
da0: 991MB (2030592 512 byte sectors: 64H 32S/T 991C)
lapic1: Forcing LINT1 to edge trigger
SMP: AP CPU #1 Launched!
lapic2: Forcing LINT1 to edge trigger
SMP: AP CPU #2 Launched!
lapic3: Forcing LINT1 to edge trigger
SMP: AP CPU #3 Launched!
Trying to mount root from ufs:/dev/da0s1a
WARNING: / was not properly dismounted
<118>Configuring crash dumps…
<118>Using /dev/da0s1b for dump device.
<118>Mounting filesystems...
WARNING: R/W mount of / denied. Filesystem is not clean - run fsck
<118>mount:
<118>/dev/da0s1a
<118>:
<118>Operation not permitted
<118>** /dev/da0s1a
<118>** Last Mounted on /
<118>** Root file system
<118>** Phase 1 - Check Blocks and Sizes
<118>INCORRECT BLOCK COUNT I=47345 (4 should be 0)
<118>CORRECT? yes
<118>
<118>** Phase 2 - Check Pathnames
<118>** Phase 3 - Check Connectivity
<118>** Phase 4 - Check Reference Counts
<118>UNREF FILE I=47219 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:58 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47321 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:57 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47322 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:57 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47323 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:57 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47324 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:57 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47325 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:57 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47337 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:58 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47339 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:58 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47342 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:58 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47346 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:58 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47347 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:58 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47348 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:58 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47350 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:58 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47352 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:58 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47353 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:58 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47365 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:58 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47368 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:58 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47369 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:58 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47370 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:58 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47371 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 19:58 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47373 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 20:01 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47377 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 20:01 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47386 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 20:01 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47387 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 20:01 2012
<118>CLEAR? yes
<118>
<118>UNREF FILE I=47388 OWNER=root MODE=100644
<118>SIZE=0 MTIME=Sep 5 20:01 2012
<118>CLEAR? yes
<118>
<118>** Phase 5 - Check Cyl groups
<118>FREE BLK COUNT(S) WRONG IN SUPERBLK
<118>SALVAGE? yes
<118>
<118>SUMMARY INFORMATION BAD
<118>SALVAGE? yes
<118>
<118>BLK(S) MISSING IN BIT MAPS
<118>SALVAGE? yes
<118>
<118>8375 files, 148061 used, 213526 free (142 frags, 26673 blocks, 0.0% fragmentation)
<118>
<118>***** FILE SYSTEM MARKED CLEAN *****
<118>
<118>***** FILE SYSTEM WAS MODIFIED *****
<118>
<118> ___
<118> / f
<118>/ p _/ Sense
<118>_/
<118> __/
<118>
<118>
<118>Welcome to pfSense 2.0.1-RELEASE ...
<118>
<118>No core dumps found.
<118>Creating symlinks...
<118>.
<118>.
<118>.
<118>done.
<118>External config loader 1.0 is now starting...
<118> da0s1b
<118>
<118>Launching the init system...
<118> done.
<118>Initializing...
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118> done.
<118>Starting device manager (devd)...
<118>done.
<118>Loading configuration...
<118>.
<118>.
<118>.
<118>done.
<118>Updating configuration...
<118>done.
<118>Cleaning backup cache...
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>.
<118>done.
<118>Setting up extended sysctls...
<118>done.
<118>Setting timezone...
<118>done.
<118>Starting Secure Shell Services...
<118>done.
<118>Setting up polling defaults...
<118>done.
<118>Setting up interfaces microcode...
<118>done.
<118>Configuring LAGG interfaces...
<118>done.
<118>Configuring VLAN interfaces...
<118>done.
<118>Configuring QinQ interfaces...
<118>done.
<118>Configuring WAN interface...
<118>done.
<118>Configuring LAN interface...
<118>done.
<118>Configuring OPT1 interface...
<118>done.
<118>Configuring OPT2 interface...
<118>done.
<118>Syncing OpenVPN settings...
<118>done.
<118>Starting syslog...
<118>done.
<118>Configuring firewall
<118>.

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x62
fault code = supervisor write data, page not present
instruction pointer = 0x20:0xffffffff809b6b41
stack pointer = 0x28:0xffffff803fe24850
frame pointer = 0x28:0xffffff803fe248e0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 24640 (syslogd)

0xffffff009d5341d8: tag ufs, type VREG
usecount 4, writecount 0, refcount 6 mountedhere 0
flags (VV_TEXT)
v_object 0xffffff009d621438 ref 2 pages 30
lock type ufs: SHARED (count 1)
ino 3471, on dev da0s1a
��version.txt��0600��0��0��240��12022110777� 7606� ��ustar��root��wheel��FreeBSD 8.1-RELEASE-p6 #0: Mon Dec 12 18:15:35 EST 2011
root@FreeBSD_8.0_pfSense_2.0-AMD64.snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8
��</lexar></lexar></lexar></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></clock_battery></cpu></cpu></cpu></cpu></generic></system></at></keyboard></at></serial></ata></ata></intel></isa></pci-isa></dp83865></mii></ns></dp83865></mii></ns></dp83865></mii></ns></dp83865></mii></ns></pci></pci-pci></acpi></acpi></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></acpi></acpi></acpi></acpi></acpi></acpi></rtl8169s></mii></realtek></acpi></acpi></intel></vga-compatible></acpi></acpi></sleep></high></acpi></acpi></acpi></acpi></intel></software></version></intel ></lahf></syscall,nx,lm></sse3,dtes64,mon,ds_cpl,tm2,ssse3,cx16,xtpr,pdcm,movbe></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>

heper

is this a new install ? is this new hardware ? is this the first time after long time use?

i've only seen these types of kernel crashes in 2 cases:

-a buggy driver (hardware not fully supported or related)
-failty hardware

jimp

The crash happened in ufs, apparently. That means one of three things:
1. Your HDD/SSD/CF is failing
2. The contents of your HDD/SSD/CF are corrupt, perhaps because of #1
3. Some other bit of hardware failed (memory, thermal, etc), causing the OS to believe that either #1 or #2 happened.

corvey

The system is running a Intel d510mo Atom board. I bought a used Sun quad port card last week and installed it a couple of days ago on a fresh Pfsense 2.0.1 install and it worked right out of the box, which is what I needed. Installed Snort and ac ouple of other packages liek wigets, etc.. The install is on a USB stick, but that could be getting really old by now I guess. I will try a new USB stick. I hope it's not the quad card failing already. Bad drivers for the quad card maybe?

Thank you for checking the log out guys, I really appreciate it.

jimp

Possible, but doubtful. If it were, then the crash would have been in igb or somewhere in networking at least, or perhaps memory allocation. But the crash was in UFS, which is the filesystem, and that code is solid in FreeBSD. A crash there is nearly always hardware, and most likely the storage.

I'd definitely suspect the USB stick at that point.

corvey

Thank you. I will be updating to a new USB drive.

corvey

UPDATE:

I replace the flash drive with a new one, same crash. Replaced the Power supply with a beefy 500 watt that is a known good one, still crash. I took the Sun quad port card nic out and it's been running every since.

The problem has to be with the card itself or the drivers… SIGH...