Pfsense crashed at 8am. Can anyone help me figure out what caused it?



  • Can anyone determine what caused the crash; a hardware or software package add-on perhaps?  This morning at 8am the internet was gone while I surfing previously the moment before, but I discovered pfsense was frozen 99.99% (still a 8 second delay in caps lock and num lock) and when I tried use reboot in the console I couldn't access the keys, so I hit the reset button and it said NOT READY, so I waited for like 10 minutes, still not ready it says.  I said screw it, I need internet and forced a power down.  I booted it back up and pfsense reboot itself once and then booted up fine the second time.   Then when I went into the Pfsense url, it gave me the crash report.

    –--------------------------------------------------------------------------------
    Crash report begins.  Anonymous machine information:

    amd64
    8.1-RELEASE-p6
    FreeBSD 8.1-RELEASE-p6 #0: Mon Dec 12 18:15:35 EST 2011     root@FreeBSD_8.0_pfSense_2.0-AMD64.snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8

    Crash report details:

    Filename: /var/crash/bounds
    1

    Filename: /var/crash/info.0
    Dump header from device /dev/da0s1b
     Architecture: amd64
     Architecture Version: 1
     Dump Length: 62976B (0 MB)
     Blocksize: 512
     Dumptime: Thu Sep  6 12:07:27 2012
     Hostname: brick.localdomain
     Magic: FreeBSD Text Dump
     Version String: FreeBSD 8.1-RELEASE-p6 #0: Mon Dec 12 18:15:35 EST 2011
       root@FreeBSD_8.0_pfSense_2.0-AMD64.snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8
     Panic String:
     Dump Parity: 3615442783
     Bounds: 0
     Dump Status: good

    Filename: /var/crash/minfree
    2048

    Filename: /var/crash/textdump.tar.0
    ddb.txt���������������������������������������������������������������������������������������������0600����0�������0�������114346������12022110777�  7107� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������db:0:kdb.enter.default>  run lockinfo
    db:1:lockinfo> show locks
    No such command
    db:1:locks>  show alllocks
    No such command
    db:1:alllocks>  show lockedvnods
    Locked vnodes
    db:0:kdb.enter.default>  show pcpu
    cpuid        = 2
    dynamic pcpu    = 0xffffff807ef11300
    curthread    = 0xffffff009d49bba0: pid 24640 "syslogd"
    curpcb       = 0xffffff803fe24d40
    fpcurthread  = none
    idlethread   = 0xffffff00023583e0: pid 11 "idle: cpu2"
    curpmap         = 0
    tssp            = 0xffffffff811dabd0
    commontssp      = 0xffffffff811dabd0
    rsp0            = 0xffffff803fe24d40
    gs32p           = 0xffffffff811d9a08
    ldt             = 0xffffffff811d9a48
    tss             = 0xffffffff811d9a38
    db:0:kdb.enter.default>  bt
    Tracing pid 24640 tid 64115 td 0xffffff009d49bba0
    vm_page_flag_set() at vm_page_flag_set+0x1
    vm_fault() at vm_fault+0x14dc
    trap_pfault() at trap_pfault+0x132
    trap() at trap+0x4dc
    calltrap() at calltrap+0x8
    --- trap 0xc, rip = 0x80050d8ee, rsp = 0x7fffffffd980, rbp = 0x800539000 ---
    db:0:kdb.enter.default>  ps
     pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
    30454   262    24     0  LL+    *vm page  0xffffff0003602780 sh
    24640 24073 24640     0  RLs     CPU 2                       syslogd
    24328     1    24     0  S+      piperd   0xffffff009d482b60 logger
    24255     1    24     0  RL+     CPU 0                       tcpdump
    24073     1    24     0  S+      wait     0xffffff009d4df000 initial thread
    17648     1 17648    65  Ss      select   0xffffff009d40bb40 dhclient
    8668     1  8668     0  Ss      select   0xffffff009d2e9bc0 dhclient
     270     1   270     0  Ss      select   0xffffff009d2eb3c0 devd
     262    24    24     0  S+      piperd   0xffffff009d3442d8 initial thread
     259   257   257     0  S       kqread   0xffffff009d2e6000 check_reload_status
     257     1   257     0  Ss      kqread   0xffffff009d2e6700 check_reload_status
      39     0     0     0  SL      mdwait   0xffffff0004fad800 [md0]
      24     1    24     0  Ss+     wait     0xffffff009d27e460 sh
      23     0     0     0  SL      sdflush  0xffffffff811a0498 [softdepflush]
      22     0     0     0  SL      syncer   0xffffffff8117dbc0 [syncer]
      21     0     0     0  SL      vlruwt   0xffffff009d261460 [vnlru]
      20     0     0     0  SL      psleep   0xffffffff8117d6e8 [bufdaemon]
      19     0     0     0  SL      pollid   0xffffffff811587c8 [idlepoll]
      18     0     0     0  SL      pgzero   0xffffffff811a1f2c [pagezero]
      17     0     0     0  SL      psleep   0xffffffff811a12c8 [vmdaemon]
      16     0     0     0  SL      psleep   0xffffffff811a128c [pagedaemon]
       9     0     0     0  SL      ccb_scan 0xffffffff8111f760 [xpt_thrd]
       8     0     0     0  SL      pftm     0xffffffff80204b30 [pfpurge]
       7     0     0     0  SL      waiting_ 0xffffffff81189b80 [sctp_iterator]
      15     0     0     0  SL      (threaded)                  usb
    64058                   D       -        0xffffff80003d9dd0 [usbus4]
    64057                   D       -        0xffffff80003d9d78 [usbus4]
    64056                   D       -        0xffffff80003d9d20 [usbus4]
    64055                   D       -        0xffffff80003d9cc8 [usbus4]
    64054                   D       -        0xffffff80003d0ef0 [usbus3]
    64053                   D       -        0xffffff80003d0e98 [usbus3]
    64052                   D       -        0xffffff80003d0e40 [usbus3]
    64051                   D       -        0xffffff80003d0de8 [usbus3]
    64049                   D       -        0xffffff80003c7ef0 [usbus2]
    64048                   D       -        0xffffff80003c7e98 [usbus2]
    64047                   D       -        0xffffff80003c7e40 [usbus2]
    64046                   D       -        0xffffff80003c7de8 [usbus2]
    64044                   D       -        0xffffff80003beef0 [usbus1]
    64043                   D       -        0xffffff80003bee98 [usbus1]
    64042                   D       -        0xffffff80003bee40 [usbus1]
    64041                   D       -        0xffffff80003bede8 [usbus1]
    64039                   D       -        0xffffff80003b5ef0 [usbus0]
    64038                   D       -        0xffffff80003b5e98 [usbus0]
    64037                   D       -        0xffffff80003b5e40 [usbus0]
    64036                   D       -        0xffffff80003b5de8 [usbus0]
      14     0     0     0  SL      -        0xffffffff81159684 [yarrow]
       6     0     0     0  SL      crypto_r 0xffffffff8119f3c0 [crypto returns]
       5     0     0     0  SL      crypto_w 0xffffffff8119f380 [crypto]
       4     0     0     0  SL      -        0xffffffff81155468 [g_down]
       3     0     0     0  SL      -        0xffffffff81155460 [g_up]
       2     0     0     0  SL      -        0xffffffff81155450 [g_event]
      13     0     0     0  SL      (threaded)                  ng_queue
    64017                   D       sleep    0xffffffff810ce850 [ng_queue3]
    64016                   D       sleep    0xffffffff810ce850 [ng_queue2]
    64015                   D       sleep    0xffffffff810ce850 [ng_queue1]
    64014                   D       sleep    0xffffffff810ce850 [ng_queue0]
      12     0     0     0  WL      (threaded)                  intr
    64063                   I                                   [irq1: atkbd0]
    64050                   I                                   [irq16: uhci3]
    64045                   I                                   [irq18: uhci2]
    64040                   I                                   [irq19: uhci1+]
    64035                   I                                   [irq23: cas2 uhci0+]
    64034                   I                                   [irq9: acpi0]
    64033                   I                                   [swi5: +]
    64028                   I                                   [swi2: cambio]
    64027                   I                                   [swi6: task queue]
    64026                   I                                   [swi6: Giant taskq]
    64013                   I                                   [swi4: clock]
    64012                   I                                   [swi4: clock]
    64011                   I                                   [swi4: clock]
    64010                   I                                   [swi4: clock]
    64009                   I                                   [swi1: netisr 0]
    64008                   I                                   [swi3: vm]
      11     0     0     0  RL      (threaded)                  idle
    64007                   CanRun                              [idle: cpu0]
    64006                   Run     CPU 1                       [idle: cpu1]
    64005                   CanRun                              [idle: cpu2]
    64004                   Run     CPU 3                       [idle: cpu3]
       1     0     1     0  SLs     wait     0xffffff00023558c0 [init]
      10     0     0     0  SL      audit_wo 0xffffffff8119f8f0 [audit]
       0     0     0     0  SLs     (threaded)                  kernel
    64062                   D       -        0xffffff0004004d80 [cas3 taskq]
    64061                   D       -        0xffffff0004004480 [cas2 taskq]
    64060                   D       -        0xffffff0004005700 [cas1 taskq]
    64059                   D       -        0xffffff0003b2b280 [cas0 taskq]
    64032                   D       -        0xffffff000249db80 [kqueue taskq]
    64031                   D       -        0xffffff000249dc00 [acpi_task_2]
    64030                   D       -        0xffffff000249dc00 [acpi_task_1]
    64029                   D       -        0xffffff000249dc00 [acpi_task_0]
    64025                   D       -        0xffffff0002441500



  • PART 2 of crash:

    options GEOM_MIRROR
    options DDB
    options KDB
    options USB_DEBUG
    options AH_SUPPORT_AR5416
    options IEEE80211_SUPPORT_MESH
    options IEEE80211_AMPDU_AGE
    options IEEE80211_DEBUG
    options AHD_REG_PRETTY_PRINT
    options AHC_REG_PRETTY_PRINT
    options ATA_STATIC_ID
    options SMP
    options INCLUDE_CONFIG_FILE
    options AUDIT
    options HWPMC_HOOKS
    options KBD_INSTALL_CDEV
    options PRINTF_BUFR_SIZE=128
    options KPOSIX_PRIORITY_SCHEDULING
    options P1003_1B_SEMAPHORES
    options SYSVSEM
    options SYSVMSG
    options SYSVSHM
    options STACK
    options KTRACE
    options SCSI_DELAY=5000
    options COMPAT_FREEBSD7
    options COMPAT_FREEBSD32
    options GEOM_LABEL
    options GEOM_PART_GPT
    options PSEUDOFS
    options PROCFS
    options CD9660
    options MSDOSFS
    options NFS_ROOT
    options NFSLOCKD
    options NFSSERVER
    options NFSCLIENT
    options MD_ROOT
    options UFS_GJOURNAL
    options UFS_DIRHASH
    options UFS_ACL
    options SOFTUPDATES
    options FFS
    options SCTP
    options INET6
    options INET
    options PREEMPTION
    options SCHED_ULE
    options GEOM_PART_MBR
    options GEOM_PART_EBR_COMPAT
    options GEOM_PART_EBR
    options GEOM_PART_BSD
    device isa
    device mem
    device io
    device uart_ns8250
    device cpufreq
    device acpi
    device pci
    device fdc
    device ata
    device atadisk
    device ataraid
    device atapicd
    device atapifd
    device atapist
    device ahc
    device ahd
    device amd
    device hptiop
    device isp
    device mpt
    device sym
    device trm
    device adv
    device adw
    device aic
    device bt
    device scbus
    device ch
    device da
    device sa
    device cd
    device pass
    device ses
    device amr
    device arcmsr
    device ciss
    device dpt
    device hptmv
    device hptrr
    device iir
    device ips
    device mly
    device twa
    device aac
    device aacp
    device ida
    device mfi
    device mlx
    device twe
    device atkbdc
    device atkbd
    device psm
    device kbdmux
    device vga
    device splash
    device sc
    device agp
    device cbb
    device pccard
    device cardbus
    device uart
    device ppc
    device ppbus
    device lpt
    device plip
    device ppi
    device de
    device em
    device igb
    device ixgbe
    device le
    device ti
    device txp
    device vx
    device miibus
    device ae
    device age
    device alc
    device ale
    device bce
    device bfe
    device bge
    device dc
    device et
    device fxp
    device jme
    device lge
    device msk
    device nfe
    device nge
    device pcn
    device re
    device rl
    device sf
    device sge
    device sis
    device sk
    device ste
    device stge
    device tl
    device tx
    device vge
    device vr
    device wb
    device xl
    device cs
    device ed
    device ex
    device ep
    device fe
    device sn
    device xe
    device wlan
    device wlan_wep
    device wlan_ccmp
    device wlan_tkip
    device wlan_amrr
    device an
    device ath
    device ath_hal
    device ath_rate_sample
    device ral
    device wi
    device loop
    device random
    device ether
    device vlan
    device tun
    device pty
    device md
    device gif
    device faith
    device firmware
    device bpf
    device uhci
    device ohci
    device ehci
    device usb
    device uhid
    device ukbd
    device ulpt
    device umass
    device ums
    device urio
    device uark
    device ubsa
    device uftdi
    device uipaq
    device uplcom
    device uslcom
    device uvisor
    device uvscom
    device aue
    device axe
    device cdce
    device cue
    device kue
    device rue
    device udav
    device rum
    device uath
    device ural
    device zyd
    device firewire
    device fwe
    device fwip
    device dcons
    device dcons_crom
    device enc
    device wlan_xauth
    device wlan_acl
    device iwi
    device ipw
    device ipwfw
    device wpi
    device wpifw
    device iwn
    device iwnfw
    device ralfw
    device urtw
    device mwl
    device mwlfw
    device malo
    device upgt
    device bwi
    device siba_bwn
    device bwn
    device run
    device carp
    device pf
    device pflog
    device pfsync
    device gre
    device ufoma
    device ucom
    device umct
    device umodem
    device u3g
    device if_bridge
    device tap
    device hme
    device speaker
    device lagg
    device bktr
    device safe
    device mxge
    device cxgb
    device cas
    device crypto
    device cryptodev
    device hifn
    device ubsec
    device padlock
    ��������������������������������������������������������������������������������������������������������������������������������������������������������������msgbuf.txt������������������������������������������������������������������������������������������0600����0�������0�������36045�������12022110777�  7560� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������Copyright © 1992-2010 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 8.1-RELEASE-p6 #0: Mon Dec 12 18:15:35 EST 2011
        root@FreeBSD_8.0_pfSense_2.0-AMD64.snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8 amd64
    Timecounter "i8254" frequency 1193182 Hz quality 0
    CPU: Intel(R) Atom(TM) CPU D510  @ 1.66GHz (1682.64-MHz K8-class CPU)
      Origin = "GenuineIntel"  Id = 0x106ca  Family = 6  Model = 1c  Stepping = 10
      Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x40e31d <sse3,dtes64,mon,ds_cpl,tm2,ssse3,cx16,xtpr,pdcm,movbe>AMD Features=0x20100800 <syscall,nx,lm>AMD Features2=0x1 <lahf>TSC: P-state invariant
    real memory  = 4294967296 (4096 MB)
    avail memory = 4085641216 (3896 MB)
    ACPI APIC Table: <intel  mopnv10n="">FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
    FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 HTT threads
    cpu0 (BSP): APIC ID:  0
    cpu1 (AP/HT): APIC ID:  1
    cpu2 (AP): APIC ID:  2
    cpu3 (AP/HT): APIC ID:  3
    ioapic0: Changing APIC ID to 8
    ioapic0 <version 2.0="">irqs 0-23 on motherboard
    lapic0: Forcing LINT1 to edge trigger
    netisr_init: forcing maxthreads to 1 and bindthreads to 0 for device polling
    wlan: mac acl policy registered
    ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80470e10, 0) error 1
    wpi: You need to read the LICENSE file in /usr/share/doc/legal/intel_wpi/.
    wpi: If you agree with the license, set legal.intel_wpi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (wpi_fw, 0xffffffff806548d0, 0) error 1
    ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff80470cd0, 0) error 1
    ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80470d70, 0) error 1
    kbd1 at kbdmux0
    cryptosoft0: <software crypto="">on motherboard
    padlock0: No ACE support.
    acpi0: <intel mopnv10n="">on motherboard
    acpi0: [ITHREAD]
    acpi0: Power Button (fixed)
    Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
    acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
    cpu0: <acpi cpu="">on acpi0
    cpu1: <acpi cpu="">on acpi0
    cpu2: <acpi cpu="">on acpi0
    cpu3: <acpi cpu="">on acpi0
    acpi_hpet0: <high precision="" event="" timer="">iomem 0xfed00000-0xfed003ff on acpi0
    Timecounter "HPET" frequency 14318180 Hz quality 900
    acpi_button0: <sleep button="">on acpi0
    pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
    pci0: <acpi pci="" bus="">on pcib0
    vgapci0: <vga-compatible display="">port 0x20c0-0x20c7 mem 0xf0a00000-0xf0a7ffff,0xe0000000-0xefffffff,0xf0900000-0xf09fffff irq 16 at device 2.0 on pci0
    agp0: <intel pineview="" svga="" controller="">on vgapci0
    agp0: detected 8188k stolen memory
    agp0: aperture size is 256M
    pcib1: <acpi pci-pci="" bridge="">at device 28.0 on pci0
    pci1: <acpi pci="" bus="">on pcib1
    re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" pcie="" gigabit="" ethernet="">port 0x1000-0x10ff mem 0xf0804000-0xf0804fff,0xf0800000-0xf0803fff irq 16 at device 0.0 on pci1
    re0: Using 1 MSI messages
    re0: Chip rev. 0x28000000
    re0: MAC rev. 0x00000000
    miibus0: <mii bus="">on re0
    rgephy0: <rtl8169s 8110s="" 8211b="" media="" interface="">PHY 1 on miibus0
    rgephy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
    re0: [FILTER]
    pcib2: <acpi pci-pci="" bridge="">at device 28.1 on pci0
    pci2: <acpi pci="" bus="">on pcib2
    pcib3: <acpi pci-pci="" bridge="">at device 28.2 on pci0
    pci3: <acpi pci="" bus="">on pcib3
    pcib4: <acpi pci-pci="" bridge="">at device 28.3 on pci0
    pci4: <acpi pci="" bus="">on pcib4
    uhci0: <intel 82801g="" (ich7)="" usb="" controller="" usb-a="">port 0x2080-0x209f irq 23 at device 29.0 on pci0
    uhci0: [ITHREAD]
    uhci0: LegSup = 0x2f00
    usbus0: <intel 82801g="" (ich7)="" usb="" controller="" usb-a="">on uhci0
    uhci1: <intel 82801g="" (ich7)="" usb="" controller="" usb-b="">port 0x2060-0x207f irq 19 at device 29.1 on pci0
    uhci1: [ITHREAD]
    uhci1: LegSup = 0x2f00
    usbus1: <intel 82801g="" (ich7)="" usb="" controller="" usb-b="">on uhci1
    uhci2: <intel 82801g="" (ich7)="" usb="" controller="" usb-c="">port 0x2040-0x205f irq 18 at device 29.2 on pci0
    uhci2: [ITHREAD]
    uhci2: LegSup = 0x2f00
    usbus2: <intel 82801g="" (ich7)="" usb="" controller="" usb-c="">on uhci2
    uhci3: <intel 82801g="" (ich7)="" usb="" controller="" usb-d="">port 0x2020-0x203f irq 16 at device 29.3 on pci0
    uhci3: [ITHREAD]
    uhci3: LegSup = 0x2f00
    usbus3: <intel 82801g="" (ich7)="" usb="" controller="" usb-d="">on uhci3
    ehci0: <intel 82801gb="" r="" (ich7)="" usb="" 2.0="" controller="">mem 0xf0a80400-0xf0a807ff irq 23 at device 29.7 on pci0
    ehci0: [ITHREAD]
    usbus4: EHCI version 1.0
    usbus4: <intel 82801gb="" r="" (ich7)="" usb="" 2.0="" controller="">on ehci0
    pcib5: <acpi pci-pci="" bridge="">at device 30.0 on pci0
    pci5: <acpi pci="" bus="">on pcib5
    pcib6: <pci-pci bridge="">at device 0.0 on pci5
    pci6: <pci bus="">on pcib6
    cas0: <ns dp83065="" saturn="" gigabit="" ethernet="">mem 0xf0600000-0xf07fffff irq 21 at device 0.0 on pci6
    miibus1: <mii bus="">on cas0
    nsgphy0: <dp83865 10="" 100="" 1000="" media="" interface="">PHY 1 on miibus1
    nsgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
    cas0: 16kB RX FIFO, 9kB TX FIFO
    cas0: [FILTER]
    cas1: <ns dp83065="" saturn="" gigabit="" ethernet="">mem 0xf0400000-0xf05fffff irq 22 at device 1.0 on pci6
    miibus2: <mii bus="">on cas1
    nsgphy1: <dp83865 10="" 100="" 1000="" media="" interface="">PHY 1 on miibus2
    nsgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
    cas1: 16kB RX FIFO, 9kB TX FIFO
    cas1: [FILTER]
    cas2: <ns dp83065="" saturn="" gigabit="" ethernet="">mem 0xf0200000-0xf03fffff irq 23 at device 2.0 on pci6
    miibus3: <mii bus="">on cas2
    nsgphy2: <dp83865 10="" 100="" 1000="" media="" interface="">PHY 1 on miibus3
    nsgphy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
    cas2: 16kB RX FIFO, 9kB TX FIFO
    cas2: [FILTER]
    cas3: <ns dp83065="" saturn="" gigabit="" ethernet="">mem 0xf0000000-0xf01fffff irq 20 at device 3.0 on pci6
    miibus4: <mii bus="">on cas3
    nsgphy3: <dp83865 10="" 100="" 1000="" media="" interface="">PHY 1 on miibus4
    nsgphy3:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
    cas3: 16kB RX FIFO, 9kB TX FIFO
    cas3: [FILTER]
    isab0: <pci-isa bridge="">at device 31.0 on pci0
    isa0: <isa bus="">on isab0
    atapci0: <intel ich7="" sata300="" controller="">port 0x20b8-0x20bf,0x20cc-0x20cf,0x20b0-0x20b7,0x20c8-0x20cb,0x20a0-0x20af mem 0xf0a80000-0xf0a803ff irq 19 at device 31.2 on pci0
    atapci0: [ITHREAD]
    atapci0: AHCI called from vendor specific driver
    atapci0: AHCI v1.10 controller with 4 3Gbps ports, PM not supported
    ata2: <ata 0="" channel="">on atapci0
    ata2: [ITHREAD]
    ata3: <ata 1="" channel="">on atapci0
    ata3: [ITHREAD]
    pci0: <serial bus,="" smbus="">at device 31.3 (no driver attached)
    atrtc0: <at realtime="" clock="">port 0x70-0x71,0x74-0x77 irq 8 on acpi0
    atkbdc0: <keyboard controller="" (i8042)="">port 0x60,0x64 irq 1 on acpi0
    atkbd0: <at keyboard="">irq 1 on atkbdc0
    kbd0 at atkbd0
    atkbd0: [GIANT-LOCKED]
    atkbd0: [ITHREAD]
    sc0: <system console="">at flags 0x100 on isa0
    sc0: VGA <16 virtual consoles, flags=0x300>
    vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
    ppc0: cannot reserve I/O port range
    p4tcc0: <cpu frequency="" thermal="" control="">on cpu0
    p4tcc1: <cpu frequency="" thermal="" control="">on cpu1
    p4tcc2: <cpu frequency="" thermal="" control="">on cpu2
    p4tcc3: <cpu frequency="" thermal="" control="">on cpu3
    RTC BIOS diagnostic error 80 <clock_battery>Timecounters tick every 1.000 msec
    IPsec: Initialized Security Association Processing.
    usbus0: 12Mbps Full Speed USB v1.0
    usbus1: 12Mbps Full Speed USB v1.0
    usbus2: 12Mbps Full Speed USB v1.0
    usbus3: 12Mbps Full Speed USB v1.0
    usbus4: 480Mbps High Speed USB v2.0
    ugen0.1: <intel>at usbus0
    uhub0: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus0
    ugen1.1: <intel>at usbus1
    uhub1: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus1
    ugen2.1: <intel>at usbus2
    uhub2: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus2
    ugen3.1: <intel>at usbus3
    uhub3: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus3
    ugen4.1: <intel>at usbus4
    uhub4: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus4
    uhub0: 2 ports with 2 removable, self powered
    uhub1: 2 ports with 2 removable, self powered
    uhub2: 2 ports with 2 removable, self powered
    uhub3: 2 ports with 2 removable, self powered
    uhub4: 8 ports with 8 removable, self powered
    ugen4.2: <lexar media="">at usbus4
    umass0: <lexar 0="" 2="" media="" jd="" firefly,="" class="" 0,="" rev="" 2.00="" 30.00,="" addr="">on usbus4
    umass0:  SCSI over Bulk-Only; quirks = 0x0000
    umass0:0:0👎 Attached to scbus0
    da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
    da0: <lexar 3000="" jd="" firefly="">Removable Direct Access SCSI-0 device
    da0: 40.000MB/s transfers
    da0: 991MB (2030592 512 byte sectors: 64H 32S/T 991C)
    lapic1: Forcing LINT1 to edge trigger
    SMP: AP CPU #1 Launched!
    lapic2: Forcing LINT1 to edge trigger
    SMP: AP CPU #2 Launched!
    lapic3: Forcing LINT1 to edge trigger
    SMP: AP CPU #3 Launched!
    Trying to mount root from ufs:/dev/da0s1a
    WARNING: / was not properly dismounted
    <118>Configuring crash dumps…
    <118>Using /dev/da0s1b for dump device.
    <118>Mounting filesystems...
    WARNING: R/W mount of / denied.  Filesystem is not clean - run fsck
    <118>mount:
    <118>/dev/da0s1a
    <118>:
    <118>Operation not permitted
    <118>** /dev/da0s1a
    <118>** Last Mounted on /
    <118>** Root file system
    <118>** Phase 1 - Check Blocks and Sizes
    <118>INCORRECT BLOCK COUNT I=47345 (4 should be 0)
    <118>CORRECT? yes
    <118>
    <118>** Phase 2 - Check Pathnames
    <118>** Phase 3 - Check Connectivity
    <118>** Phase 4 - Check Reference Counts
    <118>UNREF FILE I=47219  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:58 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47321  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:57 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47322  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:57 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47323  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:57 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47324  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:57 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47325  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:57 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47337  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:58 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47339  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:58 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47342  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:58 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47346  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:58 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47347  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:58 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47348  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:58 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47350  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:58 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47352  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:58 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47353  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:58 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47365  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:58 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47368  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:58 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47369  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:58 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47370  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:58 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47371  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 19:58 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47373  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 20:01 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47377  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 20:01 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47386  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 20:01 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47387  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 20:01 2012
    <118>CLEAR? yes
    <118>
    <118>UNREF FILE I=47388  OWNER=root MODE=100644
    <118>SIZE=0 MTIME=Sep  5 20:01 2012
    <118>CLEAR? yes
    <118>
    <118>** Phase 5 - Check Cyl groups
    <118>FREE BLK COUNT(S) WRONG IN SUPERBLK
    <118>SALVAGE? yes
    <118>
    <118>SUMMARY INFORMATION BAD
    <118>SALVAGE? yes
    <118>
    <118>BLK(S) MISSING IN BIT MAPS
    <118>SALVAGE? yes
    <118>
    <118>8375 files, 148061 used, 213526 free (142 frags, 26673 blocks, 0.0% fragmentation)
    <118>
    <118>***** FILE SYSTEM MARKED CLEAN *****
    <118>
    <118>***** FILE SYSTEM WAS MODIFIED *****
    <118>
    <118>    ___
    <118> / f
    <118>/ p _
    / Sense
    <118>_

    <118>    ___/
    <118>
    <118>
    <118>Welcome to pfSense 2.0.1-RELEASE  ...
    <118>
    <118>No core dumps found.
    <118>Creating symlinks...
    <118>.
    <118>.
    <118>.
    <118>done.
    <118>External config loader 1.0 is now starting...
    <118> da0s1b
    <118>
    <118>Launching the init system...
    <118> done.
    <118>Initializing...
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118> done.
    <118>Starting device manager (devd)...
    <118>done.
    <118>Loading configuration...
    <118>.
    <118>.
    <118>.
    <118>done.
    <118>Updating configuration...
    <118>done.
    <118>Cleaning backup cache...
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>.
    <118>done.
    <118>Setting up extended sysctls...
    <118>done.
    <118>Setting timezone...
    <118>done.
    <118>Starting Secure Shell Services...
    <118>done.
    <118>Setting up polling defaults...
    <118>done.
    <118>Setting up interfaces microcode...
    <118>done.
    <118>Configuring LAGG interfaces...
    <118>done.
    <118>Configuring VLAN interfaces...
    <118>done.
    <118>Configuring QinQ interfaces...
    <118>done.
    <118>Configuring WAN interface...
    <118>done.
    <118>Configuring LAN interface...
    <118>done.
    <118>Configuring OPT1 interface...
    <118>done.
    <118>Configuring OPT2 interface...
    <118>done.
    <118>Syncing OpenVPN settings...
    <118>done.
    <118>Starting syslog...
    <118>done.
    <118>Configuring firewall
    <118>.

    Fatal trap 12: page fault while in kernel mode
    cpuid = 0; apic id = 00
    fault virtual address = 0x62
    fault code = supervisor write data, page not present
    instruction pointer = 0x20:0xffffffff809b6b41
    stack pointer         = 0x28:0xffffff803fe24850
    frame pointer         = 0x28:0xffffff803fe248e0
    code segment = base 0x0, limit 0xfffff, type 0x1b
    = DPL 0, pres 1, long 1, def32 0, gran 1
    processor eflags = interrupt enabled, resume, IOPL = 0
    current process = 24640 (syslogd)

    0xffffff009d5341d8: tag ufs, type VREG
        usecount 4, writecount 0, refcount 6 mountedhere 0
        flags (VV_TEXT)
        v_object 0xffffff009d621438 ref 2 pages 30
        lock type ufs: SHARED (count 1)
    ino 3471, on dev da0s1a
    �������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������version.txt�����������������������������������������������������������������������������������������0600����0�������0�������240���������12022110777�  7606� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������FreeBSD 8.1-RELEASE-p6 #0: Mon Dec 12 18:15:35 EST 2011
        root@FreeBSD_8.0_pfSense_2.0-AMD64.snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8
    ����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������</lexar></lexar></lexar></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></clock_battery></cpu></cpu></cpu></cpu></generic></system></at></keyboard></at></serial></ata></ata></intel></isa></pci-isa></dp83865></mii></ns></dp83865></mii></ns></dp83865></mii></ns></dp83865></mii></ns></pci></pci-pci></acpi></acpi></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></acpi></acpi></acpi></acpi></acpi></acpi></rtl8169s></mii></realtek></acpi></acpi></intel></vga-compatible></acpi></acpi></sleep></high></acpi></acpi></acpi></acpi></intel></software></version></intel ></lahf></syscall,nx,lm></sse3,dtes64,mon,ds_cpl,tm2,ssse3,cx16,xtpr,pdcm,movbe></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>



  • is this a new install ? is this new hardware ? is this the first time after long time use?

    i've only seen these types of kernel crashes in 2 cases:

    -a buggy driver (hardware not fully supported or related)
    -failty hardware


  • Rebel Alliance Developer Netgate

    The crash happened in ufs, apparently. That means one of three things:
    1. Your HDD/SSD/CF is failing
    2. The contents of your HDD/SSD/CF are corrupt, perhaps because of #1
    3. Some other bit of hardware failed (memory, thermal, etc), causing the OS to believe that either #1 or #2 happened.



  • The system is running a Intel d510mo Atom board.  I bought a used Sun quad port card last week and installed it a couple of days ago on a fresh Pfsense 2.0.1 install and it worked right out of the box, which is what I needed.  Installed Snort and ac ouple of other packages liek wigets, etc.. The install is on a USB stick, but that could be getting really old by now I guess.  I will try a new USB stick. I hope it's not the quad card failing already.   Bad drivers for the quad card maybe?

    Thank you for checking the log out guys, I really appreciate it.


  • Rebel Alliance Developer Netgate

    Possible, but doubtful. If it were, then the crash would have been in igb or somewhere in networking at least, or perhaps memory allocation. But the crash was in UFS, which is the filesystem, and that code is solid in FreeBSD. A crash there is nearly always hardware, and most likely the storage.

    I'd definitely suspect the USB stick at that point.



  • Thank you.  I will be updating to a new USB drive.



  • UPDATE:

    I replace the flash drive with a new one, same crash.  Replaced the Power supply with a beefy 500 watt that is a known good one, still crash.  I took the Sun quad port card nic out and it's been running every since.

    The problem has to be with the card itself or the drivers… SIGH...


Locked