Kernel panic



  • Всем доброго времени суток.

    В первый рабочий день после отпуска в одном из филиалов поймал kernel panic.

    Помогите разобраться куда копать… Как лечить?

    msgbuf.txt

    
    Copyright (c) 1992-2010 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    	The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 8.1-RELEASE-p6 #0: Mon Dec 12 17:53:00 EST 2011
        root@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8 i386
    Timecounter "i8254" frequency 1193182 Hz quality 0
    CPU: Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz (3193.02-MHz 686-class CPU)
      Origin = "GenuineIntel"  Id = 0x206a7  Family = 6  Model = 2a  Stepping = 7
      Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x159ae3bf<sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,<b17>,SSE4.1,SSE4.2,POPCNT,<b24>,XSAVE,<b28>>
      AMD Features=0x28100000 <nx,rdtscp,lm>AMD Features2=0x1 <lahf>TSC: P-state invariant
    real memory  = 2147483648 (2048 MB)
    avail memory = 2064588800 (1968 MB)
    ACPI APIC Table: <alaska a="" m="" i="">FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
    FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 SMT threads
     cpu0 (BSP): APIC ID:  0
     cpu1 (AP): APIC ID:  1
     cpu2 (AP): APIC ID:  2
     cpu3 (AP): APIC ID:  3
    ioapic0 <version 2.0="">irqs 0-23 on motherboard
    netisr_init: forcing maxthreads to 1 and bindthreads to 0 for device polling
    wpi: You need to read the LICENSE file in /usr/share/doc/legal/intel_wpi/.
    wpi: If you agree with the license, set legal.intel_wpi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (wpi_fw, 0xc0988300, 0) error 1
    ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_bss_fw, 0xc0789340, 0) error 1
    ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_ibss_fw, 0xc07893e0, 0) error 1
    ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_monitor_fw, 0xc0789480, 0) error 1
    wlan: mac acl policy registered
    kbd1 at kbdmux0
    cryptosoft0: <software crypto="">on motherboard
    padlock0: No ACE support.
    acpi0: <alaska a="" m="" i="">on motherboard
    acpi0: [ITHREAD]
    ACPI Error (psargs-0464): [RAMB] Namespace lookup failure, AE_NOT_FOUND
    ACPI Exception: AE_NOT_FOUND, Could not execute arguments for [RAMW] (Region) (20100331/nsinit-442)
    acpi0: Power Button (fixed)
    Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
    acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
    cpu0: <acpi cpu="">on acpi0
    cpu1: <acpi cpu="">on acpi0
    cpu2: <acpi cpu="">on acpi0
    cpu3: <acpi cpu="">on acpi0
    acpi_hpet0: <high precision="" event="" timer="">iomem 0xfed00000-0xfed003ff on acpi0
    Timecounter "HPET" frequency 14318180 Hz quality 900
    pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
    pci0: <acpi pci="" bus="">on pcib0
    pcib1: <acpi pci-pci="" bridge="">irq 16 at device 1.0 on pci0
    pci1: <acpi pci="" bus="">on pcib1
    vgapci0: <vga-compatible display="">port 0xe000-0xe0ff mem 0xc0000000-0xcfffffff,0xfe620000-0xfe63ffff irq 16 at device 0.0 on pci1
    pci1: <multimedia, hda="">at device 0.1 (no driver attached)
    pci0: <simple comms="">at device 22.0 (no driver attached)
    ehci0: <ehci (generic)="" usb="" 2.0="" controller="">mem 0xfe702000-0xfe7023ff irq 23 at device 26.0 on pci0
    ehci0: [ITHREAD]
    usbus0: EHCI version 1.0
    usbus0: <ehci (generic)="" usb="" 2.0="" controller="">on ehci0
    pcib2: <acpi pci-pci="" bridge="">irq 17 at device 28.0 on pci0
    pci2: <acpi pci="" bus="">on pcib2
    pcib3: <acpi pci-pci="" bridge="">irq 16 at device 28.1 on pci0
    pci3: <acpi pci="" bus="">on pcib3
    pcib4: <acpi pci-pci="" bridge="">irq 18 at device 28.2 on pci0
    pci4: <acpi pci="" bus="">on pcib4
    re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" pcie="" gigabit="" ethernet="">port 0xd000-0xd0ff mem 0xd0004000-0xd0004fff,0xd0000000-0xd0003fff irq 18 at device 0.0 on pci4
    re0: Using 1 MSI messages
    re0: Chip rev. 0x2c800000
    re0: MAC rev. 0x00000000
    re0: Unknown H/W revision: 0x2c800000
    device_attach: re0 attach returned 6
    pcib5: <acpi pci-pci="" bridge="">irq 19 at device 28.3 on pci0
    pci5: <acpi pci="" bus="">on pcib5
    pcib6: <acpi pci-pci="" bridge="">irq 17 at device 28.4 on pci0
    pci6: <acpi pci="" bus="">on pcib6
    pcib7: <acpi pci-pci="" bridge="">irq 17 at device 0.0 on pci6
    pci7: <acpi pci="" bus="">on pcib7
    rl0: <realtek 10="" 8139="" 100basetx="">port 0xc100-0xc1ff mem 0xfe521000-0xfe5210ff irq 16 at device 0.0 on pci7
    miibus0: <mii bus="">on rl0
    rlphy0: <realtek internal="" media="" interface="">PHY 0 on miibus0
    rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
    rl0: [ITHREAD]
    rl1: <realtek 10="" 8139="" 100basetx="">port 0xc000-0xc0ff mem 0xfe520000-0xfe5200ff irq 17 at device 1.0 on pci7
    miibus1: <mii bus="">on rl1
    rlphy1: <realtek internal="" media="" interface="">PHY 0 on miibus1
    rlphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
    rl1: [ITHREAD]
    pcib8: <acpi pci-pci="" bridge="">irq 16 at device 28.5 on pci0
    pci8: <acpi pci="" bus="">on pcib8
    ehci1: <ehci (generic)="" usb="" 2.0="" controller="">mem 0xfe701000-0xfe7013ff irq 23 at device 29.0 on pci0
    ehci1: [ITHREAD]
    usbus1: EHCI version 1.0
    usbus1: <ehci (generic)="" usb="" 2.0="" controller="">on ehci1
    isab0: <pci-isa bridge="">at device 31.0 on pci0
    isa0: <isa bus="">on isab0
    atapci0: <intel ata="" controller="">port 0xf0d0-0xf0d7,0xf0c0-0xf0c3,0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf090-0xf09f,0xf080-0xf08f irq 20 at device 31.2 on pci0
    atapci0: [ITHREAD]
    ata2: <ata 0="" channel="">on atapci0
    ata2: [ITHREAD]
    ata3: <ata 1="" channel="">on atapci0
    ata3: [ITHREAD]
    pci0: <serial bus,="" smbus="">at device 31.3 (no driver attached)
    atapci1: <intel ata="" controller="">port 0xf070-0xf077,0xf060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf030-0xf03f,0xf020-0xf02f irq 20 at device 31.5 on pci0
    atapci1: [ITHREAD]
    ata4: <ata 0="" channel="">on atapci1
    ata4: [ITHREAD]
    ata5: <ata 1="" channel="">on atapci1
    ata5: [ITHREAD]
    acpi_button0: <power button="">on acpi0
    ppc1: <parallel port="">port 0x378-0x37f irq 5 on acpi0
    ppc1: Generic chipset (NIBBLE-only) in COMPATIBLE mode
    ppc1: [ITHREAD]
    ppbus0: <parallel port="" bus="">on ppc1
    plip0: <plip network="" interface="">on ppbus0
    plip0: [ITHREAD]
    lpt0: <printer>on ppbus0
    lpt0: [ITHREAD]
    lpt0: Interrupt-driven port
    ppi0: <parallel i="" o="">on ppbus0
    atrtc0: <at realtime="" clock="">port 0x70-0x71 irq 8 on acpi0
    uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
    uart0: [FILTER]
    pmtimer0 on isa0
    orm0: <isa option="" rom="">at iomem 0xc0000-0xcffff pnpid ORM0000 on isa0
    sc0: <system console="">at flags 0x100 on isa0
    sc0: VGA <16 virtual consoles, flags=0x300>
    vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
    ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
    ata0: [ITHREAD]
    ata1 at port 0x170-0x177,0x376 irq 15 on isa0
    ata1: [ITHREAD]
    atkbdc0: <keyboard controller="" (i8042)="">at port 0x60,0x64 on isa0
    atkbd0: <at keyboard="">irq 1 on atkbdc0
    kbd0 at atkbd0
    atkbd0: [GIANT-LOCKED]
    atkbd0: [ITHREAD]
    ppc0: cannot reserve I/O port range
    est0: <enhanced speedstep="" frequency="" control="">on cpu0
    p4tcc0: <cpu frequency="" thermal="" control="">on cpu0
    est1: <enhanced speedstep="" frequency="" control="">on cpu1
    p4tcc1: <cpu frequency="" thermal="" control="">on cpu1
    est2: <enhanced speedstep="" frequency="" control="">on cpu2
    p4tcc2: <cpu frequency="" thermal="" control="">on cpu2
    est3: <enhanced speedstep="" frequency="" control="">on cpu3
    p4tcc3: <cpu frequency="" thermal="" control="">on cpu3
    Timecounters tick every 1.000 msec
    IPsec: Initialized Security Association Processing.
    usbus0: 480Mbps High Speed USB v2.0
    usbus1: 480Mbps High Speed USB v2.0
    ad4: 238475MB <st250dm000 1bd141="" kc45="">at ata2-master UDMA33 
    ugen0.1: <intel>at usbus0
    uhub0: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus0
    ugen1.1: <intel>at usbus1
    uhub1: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus1
    SMP: AP CPU #3 Launched!
    SMP: AP CPU #2 Launched!
    SMP: AP CPU #1 Launched!
    Root mount waiting for: usbus1 usbus0
    uhub0: 2 ports with 2 removable, self powered
    uhub1: 2 ports with 2 removable, self powered
    Root mount waiting for: usbus1 usbus0
    ugen0.2: <vendor 0x8087="">at usbus0
    uhub2: <vendor 2="" 9="" 0x8087="" product="" 0x0024,="" class="" 0,="" rev="" 2.00="" 0.00,="" addr="">on usbus0
    ugen1.2: <vendor 0x8087="">at usbus1
    uhub3: <vendor 2="" 9="" 0x8087="" product="" 0x0024,="" class="" 0,="" rev="" 2.00="" 0.00,="" addr="">on usbus1
    Root mount waiting for: usbus1 usbus0
    uhub2: 4 ports with 4 removable, self powered
    uhub3: 6 ports with 6 removable, self powered
    Trying to mount root from ufs:/dev/ad4s1a
    <118>Configuring crash dumps...
    <118>Using /dev/ad4s1b for dump device.
    <118>Mounting filesystems...
    <118>
    <118>     ___
    <118> ___/ f \
    <118>/ p \___/ Sense
    <118>\___/   \
    <118>    \___/
    <118>
    <118>
    <118>Welcome to pfSense 2.0.1-RELEASE  ...
    <118>
    <118>No core dumps found.
    <118>Creating symlinks...
    <118>ln: 
    <118>/tmp/tmp
    <118>: 
    <118>File exists
    <118>rm: 
    <118>/tmp/tmp: Bad file descriptor
    <118>
    <118>.
    free inode //23881745 had 1056 blocks
    panic: ffs_write: type 0xc634653c 0 (0,16384)
    cpuid = 2
    KDB: enter: panic
    
    0xc634653c: tag ufs, type VNON
        usecount 1, writecount 1, refcount 1 mountedhere 0
        flags ()
        lock type ufs: EXCL by thread 0xc6318780 (pid 87)
    	ino 23881774, on dev ad4s1a</vendor></vendor></vendor></vendor></intel></intel></intel></intel></st250dm000></cpu></enhanced></cpu></enhanced></cpu></enhanced></cpu></enhanced></at></keyboard></generic></system></isa></at></parallel></printer></plip></parallel></parallel></power></ata></ata></intel></serial></ata></ata></intel></isa></pci-isa></ehci></ehci></acpi></acpi></realtek></mii></realtek></realtek></mii></realtek></acpi></acpi></acpi></acpi></acpi></acpi></realtek></acpi></acpi></acpi></acpi></acpi></acpi></ehci></ehci></simple></multimedia,></vga-compatible></acpi></acpi></acpi></acpi></high></acpi></acpi></acpi></acpi></alaska></software></version></alaska></lahf></nx,rdtscp,lm></b28></b24></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,<b17></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe> 
    

    Panic.txt

    
    ffs_write: type 0xc634653c 0 (0,16384)
    
    

    Version.txt

    
    FreeBSD 8.1-RELEASE-p6 #0: Mon Dec 12 17:53:00 EST 2011
        root@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8
    
    

    config.txt

    
    options	CONFIG_AUTOGENERATED
    ident	pfSense8_SMP
    machine	i386
    cpu	I686_CPU
    cpu	I586_CPU
    cpu	I486_CPU
    options	ALTQ_NOPCC
    options	PPS_SYNC
    options	ALTQ_FAIRQ
    options	MROUTING
    options	DEVICE_POLLING
    options	NULLFS
    options	MSGTQL=2048
    options	MSGSSZ=32
    options	MSGSEG=512
    options	MSGMNI=40
    options	MSGMNB=8192
    options	ALTQ_PRIQ
    options	ALTQ_HFSC
    options	ALTQ_RIO
    options	ALTQ_RED
    options	ALTQ_CBQ
    options	ALTQ
    options	TCP_SIGNATURE
    options	IPSEC_NAT_T
    options	IPSEC
    options	NETGRAPH_PRED1
    options	NETGRAPH_DEFLATE
    options	NETGRAPH_CAR
    options	NETGRAPH_PIPE
    options	NETGRAPH_TCPMSS
    options	NETGRAPH_VJC
    options	NETGRAPH_UI
    options	NETGRAPH_TTY
    options	NETGRAPH_TEE
    options	NETGRAPH_SOCKET
    options	NETGRAPH_RFC1490
    options	NETGRAPH_PPTPGRE
    options	NETGRAPH_PPPOE
    options	NETGRAPH_PPP
    options	NETGRAPH_ONE2MANY
    options	NETGRAPH_MPPC_ENCRYPTION
    options	NETGRAPH_LMI
    options	NETGRAPH_KSOCKET
    options	NETGRAPH_EIFACE
    options	NETGRAPH_IFACE
    options	NETGRAPH_HOLE
    options	NETGRAPH_FRAME_RELAY
    options	NETGRAPH_ETHER
    options	NETGRAPH_ECHO
    options	NETGRAPH_CISCO
    options	NETGRAPH_BRIDGE
    options	NETGRAPH_BPF
    options	NETGRAPH_ASYNC
    options	NETGRAPH_L2TP
    options	NETGRAPH_VLAN
    options	NETGRAPH
    options	IPSTEALTH
    options	IPFIREWALL_VERBOSE_LIMIT
    options	IPFIREWALL_VERBOSE
    options	IPFIREWALL_FORWARD
    options	IPFIREWALL_DEFAULT_TO_ACCEPT
    options	UNIONFS
    options	GEOM_BDE
    options	GEOM_ELI
    options	GEOM_UZIP
    options	GEOM_MIRROR
    options	DDB
    options	KDB
    options	USB_DEBUG
    options	AH_SUPPORT_AR5416
    options	IEEE80211_SUPPORT_MESH
    options	IEEE80211_AMPDU_AGE
    options	IEEE80211_DEBUG
    options	AHD_REG_PRETTY_PRINT
    options	AHC_REG_PRETTY_PRINT
    options	ATA_STATIC_ID
    options	SMP
    options	INCLUDE_CONFIG_FILE
    options	AUDIT
    options	HWPMC_HOOKS
    options	KBD_INSTALL_CDEV
    options	PRINTF_BUFR_SIZE=128
    options	_KPOSIX_PRIORITY_SCHEDULING
    options	P1003_1B_SEMAPHORES
    options	SYSVSEM
    options	SYSVMSG
    options	SYSVSHM
    options	STACK
    options	KTRACE
    options	SCSI_DELAY=5000
    options	COMPAT_FREEBSD7
    options	GEOM_LABEL
    options	GEOM_PART_GPT
    options	PSEUDOFS
    options	PROCFS
    options	CD9660
    options	MSDOSFS
    options	NFS_ROOT
    options	NFSLOCKD
    options	NFSSERVER
    options	NFSCLIENT
    options	MD_ROOT
    options	UFS_GJOURNAL
    options	UFS_DIRHASH
    options	UFS_ACL
    options	SOFTUPDATES
    options	FFS
    options	SCTP
    options	INET6
    options	INET
    options	PREEMPTION
    options	SCHED_ULE
    options	NATIVE
    options	GEOM_PART_MBR
    options	GEOM_PART_EBR_COMPAT
    options	GEOM_PART_EBR
    options	GEOM_PART_BSD
    options	ISAPNP
    device	isa
    device	npx
    device	mem
    device	io
    device	uart_ns8250
    device	atpic
    device	apic
    device	cpufreq
    device	acpi
    device	eisa
    device	pci
    device	fdc
    device	ata
    device	atadisk
    device	ataraid
    device	atapicd
    device	atapifd
    device	atapist
    device	ahb
    device	ahc
    device	ahd
    device	amd
    device	hptiop
    device	isp
    device	mpt
    device	sym
    device	trm
    device	adv
    device	adw
    device	aha
    device	aic
    device	bt
    device	ncv
    device	nsp
    device	stg
    device	scbus
    device	ch
    device	da
    device	sa
    device	cd
    device	pass
    device	ses
    device	amr
    device	arcmsr
    device	asr
    device	ciss
    device	dpt
    device	hptmv
    device	hptrr
    device	iir
    device	ips
    device	mly
    device	twa
    device	aac
    device	aacp
    device	ida
    device	mfi
    device	mlx
    device	pst
    device	twe
    device	atkbdc
    device	atkbd
    device	psm
    device	kbdmux
    device	vga
    device	splash
    device	sc
    device	agp
    device	pmtimer
    device	cbb
    device	pccard
    device	cardbus
    device	uart
    device	ppc
    device	ppbus
    device	lpt
    device	plip
    device	ppi
    device	de
    device	em
    device	igb
    device	ixgb
    device	le
    device	ti
    device	txp
    device	vx
    device	miibus
    device	ae
    device	age
    device	alc
    device	ale
    device	bce
    device	bfe
    device	bge
    device	dc
    device	et
    device	fxp
    device	jme
    device	lge
    device	msk
    device	nfe
    device	nge
    device	pcn
    device	re
    device	rl
    device	sf
    device	sge
    device	sis
    device	sk
    device	ste
    device	stge
    device	tl
    device	tx
    device	vge
    device	vr
    device	wb
    device	xl
    device	cs
    device	ed
    device	ex
    device	ep
    device	fe
    device	ie
    device	sn
    device	xe
    device	wlan
    device	wlan_wep
    device	wlan_ccmp
    device	wlan_tkip
    device	wlan_amrr
    device	an
    device	ath
    device	ath_hal
    device	ath_rate_sample
    device	ral
    device	wi
    device	loop
    device	random
    device	ether
    device	vlan
    device	tun
    device	pty
    device	md
    device	gif
    device	faith
    device	firmware
    device	bpf
    device	uhci
    device	ohci
    device	ehci
    device	usb
    device	uhid
    device	ukbd
    device	ulpt
    device	umass
    device	ums
    device	urio
    device	u3g
    device	uark
    device	ubsa
    device	uftdi
    device	uipaq
    device	uplcom
    device	uslcom
    device	uvisor
    device	uvscom
    device	aue
    device	axe
    device	cdce
    device	cue
    device	kue
    device	rue
    device	udav
    device	rum
    device	uath
    device	ural
    device	zyd
    device	firewire
    device	fwe
    device	fwip
    device	dcons
    device	dcons_crom
    device	enc
    device	wlan_xauth
    device	wlan_acl
    device	iwi
    device	ipw
    device	ipwfw
    device	wpi
    device	wpifw
    device	iwn
    device	iwnfw
    device	ralfw
    device	urtw
    device	mwl
    device	mwlfw
    device	malo
    device	upgt
    device	bwi
    device	siba_bwn
    device	bwn
    device	run
    device	carp
    device	pf
    device	pflog
    device	pfsync
    device	gre
    device	ufoma
    device	ucom
    device	umct
    device	umodem
    device	if_bridge
    device	tap
    device	hme
    device	speaker
    device	lagg
    device	bktr
    device	safe
    device	mxge
    device	cxgb
    device	ixgbe
    device	cas
    device	crypto
    device	cryptodev
    device	hifn
    device	ubsec
    device	padlock
    
    

    ddb.txt

    
    db:0:kdb.enter.default>  run lockinfo
    db:1:lockinfo> show locks
    No such command
    db:1:locks>  show alllocks
    No such command
    db:1:alllocks>  show lockedvnods
    Locked vnodes
    db:0:kdb.enter.default>  show pcpu
    cpuid        = 2
    dynamic pcpu    = 0x467b880
    curthread    = 0xc6318780: pid 87 "clog"
    curpcb       = 0xc5ba9d90
    fpcurthread  = none
    idlethread   = 0xc5d8f780: pid 11 "idle: cpu2"
    APIC ID      = 2
    currentldt   = 0x50
    db:0:kdb.enter.default>  bt
    Tracing pid 87 tid 64083 td 0xc6318780
    kdb_enter(c0ee668e,c0ee668e,c0f08858,c5ba9a68,2,...) at kdb_enter+0x3b
    panic(c0f08858,c634653c,0,0,4000,...) at panic+0xff
    ffs_write(c5ba9bbc,80400,c6346594,0,c5ba9b54,...) at ffs_write+0x7d
    VOP_WRITE_APV(c12ebec0,c5ba9bbc,c0eee7f0,25f,0,...) at VOP_WRITE_APV+0x96
    vn_write(c62b2700,c5ba9c54,c5d8a400,0,c6318780,...) at vn_write+0x166
    dofilewrite(c5ba9c54,ffffffff,ffffffff,0,c62b2700,...) at dofilewrite+0x84
    kern_writev(c6318780,3,c5ba9c54,c5ba9c74,1,...) at kern_writev+0x6a
    write(c6318780,c5ba9cf8,c,c6318780,c,...) at write+0x50
    syscall(c5ba9d38) at syscall+0x1ad
    Xint0x80_syscall() at Xint0x80_syscall+0x20
    --- syscall (4, FreeBSD ELF32, write), eip = 0x28184573, esp = 0xbfbfad9c, ebp = 0xbfbfede8 ---
    db:0:kdb.enter.default>  ps
      pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
       87    24    24     0  R+      CPU 2               clog
       39     0     0     0  SL      mdwait   0xc6250000 [md0]
       24     1    24     0  Ss+     wait     0xc5e997f8 sh
       23     0     0     0  SL      sdflush  0xc136ac60 [softdepflush]
       22     0     0     0  SL      syncer   0xc134fd94 [syncer]
       21     0     0     0  SL      vlruwt   0xc6204000 [vnlru]
       20     0     0     0  SL      psleep   0xc134fac8 [bufdaemon]
       19     0     0     0  SL      pollid   0xc133b77c [idlepoll]
       18     0     0     0  SL      pgzero   0xc136b914 [pagezero]
       17     0     0     0  SL      psleep   0xc136b53c [vmdaemon]
       16     0     0     0  SL      psleep   0xc136b504 [pagedaemon]
        9     0     0     0  SL      ccb_scan 0xc1306454 [xpt_thrd]
        8     0     0     0  SL      pftm     0xc04f6d80 [pfpurge]
        7     0     0     0  SL      waiting_ 0xc13571d8 [sctp_iterator]
       15     0     0     0  SL      (threaded)          usb
     64045                   D       -        0xc6004d0c [usbus1]
     64044                   D       -        0xc6004cdc [usbus1]
     64043                   D       -        0xc6004cac [usbus1]
     64042                   D       -        0xc6004c7c [usbus1]
     64039                   D       -        0xc5fd5d0c [usbus0]
     64038                   D       -        0xc5fd5cdc [usbus0]
     64037                   D       -        0xc5fd5cac [usbus0]
     64036                   D       -        0xc5fd5c7c [usbus0]
       14     0     0     0  SL      -        0xc133c084 [yarrow]
        6     0     0     0  SL      crypto_r 0xc136a28c [crypto returns]
        5     0     0     0  SL      crypto_w 0xc136a268 [crypto]
        4     0     0     0  SL      -        0xc1339924 [g_down]
        3     0     0     0  SL      -        0xc1339920 [g_up]
        2     0     0     0  SL      -        0xc1339918 [g_event]
       13     0     0     0  SL      (threaded)          ng_queue
     64017                   D       sleep    0xc12dcae0 [ng_queue3]
     64016                   D       sleep    0xc12dcae0 [ng_queue2]
     64015                   D       sleep    0xc12dcae0 [ng_queue1]
     64014                   D       sleep    0xc12dcae0 [ng_queue0]
       12     0     0     0  WL      (threaded)          intr
     64051                   I                           [irq1: atkbd0]
     64050                   I                           [irq15: ata1]
     64049                   I                           [irq14: ata0]
     64048                   I                           [swi0: uart]
     64047                   I                           [irq5: ppc1]
     64046                   I                           [irq20: atapci0+]
     64041                   I                           [irq17: rl1]
     64040                   I                           [irq16: rl0]
     64035                   I                           [irq23: ehci0 ehci1]
     64034                   I                           [irq9: acpi0]
     64030                   I                           [swi6: task queue]
     64029                   I                           [swi6: Giant taskq]
     64027                   I                           [swi5: +]
     64025                   I                           [swi2: cambio]
     64013                   I                           [swi4: clock]
     64012                   I                           [swi4: clock]
     64011                   I                           [swi4: clock]
     64010                   I                           [swi4: clock]
     64009                   I                           [swi1: netisr 0]
     64008                   I                           [swi3: vm]
       11     0     0     0  RL      (threaded)          idle
     64007                   Run     CPU 0               [idle: cpu0]
     64006                   Run     CPU 1               [idle: cpu1]
     64005                   CanRun                      [idle: cpu2]
     64004                   Run     CPU 3               [idle: cpu3]
        1     0     1     0  SLs     wait     0xc5d8dd48 [init]
       10     0     0     0  SL      audit_wo 0xc136a600 [audit]
        0     0     0     0  SLs     (threaded)          kernel
     64033                   D       -        0xc5f5f180 [acpi_task_2]
     64032                   D       -        0xc5f5f180 [acpi_task_1]
     64031                   D       -        0xc5f5f180 [acpi_task_0]
     64028                   D       -        0xc5f5f2c0 [thread]
     64026                   D       -        0xc5f5f680 [kqueue taskq]
     64018                   D       -        0xc5d73600 [firmware taskq]
     64001                   D       sched    0xc1339a00 [swapper]
    db:0:kdb.enter.default>  alltrace
    
    Tracing command clog pid 87 tid 64083 td 0xc6318780
    kdb_enter(c0ee668e,c0ee668e,c0f08858,c5ba9a68,2,...) at kdb_enter+0x3b
    panic(c0f08858,c634653c,0,0,4000,...) at panic+0xff
    ffs_write(c5ba9bbc,80400,c6346594,0,c5ba9b54,...) at ffs_write+0x7d
    VOP_WRITE_APV(c12ebec0,c5ba9bbc,c0eee7f0,25f,0,...) at VOP_WRITE_APV+0x96
    vn_write(c62b2700,c5ba9c54,c5d8a400,0,c6318780,...) at vn_write+0x166
    dofilewrite(c5ba9c54,ffffffff,ffffffff,0,c62b2700,...) at dofilewrite+0x84
    kern_writev(c6318780,3,c5ba9c54,c5ba9c74,1,...) at kern_writev+0x6a
    write(c6318780,c5ba9cf8,c,c6318780,c,...) at write+0x50
    syscall(c5ba9d38) at syscall+0x1ad
    Xint0x80_syscall() at Xint0x80_syscall+0x20
    --- syscall (4, FreeBSD ELF32, write), eip = 0x28184573, esp = 0xbfbfad9c, ebp = 0xbfbfede8 ---
    
    Tracing command md0 pid 39 tid 64078 td 0xc6319000
    sched_switch(c6319000,0,104,eb4aa2fe,21,...) at sched_switch+0xb9
    mi_switch(104,0,4c,c6250000,c5b96c84,...) at mi_switch+0xd6
    sleepq_switch(c6319000,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c6250000,4c,c0eb850c,0,0,...) at sleepq_wait+0x45
    _sleep(c6250000,c6250020,24c,c0eb850c,0,...) at _sleep+0x33e
    md_kthread(c6250000,c5b96d38,0,0,0,...) at md_kthread+0x1f6
    fork_exit(c07eac30,c6250000,c5b96d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5b96d70, ebp = 0 ---
    
    Tracing command sh pid 24 tid 64063 td 0xc6207c80
    sched_switch(c6207c80,0,104,5cff3608,22,...) at sched_switch+0xb9
    mi_switch(104,0,0,c5e99880,c5b5ab7c,...) at mi_switch+0xd6
    sleepq_switch(c6207c80,0,c0ee9712,1a0,0,...) at sleepq_switch+0xae
    sleepq_catch_signals(0,c5e99880,c5b5abc8,c0a448a9,c5e997f8,...) at sleepq_catch_signals+0x33b
    sleepq_wait_sig(c5e997f8,5c,c0efafc7,100,0,...) at sleepq_wait_sig+0x18
    _sleep(c5e997f8,c5e99880,15c,c0efafc7,0,...) at _sleep+0x279
    kern_wait(c6207c80,ffffffff,c5b5ac70,2,0,...) at kern_wait+0x747
    wait4(c6207c80,c5b5acf8,c,c6207c80,c,...) at wait4+0x3b
    syscall(c5b5ad38) at syscall+0x1ad
    Xint0x80_syscall() at Xint0x80_syscall+0x20
    --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2816469b, esp = 0xbfbfe52c, ebp = 0xbfbfe548 ---
    
    Tracing command softdepflush pid 23 tid 64062 td 0xc6209000
    sched_switch(c6209000,0,104,a1befeb5,21,...) at sched_switch+0xb9
    mi_switch(104,0,c6209000,c136ac04,c5b56c88,...) at mi_switch+0xd6
    sleepq_switch(c6209000,0,c0ee9712,283,0,...) at sleepq_switch+0xae
    sleepq_timedwait(c136ac60,44,c0f07913,0,0,...) at sleepq_timedwait+0x3e
    _sleep(c136ac60,c136ac04,44,c0f07913,3e8,...) at _sleep+0x323
    softdep_flush(0,c5b56d38,d141a87b,3e8c1ea,44f78a45,...) at softdep_flush+0x3b1
    fork_exit(c0cd9090,0,c5b56d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5b56d70, ebp = 0 ---
    
    Tracing command syncer pid 22 tid 64061 td 0xc6209280
    sched_switch(c6209280,0,104,a1efb2cd,21,...) at sched_switch+0xb9
    mi_switch(104,0,c6209280,c6209280,c5b53c64,...) at mi_switch+0xd6
    sleepq_switch(c6209280,0,c0ee9712,283,0,...) at sleepq_switch+0xae
    sleepq_timedwait(c134fd94,0,c0eedc47,1,0,...) at sleepq_timedwait+0x3e
    _cv_timedwait(c134fd94,c134fd80,3e8,c5e99d48,4e20,...) at _cv_timedwait+0x133
    sched_sync(0,c5b53d38,ffffffff,535f2e5c,53445f42,...) at sched_sync+0x492
    fork_exit(c0ad06e0,0,c5b53d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5b53d70, ebp = 0 ---
    
    Tracing command vnlru pid 21 tid 64060 td 0xc6209500
    sched_switch(c6209500,0,104,a1da8421,21,...) at sched_switch+0xb9
    mi_switch(104,0,c6209500,c134fd54,c5b50c58,...) at mi_switch+0xd6
    sleepq_switch(c6209500,0,c0ee9712,283,0,...) at sleepq_switch+0xae
    sleepq_timedwait(c6204000,50,c0eee420,0,0,...) at sleepq_timedwait+0x3e
    _sleep(c6204000,c134fd54,250,c0eee420,3e8,...) at _sleep+0x323
    vnlru_proc(0,c5b50d38,a305845,2f5c8602,42535f03,...) at vnlru_proc+0x659
    fork_exit(c0aceb80,0,c5b50d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5b50d70, ebp = 0 ---
    
    Tracing command bufdaemon pid 20 tid 64059 td 0xc6209780
    sched_switch(c6209780,0,104,a1d28222,21,...) at sched_switch+0xb9
    mi_switch(104,0,c6209780,c134facc,c5b4dc98,...) at mi_switch+0xd6
    sleepq_switch(c6209780,0,c0ee9712,283,0,...) at sleepq_switch+0xae
    sleepq_timedwait(c134fac8,44,c0eec633,0,0,...) at sleepq_timedwait+0x3e
    _sleep(c134fac8,c134facc,44,c0eec633,3e8,...) at _sleep+0x323
    buf_daemon(0,c5b4dd38,0,1ee10000,0,...) at buf_daemon+0x1bd
    fork_exit(c0ab42f0,0,c5b4dd38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5b4dd70, ebp = 0 ---
    
    Tracing command idlepoll pid 19 tid 64058 td 0xc5fee780
    sched_switch(c5fee780,0,104,a1d26f4c,21,...) at sched_switch+0xb9
    mi_switch(104,0,c5fee780,0,c5b4ac88,...) at mi_switch+0xd6
    sleepq_switch(c5fee780,0,c0ee9712,283,0,...) at sleepq_switch+0xae
    sleepq_timedwait(c133b77c,0,c0ee5ab1,0,0,...) at sleepq_timedwait+0x3e
    _sleep(c133b77c,0,0,c0ee5ab1,bb8,...) at _sleep+0x323
    poll_idle(0,c5b4ad38,1000986,fee00000,10000,...) at poll_idle+0x1aa
    fork_exit(c0a2dd30,0,c5b4ad38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5b4ad70, ebp = 0 ---
    
    Tracing command pagezero pid 18 tid 64057 td 0xc5feea00
    sched_switch(c5feea00,0,104,66f96d29,1f,...) at sched_switch+0xb9
    mi_switch(104,0,c5feea00,c136b4d0,c5b47c98,...) at mi_switch+0xd6
    sleepq_switch(c5feea00,0,c0ee9712,283,0,...) at sleepq_switch+0xae
    sleepq_timedwait(c136b914,0,c0f0b4d5,0,0,...) at sleepq_timedwait+0x3e
    _sleep(c136b914,c136b4d0,0,c0f0b4d5,493e0,...) at _sleep+0x323
    vm_pagezero(0,c5b47d38,9410046e,45bc1d26,cade58b6,...) at vm_pagezero+0x7c
    fork_exit(c0d1d900,0,c5b47d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5b47d70, ebp = 0 ---
    
    Tracing command vmdaemon pid 17 tid 64056 td 0xc5feec80
    sched_switch(c5feec80,0,104,66f8815f,1f,...) at sched_switch+0xb9
    mi_switch(104,0,68,c136b53c,c5b44c5c,...) at mi_switch+0xd6
    sleepq_switch(c5feec80,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c136b53c,68,c0eec633,0,0,...) at sleepq_wait+0x45
    _sleep(c136b53c,c136b540,68,c0eec633,0,...) at _sleep+0x33e
    vm_daemon(0,c5b44d38,5824448b,550d8d4c,48000027,...) at vm_daemon+0x56
    fork_exit(c0d1a1d0,0,c5b44d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5b44d70, ebp = 0 ---
    
    Tracing command pagedaemon pid 16 tid 64055 td 0xc6027000
    sched_switch(c6027000,0,104,66fb45eb,1f,...) at sched_switch+0xb9
    mi_switch(104,0,c6027000,c136b4d0,c5b41bf4,...) at mi_switch+0xd6
    sleepq_switch(c6027000,0,c0ee9712,283,0,...) at sleepq_switch+0xae
    sleepq_timedwait(c136b504,44,c0eec633,0,0,...) at sleepq_timedwait+0x3e
    _sleep(c136b504,c136b4d0,44,c0eec633,1388,...) at _sleep+0x323
    vm_pageout(0,c5b41d38,8d487974,802494,cb8b0000,...) at vm_pageout+0x823
    fork_exit(c0d18bc0,0,c5b41d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5b41d70, ebp = 0 ---
    
    Tracing command xpt_thrd pid 9 tid 64054 td 0xc6027280
    sched_switch(c6027280,0,104,654b4db1,1f,...) at sched_switch+0xb9
    mi_switch(104,0,4c,c1306454,c5b3ec88,...) at mi_switch+0xd6
    sleepq_switch(c6027280,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c1306454,4c,c0e752ac,0,0,...) at sleepq_wait+0x45
    _sleep(c1306454,c1306498,4c,c0e752ac,0,...) at _sleep+0x33e
    xpt_scanner_thread(0,c5b3ed38,48382444,4024548d,248c8d48,...) at xpt_scanner_thread+0x11c
    fork_exit(c0494640,0,c5b3ed38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5b3ed70, ebp = 0 ---
    
    Tracing command pfpurge pid 8 tid 64053 td 0xc6027500
    sched_switch(c6027500,0,104,4187c5c3,22,...) at sched_switch+0xb9
    mi_switch(104,0,c6027500,0,c5b3bc8c,...) at mi_switch+0xd6
    sleepq_switch(c6027500,0,c0ee9712,283,0,...) at sleepq_switch+0xae
    sleepq_timedwait(c04f6d80,5c,c0e83c62,0,0,...) at sleepq_timedwait+0x3e
    _sleep(c04f6d80,0,5c,c0e83c62,3e8,...) at _sleep+0x323
    pf_purge_thread(0,c5b3bd38,0,0,0,...) at pf_purge_thread+0x3d
    fork_exit(c04f6d80,0,c5b3bd38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5b3bd70, ebp = 0 ---
    
    Tracing command sctp_iterator pid 7 tid 64052 td 0xc6027780
    sched_switch(c6027780,0,104,486d9921,1f,...) at sched_switch+0xb9
    mi_switch(104,0,0,c13571d8,c5b38c94,...) at mi_switch+0xd6
    sleepq_switch(c6027780,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c13571d8,0,c0efa516,0,0,...) at sleepq_wait+0x45
    _sleep(c13571d8,c13571a0,0,c0efa516,0,...) at _sleep+0x33e
    sctp_iterator_thread(0,c5b38d38,48502444,20244489,1b8,...) at sctp_iterator_thread+0x58
    fork_exit(c0ba7170,0,c5b38d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5b38d70, ebp = 0 ---
    
    Tracing command usb pid 15 tid 64045 td 0xc5f7cc80
    sched_switch(c5f7cc80,0,104,5ddbcb56,21,...) at sched_switch+0xb9
    mi_switch(104,0,0,c6004d0c,c5a7fc90,...) at mi_switch+0xd6
    sleepq_switch(c5f7cc80,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c6004d0c,0,c0edc643,1,0,...) at sleepq_wait+0x45
    _cv_wait(c6004d0c,c6004dac,c0ed4bc9,68,0,...) at _cv_wait+0x11b
    usb_process(c6004d04,c5a7fd38,0,0,0,...) at usb_process+0x15f
    fork_exit(c09297d0,c6004d04,c5a7fd38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5a7fd70, ebp = 0 ---
    
    Tracing command usb pid 15 tid 64044 td 0xc5fed000
    sched_switch(c5fed000,0,104,5ddbecaa,21,...) at sched_switch+0xb9
    mi_switch(104,0,0,c6004cdc,c5a7cc90,...) at mi_switch+0xd6
    sleepq_switch(c5fed000,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c6004cdc,0,c0edc643,1,0,...) at sleepq_wait+0x45
    _cv_wait(c6004cdc,c6004dac,c0ed4bc9,68,0,...) at _cv_wait+0x11b
    usb_process(c6004cd4,c5a7cd38,0,0,0,...) at usb_process+0x15f
    fork_exit(c09297d0,c6004cd4,c5a7cd38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5a7cd70, ebp = 0 ---
    
    Tracing command usb pid 15 tid 64043 td 0xc5fed280
    sched_switch(c5fed280,0,104,5d4d3a60,21,...) at sched_switch+0xb9
    mi_switch(104,0,0,c6004cac,c5a79c90,...) at mi_switch+0xd6
    sleepq_switch(c5fed280,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c6004cac,0,c0edc643,1,0,...) at sleepq_wait+0x45
    _cv_wait(c6004cac,c6004dac,c0ed4bc9,68,0,...) at _cv_wait+0x11b
    usb_process(c6004ca4,c5a79d38,0,0,0,...) at usb_process+0x15f
    fork_exit(c09297d0,c6004ca4,c5a79d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5a79d70, ebp = 0 ---
    
    Tracing command usb pid 15 tid 64042 td 0xc5fed500
    sched_switch(c5fed500,0,104,485e5bbd,1f,...) at sched_switch+0xb9
    mi_switch(104,0,0,c6004c7c,c5a76c90,...) at mi_switch+0xd6
    sleepq_switch(c5fed500,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c6004c7c,0,c0edc643,1,0,...) at sleepq_wait+0x45
    _cv_wait(c6004c7c,c6004dac,c0ed4bc9,68,0,...) at _cv_wait+0x11b
    usb_process(c6004c74,c5a76d38,0,0,0,...) at usb_process+0x15f
    fork_exit(c09297d0,c6004c74,c5a76d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5a76d70, ebp = 0 ---
    
    Tracing command usb pid 15 tid 64039 td 0xc5fedc80
    sched_switch(c5fedc80,0,104,30163b59,21,...) at sched_switch+0xb9
    mi_switch(104,0,0,c5fd5d0c,c5a41c90,...) at mi_switch+0xd6
    sleepq_switch(c5fedc80,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c5fd5d0c,0,c0edc643,1,0,...) at sleepq_wait+0x45
    _cv_wait(c5fd5d0c,c5fd5dac,c0ed4bc9,68,488d96cb,...) at _cv_wait+0x11b
    usb_process(c5fd5d04,c5a41d38,66cde79d,42820325,f537c069,...) at usb_process+0x15f
    fork_exit(c09297d0,c5fd5d04,c5a41d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5a41d70, ebp = 0 ---
    
    Tracing command usb pid 15 tid 64038 td 0xc5fee000
    sched_switch(c5fee000,0,104,30166147,21,...) at sched_switch+0xb9
    mi_switch(104,0,0,c5fd5cdc,c5a3ec90,...) at mi_switch+0xd6
    sleepq_switch(c5fee000,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c5fd5cdc,0,c0edc643,1,0,...) at sleepq_wait+0x45
    _cv_wait(c5fd5cdc,c5fd5dac,c0ed4bc9,68,5371a133,...) at _cv_wait+0x11b
    usb_process(c5fd5cd4,c5a3ed38,a2b3e50,d9bb3556,99d319a1,...) at usb_process+0x15f
    fork_exit(c09297d0,c5fd5cd4,c5a3ed38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5a3ed70, ebp = 0 ---
    
    Tracing command usb pid 15 tid 64037 td 0xc5fee280
    sched_switch(c5fee280,0,104,2fb48f4c,21,...) at sched_switch+0xb9
    mi_switch(104,0,0,c5fd5cac,c5a3bc90,...) at mi_switch+0xd6
    sleepq_switch(c5fee280,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c5fd5cac,0,c0edc643,1,0,...) at sleepq_wait+0x45
    _cv_wait(c5fd5cac,c5fd5dac,c0ed4bc9,68,5371a133,...) at _cv_wait+0x11b
    usb_process(c5fd5ca4,c5a3bd38,a2b3e50,d9bb3556,99d319a1,...) at usb_process+0x15f
    fork_exit(c09297d0,c5fd5ca4,c5a3bd38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5a3bd70, ebp = 0 ---
    
    Tracing command usb pid 15 tid 64036 td 0xc5fee500
    sched_switch(c5fee500,0,104,484edb69,1f,...) at sched_switch+0xb9
    mi_switch(104,0,0,c5fd5c7c,c5a38c90,...) at mi_switch+0xd6
    sleepq_switch(c5fee500,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c5fd5c7c,0,c0edc643,1,0,...) at sleepq_wait+0x45
    _cv_wait(c5fd5c7c,c5fd5dac,c0ed4bc9,68,2946555c,...) at _cv_wait+0x11b
    usb_process(c5fd5c74,c5a38d38,23606cc6,93e9a1f3,f091cb24,...) at usb_process+0x15f
    fork_exit(c09297d0,c5fd5c74,c5a38d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5a38d70, ebp = 0 ---
    
    Tracing command yarrow pid 14 tid 64024 td 0xc5dd4a00
    sched_switch(c5dd4a00,0,104,5457fd87,22,...) at sched_switch+0xb9
    mi_switch(104,0,c5dd4a00,0,c59dfc64,...) at mi_switch+0xd6
    sleepq_switch(c5dd4a00,0,c0ee9712,283,0,...) at sleepq_switch+0xae
    sleepq_timedwait(c133c084,0,c0edc643,2,0,...) at sleepq_timedwait+0x3e
    _sleep(c133c084,0,0,c0edc643,64,...) at _sleep+0x323
    pause(c0edc643,64,83e17ac,e5083a36,1c80440,...) at pause+0x30
    random_kthread(0,c59dfd38,b0794af4,f72f687f,638aed13,...) at random_kthread+0x1b8
    fork_exit(c08867f0,0,c59dfd38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59dfd70, ebp = 0 ---
    
    Tracing command crypto returns pid 6 tid 64023 td 0xc5dd4c80
    sched_switch(c5dd4c80,0,104,484ebec1,1f,...) at sched_switch+0xb9
    mi_switch(104,0,5c,c136a28c,c59d8c80,...) at mi_switch+0xd6
    sleepq_switch(c5dd4c80,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c136a28c,5c,c0f052d4,0,0,...) at sleepq_wait+0x45
    _sleep(c136a28c,c136a29c,5c,c0f052d4,0,...) at _sleep+0x33e
    crypto_ret_proc(0,c59d8d38,f5d4c1af,2644caa9,662ce65e,...) at crypto_ret_proc+0x19c
    fork_exit(c0c97320,0,c59d8d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59d8d70, ebp = 0 ---
    
    Tracing command crypto pid 5 tid 64022 td 0xc5e95000
    sched_switch(c5e95000,0,104,484eb1ed,1f,...) at sched_switch+0xb9
    mi_switch(104,0,5c,c136a268,c59d5c8c,...) at mi_switch+0xd6
    sleepq_switch(c5e95000,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c136a268,5c,c0f052c8,0,0,...) at sleepq_wait+0x45
    _sleep(c136a268,c136a278,5c,c0f052c8,0,...) at _sleep+0x33e
    crypto_proc(0,c59d5d38,f5d4c1af,2644caa9,662ce65e,...) at crypto_proc+0x160
    fork_exit(c0c96be0,0,c59d5d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59d5d70, ebp = 0 ---
    
    Tracing command g_down pid 4 tid 64021 td 0xc5e95280
    sched_switch(c5e95280,0,104,5cfb5bd8,22,...) at sched_switch+0xb9
    mi_switch(104,0,4c,c1339924,c59d2c50,...) at mi_switch+0xd6
    sleepq_switch(c5e95280,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c1339924,4c,c0edc643,0,0,...) at sleepq_wait+0x45
    _sleep(c1339924,c1339888,24c,c0edc643,0,...) at _sleep+0x33e
    g_io_schedule_down(c5e95280,4c,c0ede69d,72,0,...) at g_io_schedule_down+0x273
    g_down_procbody(0,c59d2d38,0,0,0,...) at g_down_procbody+0x5b
    fork_exit(c09c58a0,0,c59d2d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59d2d70, ebp = 0 ---
    
    Tracing command g_up pid 3 tid 64020 td 0xc5e95500
    sched_switch(c5e95500,0,104,5d1e00e8,22,...) at sched_switch+0xb9
    mi_switch(104,0,4c,c1339920,c59cfc78,...) at mi_switch+0xd6
    sleepq_switch(c5e95500,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c1339920,4c,c0edc643,0,0,...) at sleepq_wait+0x45
    _sleep(c1339920,c13398a8,24c,c0edc643,0,...) at _sleep+0x33e
    g_io_schedule_up(c5e95500,4c,c0ede69d,5b,0,...) at g_io_schedule_up+0xbd
    g_up_procbody(0,c59cfd38,0,0,0,...) at g_up_procbody+0x5b
    fork_exit(c09c5910,0,c59cfd38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59cfd70, ebp = 0 ---
    
    Tracing command g_event pid 2 tid 64019 td 0xc5e95780
    sched_switch(c5e95780,0,104,4d7d20d5,22,...) at sched_switch+0xb9
    mi_switch(104,0,c5e95780,0,c59ccc94,...) at mi_switch+0xd6
    sleepq_switch(c5e95780,0,c0ee9712,283,0,...) at sleepq_switch+0xae
    sleepq_timedwait(c1339918,4c,c0edc643,0,0,...) at sleepq_timedwait+0x3e
    _sleep(c1339918,0,4c,c0edc643,64,...) at _sleep+0x323
    g_event_procbody(0,c59ccd38,0,0,0,...) at g_event_procbody+0x97
    fork_exit(c09c5980,0,c59ccd38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59ccd70, ebp = 0 ---
    
    Tracing command ng_queue pid 13 tid 64017 td 0xc5e95c80
    sched_switch(c5e95c80,0,104,484df2cd,1f,...) at sched_switch+0xb9
    mi_switch(104,0,10,c12dcae0,c59c6c8c,...) at mi_switch+0xd6
    sleepq_switch(c5e95c80,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c12dcae0,10,c0eecb1b,0,0,...) at sleepq_wait+0x45
    _sleep(c12dcae0,c1352924,10,c0eecb1b,0,...) at _sleep+0x33e
    ngthread(0,c59c6d38,0,0,0,...) at ngthread+0x5c
    fork_exit(c0b5b350,0,c59c6d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59c6d70, ebp = 0 ---
    
    Tracing command ng_queue pid 13 tid 64016 td 0xc5d91280
    sched_switch(c5d91280,0,104,484de859,1f,...) at sched_switch+0xb9
    mi_switch(104,0,10,c12dcae0,c59c3c8c,...) at mi_switch+0xd6
    sleepq_switch(c5d91280,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c12dcae0,10,c0eecb1b,0,0,...) at sleepq_wait+0x45
    _sleep(c12dcae0,c1352924,10,c0eecb1b,0,...) at _sleep+0x33e
    ngthread(0,c59c3d38,0,0,0,...) at ngthread+0x5c
    fork_exit(c0b5b350,0,c59c3d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59c3d70, ebp = 0 ---
    
    Tracing command ng_queue pid 13 tid 64015 td 0xc5d91500
    sched_switch(c5d91500,0,104,484dde65,1f,...) at sched_switch+0xb9
    mi_switch(104,0,10,c12dcae0,c59c0c8c,...) at mi_switch+0xd6
    sleepq_switch(c5d91500,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c12dcae0,10,c0eecb1b,0,0,...) at sleepq_wait+0x45
    _sleep(c12dcae0,c1352924,10,c0eecb1b,0,...) at _sleep+0x33e
    ngthread(0,c59c0d38,0,0,0,...) at ngthread+0x5c
    fork_exit(c0b5b350,0,c59c0d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59c0d70, ebp = 0 ---
    
    Tracing command ng_queue pid 13 tid 64014 td 0xc5d91780
    sched_switch(c5d91780,0,104,484dd2a1,1f,...) at sched_switch+0xb9
    mi_switch(104,0,10,c12dcae0,c59bdc8c,...) at mi_switch+0xd6
    sleepq_switch(c5d91780,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c12dcae0,10,c0eecb1b,0,0,...) at sleepq_wait+0x45
    _sleep(c12dcae0,c1352924,10,c0eecb1b,0,...) at _sleep+0x33e
    ngthread(0,c59bdd38,0,0,0,...) at ngthread+0x5c
    fork_exit(c0b5b350,0,c59bdd38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59bdd70, ebp = 0 ---
    
    Tracing command intr pid 12 tid 64051 td 0xc6027a00
    fork_trampoline() at fork_trampoline
    
    Tracing command intr pid 12 tid 64050 td 0xc6027c80
    fork_trampoline() at fork_trampoline
    
    Tracing command intr pid 12 tid 64049 td 0xc6028000
    fork_trampoline() at fork_trampoline
    
    Tracing command intr pid 12 tid 64048 td 0xc6028280
    fork_trampoline() at fork_trampoline
    
    Tracing command intr pid 12 tid 64047 td 0xc6028500
    fork_trampoline() at fork_trampoline
    
    Tracing command intr pid 12 tid 64046 td 0xc5f7ca00
    sched_switch(c5f7ca00,0,109,5d1af8f0,22,...) at sched_switch+0xb9
    mi_switch(109,0,c0ee305c,52d,c5dd34f0,...) at mi_switch+0xd6
    ithread_loop(c601c0c0,c5a82d38,0,0,0,...) at ithread_loop+0x1a1
    fork_exit(c0a11aa0,c601c0c0,c5a82d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5a82d70, ebp = 0 ---
    
    Tracing command intr pid 12 tid 64041 td 0xc5fed780
    fork_trampoline() at fork_trampoline
    
    Tracing command intr pid 12 tid 64040 td 0xc5feda00
    fork_trampoline() at fork_trampoline
    
    Tracing command intr pid 12 tid 64035 td 0xc5f7b000
    sched_switch(c5f7b000,0,109,5ddbadc3,21,...) at sched_switch+0xb9
    mi_switch(109,0,c0ee305c,52d,c5dd3370,...) at mi_switch+0xd6
    ithread_loop(c5fcf290,c5a35d38,23606cc6,93e9a1f3,f091cb24,...) at ithread_loop+0x1a1
    fork_exit(c0a11aa0,c5fcf290,c5a35d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5a35d70, ebp = 0 ---
    
    Tracing command intr pid 12 tid 64034 td 0xc5f7b280
    fork_trampoline() at fork_trampoline
    
    Tracing command intr pid 12 tid 64030 td 0xc5f7bc80
    sched_switch(c5f7bc80,0,109,654a56a1,1f,...) at sched_switch+0xb9
    mi_switch(109,0,c0ee305c,52d,c5f7a8f0,...) at mi_switch+0xd6
    ithread_loop(c5d8c430,c59f1d38,fabadf73,259fcd1e,f4b6c36a,...) at ithread_loop+0x1a1
    fork_exit(c0a11aa0,c5d8c430,c59f1d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59f1d70, ebp = 0 ---
    
    Tracing command intr pid 12 tid 64029 td 0xc5f7c000
    fork_trampoline() at fork_trampoline
    
    Tracing command intr pid 12 tid 64027 td 0xc5f7c500
    fork_trampoline() at fork_trampoline
    
    Tracing command intr pid 12 tid 64025 td 0xc5dd4780
    fork_trampoline() at fork_trampoline
    
    Tracing command intr pid 12 tid 64013 td 0xc5d91a00
    sched_switch(c5d91a00,0,109,59057ce2,22,...) at sched_switch+0xb9
    mi_switch(109,0,c0ee305c,52d,c5dd3070,...) at mi_switch+0xd6
    ithread_loop(c5d8c0d0,c59bad38,0,0,0,...) at ithread_loop+0x1a1
    fork_exit(c0a11aa0,c5d8c0d0,c59bad38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59bad70, ebp = 0 ---
    
    Tracing command intr pid 12 tid 64012 td 0xc5d91c80
    sched_switch(c5d91c80,0,109,5456efd3,22,...) at sched_switch+0xb9
    mi_switch(109,0,c0ee305c,52d,c5dd30f0,...) at mi_switch+0xd6
    ithread_loop(c5d8c0e0,c59b7d38,0,0,0,...) at ithread_loop+0x1a1
    fork_exit(c0a11aa0,c5d8c0e0,c59b7d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59b7d70, ebp = 0 ---
    
    Tracing command intr pid 12 tid 64011 td 0xc5dd4000
    sched_switch(c5dd4000,0,109,4be245cc,22,...) at sched_switch+0xb9
    mi_switch(109,0,c0ee305c,52d,c5dd3170,...) at mi_switch+0xd6
    ithread_loop(c5d8c0f0,c59b4d38,0,0,0,...) at ithread_loop+0x1a1
    fork_exit(c0a11aa0,c5d8c0f0,c59b4d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59b4d70, ebp = 0 ---
    
    Tracing command intr pid 12 tid 64010 td 0xc5dd4280
    sched_switch(c5dd4280,0,109,5d204c88,22,...) at sched_switch+0xb9
    mi_switch(109,0,c0ee305c,52d,c5dd31f0,...) at mi_switch+0xd6
    ithread_loop(c5d8c100,c59b1d38,0,0,0,...) at ithread_loop+0x1a1
    fork_exit(c0a11aa0,c5d8c100,c59b1d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59b1d70, ebp = 0 ---
    
    Tracing command intr pid 12 tid 64009 td 0xc5dd4500
    fork_trampoline() at fork_trampoline
    
    Tracing command intr pid 12 tid 64008 td 0xc5d8f000
    fork_trampoline() at fork_trampoline
    
    Tracing command idle pid 11 tid 64007 td 0xc5d8f280
    cpustop_handler(1,c59a7c34,c0def345,369e99,0,...) at cpustop_handler+0x32
    ipi_nmi_handler(369e99,0,0,c59a7bfc,c5d8daa0,...) at ipi_nmi_handler+0x2c
    trap(c59a7c40) at trap+0x255
    calltrap() at calltrap+0x6
    --- trap 0x13, eip = 0xc0dc4bf5, esp = 0xc59a7c80, ebp = 0xc59a7c80 ---
    acpi_cpu_c1(1,c59a7c94,0,0,ffffffff,1) at acpi_cpu_c1+0x5
    acpi_cpu_idle(1,c59a7ccc,c0ee77d5,9f9,c5d8f280,...) at acpi_cpu_idle+0x1b9
    sched_idletd(0,c59a7d38,0,0,0,...) at sched_idletd+0x117
    fork_exit(c0a5e060,0,c59a7d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59a7d70, ebp = 0 ---
    
    Tracing command idle pid 11 tid 64006 td 0xc5d8f500
    cpustop_handler(2,c59a4c34,c0def345,c5e95500,c1340680,...) at cpustop_handler+0x32
    ipi_nmi_handler(c5e95500,c1340680,46,c133c29c,c5d8daa0,...) at ipi_nmi_handler+0x2c
    trap(c59a4c40) at trap+0x255
    calltrap() at calltrap+0x6
    --- trap 0x13, eip = 0xc0dc4bf5, esp = 0xc59a4c80, ebp = 0xc59a4c80 ---
    acpi_cpu_c1(1,c59a4c94,1,0,ffffffff,2) at acpi_cpu_c1+0x5
    acpi_cpu_idle(1,c59a4ccc,c0ee77d5,9f9,c5d8f500,...) at acpi_cpu_idle+0x1b9
    sched_idletd(0,c59a4d38,0,0,0,...) at sched_idletd+0x117
    fork_exit(c0a5e060,0,c59a4d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59a4d70, ebp = 0 ---
    
    Tracing command idle pid 11 tid 64005 td 0xc5d8f780
    sched_switch(c5d8f780,0,60c,5cfeb0b8,22,...) at sched_switch+0xb9
    mi_switch(60c,0,c0ee77d5,805,2,...) at mi_switch+0xd6
    sched_preempt(c5d8f780,2,1f4,c59a1c80,c0dd10ae,...) at sched_preempt+0xaf
    ipi_bitmap_handler(c0f20008,28,c5ea0028,c5ea1800,c5ea1818,...) at ipi_bitmap_handler+0x6e
    Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x2e
    --- interrupt, eip = 0xc0dc4bf5, esp = 0xc59a1c80, ebp = 0xc59a1c80 ---
    acpi_cpu_c1(1,c59a1c94,2,0,ffffffff,4) at acpi_cpu_c1+0x5
    acpi_cpu_idle(1,c59a1ccc,c0ee77d5,9f9,c5d8f780,...) at acpi_cpu_idle+0x1b9
    sched_idletd(0,c59a1d38,0,0,0,...) at sched_idletd+0x117
    fork_exit(c0a5e060,0,c59a1d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59a1d70, ebp = 0 ---
    
    Tracing command idle pid 11 tid 64004 td 0xc5d8fa00
    cpustop_handler(8,c599ec34,c0def345,c5dd4280,c1341380,...) at cpustop_handler+0x32
    ipi_nmi_handler(c5dd4280,c1341380,46,c133c314,c5d8daa0,...) at ipi_nmi_handler+0x2c
    trap(c599ec40) at trap+0x255
    calltrap() at calltrap+0x6
    --- trap 0x13, eip = 0xc0dc4bf5, esp = 0xc599ec80, ebp = 0xc599ec80 ---
    acpi_cpu_c1(1,c599ec94,3,0,ffffffff,8) at acpi_cpu_c1+0x5
    acpi_cpu_idle(1,c599eccc,c0ee77d5,9f9,c5d8fa00,...) at acpi_cpu_idle+0x1b9
    sched_idletd(0,c599ed38,0,0,0,...) at sched_idletd+0x117
    fork_exit(c0a5e060,0,c599ed38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc599ed70, ebp = 0 ---
    
    Tracing command init pid 1 tid 64003 td 0xc5d8fc80
    sched_switch(c5d8fc80,0,104,6f9ee5e0,21,...) at sched_switch+0xb9
    mi_switch(104,0,0,c5d8ddd0,c599ab7c,...) at mi_switch+0xd6
    sleepq_switch(c5d8fc80,0,c0ee9712,1a0,0,...) at sleepq_switch+0xae
    sleepq_catch_signals(0,c5d8ddd0,c599abc8,c0a448a9,c5d8dd48,...) at sleepq_catch_signals+0x33b
    sleepq_wait_sig(c5d8dd48,5c,c0efafc7,100,0,...) at sleepq_wait_sig+0x18
    _sleep(c5d8dd48,c5d8ddd0,15c,c0efafc7,0,...) at _sleep+0x279
    kern_wait(c5d8fc80,ffffffff,c599ac70,2,0,...) at kern_wait+0x747
    wait4(c5d8fc80,c599acf8,c,c5d8fc80,c,...) at wait4+0x3b
    syscall(c599ad38) at syscall+0x1ad
    Xint0x80_syscall() at Xint0x80_syscall+0x20
    --- syscall (7, FreeBSD ELF32, wait4), eip = 0x8054b7f, esp = 0xbfbfe8cc, ebp = 0xbfbfe8e8 ---
    
    Tracing command audit pid 10 tid 64002 td 0xc5d91000
    sched_switch(c5d91000,0,104,484dc531,1f,...) at sched_switch+0xb9
    mi_switch(104,0,0,c136a600,c5997c70,...) at mi_switch+0xd6
    sleepq_switch(c5d91000,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c136a600,0,c0f05e3f,1,0,...) at sleepq_wait+0x45
    _cv_wait(c136a600,c136a5e4,0,0,0,...) at _cv_wait+0x11b
    audit_worker(0,c5997d38,0,0,0,...) at audit_worker+0x362
    fork_exit(c0cbeba0,0,c5997d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc5997d70, ebp = 0 ---
    
    Tracing command kernel pid 0 tid 64033 td 0xc5f7b500
    sched_switch(c5f7b500,0,104,48a92955,1f,...) at sched_switch+0xb9
    mi_switch(104,0,0,c5f5f180,c59faca0,...) at mi_switch+0xd6
    sleepq_switch(c5f7b500,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c5f5f180,0,c0edc643,0,0,...) at sleepq_wait+0x45
    msleep_spin(c5f5f180,c5f5f198,c0edc643,0,20,...) at msleep_spin+0x1ad
    taskqueue_thread_loop(c130aa20,c59fad38,965c06be,bea6fb6d,98521aa2,...) at taskqueue_thread_loop+0x6e
    fork_exit(c0a77de0,c130aa20,c59fad38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59fad70, ebp = 0 ---
    
    Tracing command kernel pid 0 tid 64032 td 0xc5f7b780
    sched_switch(c5f7b780,0,104,48a33539,1f,...) at sched_switch+0xb9
    mi_switch(104,0,0,c5f5f180,c59f7ca0,...) at mi_switch+0xd6
    sleepq_switch(c5f7b780,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c5f5f180,0,c0edc643,0,0,...) at sleepq_wait+0x45
    msleep_spin(c5f5f180,c5f5f198,c0edc643,0,c59f7cf0,...) at msleep_spin+0x1ad
    taskqueue_thread_loop(c130aa20,c59f7d38,965c06be,bea6fb6d,98521aa2,...) at taskqueue_thread_loop+0x6e
    fork_exit(c0a77de0,c130aa20,c59f7d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59f7d70, ebp = 0 ---
    
    Tracing command kernel pid 0 tid 64031 td 0xc5f7ba00
    sched_switch(c5f7ba00,0,104,48a32b01,1f,...) at sched_switch+0xb9
    mi_switch(104,0,0,c5f5f180,c59f4ca0,...) at mi_switch+0xd6
    sleepq_switch(c5f7ba00,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c5f5f180,0,c0edc643,0,0,...) at sleepq_wait+0x45
    msleep_spin(c5f5f180,c5f5f198,c0edc643,0,c59f4cf0,...) at msleep_spin+0x1ad
    taskqueue_thread_loop(c130aa20,c59f4d38,fabadf73,259fcd1e,f4b6c36a,...) at taskqueue_thread_loop+0x6e
    fork_exit(c0a77de0,c130aa20,c59f4d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59f4d70, ebp = 0 ---
    
    Tracing command kernel pid 0 tid 64028 td 0xc5f7c280
    sched_switch(c5f7c280,0,104,6560e7f9,1f,...) at sched_switch+0xb9
    mi_switch(104,0,0,c5f5f2c0,c59ebc90,...) at mi_switch+0xd6
    sleepq_switch(c5f7c280,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c5f5f2c0,0,c0edc643,0,0,...) at sleepq_wait+0x45
    _sleep(c5f5f2c0,c5f5f2d8,0,c0edc643,0,...) at _sleep+0x33e
    taskqueue_thread_loop(c134ea88,c59ebd38,d3f18de5,6fcd59bb,9df411ef,...) at taskqueue_thread_loop+0xc7
    fork_exit(c0a77de0,c134ea88,c59ebd38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59ebd70, ebp = 0 ---
    
    Tracing command kernel pid 0 tid 64026 td 0xc5f7c780
    sched_switch(c5f7c780,0,104,486dc071,1f,...) at sched_switch+0xb9
    mi_switch(104,0,0,c5f5f680,c59e5c90,...) at mi_switch+0xd6
    sleepq_switch(c5f7c780,0,c0ee9712,260,0,...) at sleepq_switch+0xae
    sleepq_wait(c5f5f680,0,c0edc643,0,0,...) at sleepq_wait+0x45
    _sleep(c5f5f680,c5f5f698,0,c0edc643,0,...) at _sleep+0x33e
    taskqueue_thread_loop(c133a2e4,c59e5d38,99321862,bd7dfcda,ac83f96,...) at taskqueue_thread_loop+0xc7
    fork_exit(c0a77de0,c133a2e4,c59e5d38) at fork_exit+0x88
    fork_trampoline() at fork_trampoline+0x8
    --- trap 0, eip = 0, esp = 0xc59e5d70, ebp = 0 ---
    
    Tracing command kernel pid 0 tid 64018 td 0xc5e95a00
    sched_switch(c5e95a00,0,104,608bbc92,21,...) at <[/thread]
    


  • Бэкап конфига есть? Сносим полностью , ставим по-новой и заливаем конфиг. На разбор полетов вы времени больше потратите , чем на восстановление из бэкапа.
    На себе проверено не один десяток раз.



  • @werter:

    Бэкап конфига есть? Сносим полностью , ставим по-новой и заливаем конфиг. На разбор полетов вы времени больше потратите , чем на восстановление из бэкапа.
    На себе проверено не один десяток раз.

    Да бэкапы то есть, но все же как то странно у меня эта машина с января стояла.. из серии 1 раз настроил и забыл… так по мелочам гайки подкручивались на проксе, и подкручивались правила фаерволе



  • @1qoot1:

    Да бэкапы то есть, но все же как то странно у меня эта машина с января стояла.. из серии 1 раз настроил и забыл… так по мелочам гайки подкручивались на проксе, и подкручивались правила фаерволе

    И у меня такаж хрень была. Переустановил и всё ок. Но осадочек остался. :-(



  • @dr.gopher:

    @1qoot1:

    Да бэкапы то есть, но все же как то странно у меня эта машина с января стояла.. из серии 1 раз настроил и забыл… так по мелочам гайки подкручивались на проксе, и подкручивались правила фаерволе

    И у меня такаж хрень была. Переустановил и всё ок. Но осадочек остался. :-(

    Согласен что осадок есть, но у меня не одна така машинка стоит…. около 16 =(

    Вопрос чего ждать от других?

    Есть и критически важные точки за которыми стоит и телефон и всякие пользовательские приблуды типа 1с и тд...

    Грубо говоря у меня гибридная схема сети звезда-звезда.... т.е. первая звезда на pf вертится вторая на centos+vz.

    Но как бы меня не поймут если вдруг первая звезда неожиданно отвалится c kernel panic...



  • В 99 % случаев - проблема в железе. ИБП есть? Свет случаем резко и надолго не выключали?



  • @werter:

    В 99 % случаев - проблема в железе. ИБП есть? Свет случаем резко и надолго не выключали?

    Угу, куда ж без него родного…
    А на счет света вроде бы нет...


Locked