Kernel panic

[1qoot1]

Всем доброго времени суток.

В первый рабочий день после отпуска в одном из филиалов поймал kernel panic.

Помогите разобраться куда копать… Как лечить?

msgbuf.txt

Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-RELEASE-p6 #0: Mon Dec 12 17:53:00 EST 2011
    root@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8 i386
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz (3193.02-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x206a7  Family = 6  Model = 2a  Stepping = 7
  Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x159ae3bf<sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,<b17>,SSE4.1,SSE4.2,POPCNT,<b24>,XSAVE,<b28>>
  AMD Features=0x28100000 <nx,rdtscp,lm>AMD Features2=0x1 <lahf>TSC: P-state invariant
real memory  = 2147483648 (2048 MB)
avail memory = 2064588800 (1968 MB)
ACPI APIC Table: <alaska a="" m="" i="">FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 SMT threads
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
ioapic0 <version 2.0="">irqs 0-23 on motherboard
netisr_init: forcing maxthreads to 1 and bindthreads to 0 for device polling
wpi: You need to read the LICENSE file in /usr/share/doc/legal/intel_wpi/.
wpi: If you agree with the license, set legal.intel_wpi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (wpi_fw, 0xc0988300, 0) error 1
ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_bss_fw, 0xc0789340, 0) error 1
ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_ibss_fw, 0xc07893e0, 0) error 1
ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_monitor_fw, 0xc0789480, 0) error 1
wlan: mac acl policy registered
kbd1 at kbdmux0
cryptosoft0: <software crypto="">on motherboard
padlock0: No ACE support.
acpi0: <alaska a="" m="" i="">on motherboard
acpi0: [ITHREAD]
ACPI Error (psargs-0464): [RAMB] Namespace lookup failure, AE_NOT_FOUND
ACPI Exception: AE_NOT_FOUND, Could not execute arguments for [RAMW] (Region) (20100331/nsinit-442)
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
cpu0: <acpi cpu="">on acpi0
cpu1: <acpi cpu="">on acpi0
cpu2: <acpi cpu="">on acpi0
cpu3: <acpi cpu="">on acpi0
acpi_hpet0: <high precision="" event="" timer="">iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
pci0: <acpi pci="" bus="">on pcib0
pcib1: <acpi pci-pci="" bridge="">irq 16 at device 1.0 on pci0
pci1: <acpi pci="" bus="">on pcib1
vgapci0: <vga-compatible display="">port 0xe000-0xe0ff mem 0xc0000000-0xcfffffff,0xfe620000-0xfe63ffff irq 16 at device 0.0 on pci1
pci1: <multimedia, hda="">at device 0.1 (no driver attached)
pci0: <simple comms="">at device 22.0 (no driver attached)
ehci0: <ehci (generic)="" usb="" 2.0="" controller="">mem 0xfe702000-0xfe7023ff irq 23 at device 26.0 on pci0
ehci0: [ITHREAD]
usbus0: EHCI version 1.0
usbus0: <ehci (generic)="" usb="" 2.0="" controller="">on ehci0
pcib2: <acpi pci-pci="" bridge="">irq 17 at device 28.0 on pci0
pci2: <acpi pci="" bus="">on pcib2
pcib3: <acpi pci-pci="" bridge="">irq 16 at device 28.1 on pci0
pci3: <acpi pci="" bus="">on pcib3
pcib4: <acpi pci-pci="" bridge="">irq 18 at device 28.2 on pci0
pci4: <acpi pci="" bus="">on pcib4
re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" pcie="" gigabit="" ethernet="">port 0xd000-0xd0ff mem 0xd0004000-0xd0004fff,0xd0000000-0xd0003fff irq 18 at device 0.0 on pci4
re0: Using 1 MSI messages
re0: Chip rev. 0x2c800000
re0: MAC rev. 0x00000000
re0: Unknown H/W revision: 0x2c800000
device_attach: re0 attach returned 6
pcib5: <acpi pci-pci="" bridge="">irq 19 at device 28.3 on pci0
pci5: <acpi pci="" bus="">on pcib5
pcib6: <acpi pci-pci="" bridge="">irq 17 at device 28.4 on pci0
pci6: <acpi pci="" bus="">on pcib6
pcib7: <acpi pci-pci="" bridge="">irq 17 at device 0.0 on pci6
pci7: <acpi pci="" bus="">on pcib7
rl0: <realtek 10="" 8139="" 100basetx="">port 0xc100-0xc1ff mem 0xfe521000-0xfe5210ff irq 16 at device 0.0 on pci7
miibus0: <mii bus="">on rl0
rlphy0: <realtek internal="" media="" interface="">PHY 0 on miibus0
rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
rl0: [ITHREAD]
rl1: <realtek 10="" 8139="" 100basetx="">port 0xc000-0xc0ff mem 0xfe520000-0xfe5200ff irq 17 at device 1.0 on pci7
miibus1: <mii bus="">on rl1
rlphy1: <realtek internal="" media="" interface="">PHY 0 on miibus1
rlphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
rl1: [ITHREAD]
pcib8: <acpi pci-pci="" bridge="">irq 16 at device 28.5 on pci0
pci8: <acpi pci="" bus="">on pcib8
ehci1: <ehci (generic)="" usb="" 2.0="" controller="">mem 0xfe701000-0xfe7013ff irq 23 at device 29.0 on pci0
ehci1: [ITHREAD]
usbus1: EHCI version 1.0
usbus1: <ehci (generic)="" usb="" 2.0="" controller="">on ehci1
isab0: <pci-isa bridge="">at device 31.0 on pci0
isa0: <isa bus="">on isab0
atapci0: <intel ata="" controller="">port 0xf0d0-0xf0d7,0xf0c0-0xf0c3,0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf090-0xf09f,0xf080-0xf08f irq 20 at device 31.2 on pci0
atapci0: [ITHREAD]
ata2: <ata 0="" channel="">on atapci0
ata2: [ITHREAD]
ata3: <ata 1="" channel="">on atapci0
ata3: [ITHREAD]
pci0: <serial bus,="" smbus="">at device 31.3 (no driver attached)
atapci1: <intel ata="" controller="">port 0xf070-0xf077,0xf060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf030-0xf03f,0xf020-0xf02f irq 20 at device 31.5 on pci0
atapci1: [ITHREAD]
ata4: <ata 0="" channel="">on atapci1
ata4: [ITHREAD]
ata5: <ata 1="" channel="">on atapci1
ata5: [ITHREAD]
acpi_button0: <power button="">on acpi0
ppc1: <parallel port="">port 0x378-0x37f irq 5 on acpi0
ppc1: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc1: [ITHREAD]
ppbus0: <parallel port="" bus="">on ppc1
plip0: <plip network="" interface="">on ppbus0
plip0: [ITHREAD]
lpt0: <printer>on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <parallel i="" o="">on ppbus0
atrtc0: <at realtime="" clock="">port 0x70-0x71 irq 8 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
pmtimer0 on isa0
orm0: <isa option="" rom="">at iomem 0xc0000-0xcffff pnpid ORM0000 on isa0
sc0: <system console="">at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
ata0: [ITHREAD]
ata1 at port 0x170-0x177,0x376 irq 15 on isa0
ata1: [ITHREAD]
atkbdc0: <keyboard controller="" (i8042)="">at port 0x60,0x64 on isa0
atkbd0: <at keyboard="">irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ppc0: cannot reserve I/O port range
est0: <enhanced speedstep="" frequency="" control="">on cpu0
p4tcc0: <cpu frequency="" thermal="" control="">on cpu0
est1: <enhanced speedstep="" frequency="" control="">on cpu1
p4tcc1: <cpu frequency="" thermal="" control="">on cpu1
est2: <enhanced speedstep="" frequency="" control="">on cpu2
p4tcc2: <cpu frequency="" thermal="" control="">on cpu2
est3: <enhanced speedstep="" frequency="" control="">on cpu3
p4tcc3: <cpu frequency="" thermal="" control="">on cpu3
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
ad4: 238475MB <st250dm000 1bd141="" kc45="">at ata2-master UDMA33 
ugen0.1: <intel>at usbus0
uhub0: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus0
ugen1.1: <intel>at usbus1
uhub1: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus1
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
Root mount waiting for: usbus1 usbus0
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
Root mount waiting for: usbus1 usbus0
ugen0.2: <vendor 0x8087="">at usbus0
uhub2: <vendor 2="" 9="" 0x8087="" product="" 0x0024,="" class="" 0,="" rev="" 2.00="" 0.00,="" addr="">on usbus0
ugen1.2: <vendor 0x8087="">at usbus1
uhub3: <vendor 2="" 9="" 0x8087="" product="" 0x0024,="" class="" 0,="" rev="" 2.00="" 0.00,="" addr="">on usbus1
Root mount waiting for: usbus1 usbus0
uhub2: 4 ports with 4 removable, self powered
uhub3: 6 ports with 6 removable, self powered
Trying to mount root from ufs:/dev/ad4s1a
<118>Configuring crash dumps...
<118>Using /dev/ad4s1b for dump device.
<118>Mounting filesystems...
<118>
<118>     ___
<118> ___/ f \
<118>/ p \___/ Sense
<118>\___/   \
<118>    \___/
<118>
<118>
<118>Welcome to pfSense 2.0.1-RELEASE  ...
<118>
<118>No core dumps found.
<118>Creating symlinks...
<118>ln: 
<118>/tmp/tmp
<118>: 
<118>File exists
<118>rm: 
<118>/tmp/tmp: Bad file descriptor
<118>
<118>.
free inode //23881745 had 1056 blocks
panic: ffs_write: type 0xc634653c 0 (0,16384)
cpuid = 2
KDB: enter: panic

0xc634653c: tag ufs, type VNON
    usecount 1, writecount 1, refcount 1 mountedhere 0
    flags ()
    lock type ufs: EXCL by thread 0xc6318780 (pid 87)
	ino 23881774, on dev ad4s1a</vendor></vendor></vendor></vendor></intel></intel></intel></intel></st250dm000></cpu></enhanced></cpu></enhanced></cpu></enhanced></cpu></enhanced></at></keyboard></generic></system></isa></at></parallel></printer></plip></parallel></parallel></power></ata></ata></intel></serial></ata></ata></intel></isa></pci-isa></ehci></ehci></acpi></acpi></realtek></mii></realtek></realtek></mii></realtek></acpi></acpi></acpi></acpi></acpi></acpi></realtek></acpi></acpi></acpi></acpi></acpi></acpi></ehci></ehci></simple></multimedia,></vga-compatible></acpi></acpi></acpi></acpi></high></acpi></acpi></acpi></acpi></alaska></software></version></alaska></lahf></nx,rdtscp,lm></b28></b24></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,<b17></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe> 

Panic.txt

ffs_write: type 0xc634653c 0 (0,16384)

Version.txt

FreeBSD 8.1-RELEASE-p6 #0: Mon Dec 12 17:53:00 EST 2011
    root@FreeBSD_8.0_pfSense_2.0-snaps.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8

config.txt

options	CONFIG_AUTOGENERATED
ident	pfSense8_SMP
machine	i386
cpu	I686_CPU
cpu	I586_CPU
cpu	I486_CPU
options	ALTQ_NOPCC
options	PPS_SYNC
options	ALTQ_FAIRQ
options	MROUTING
options	DEVICE_POLLING
options	NULLFS
options	MSGTQL=2048
options	MSGSSZ=32
options	MSGSEG=512
options	MSGMNI=40
options	MSGMNB=8192
options	ALTQ_PRIQ
options	ALTQ_HFSC
options	ALTQ_RIO
options	ALTQ_RED
options	ALTQ_CBQ
options	ALTQ
options	TCP_SIGNATURE
options	IPSEC_NAT_T
options	IPSEC
options	NETGRAPH_PRED1
options	NETGRAPH_DEFLATE
options	NETGRAPH_CAR
options	NETGRAPH_PIPE
options	NETGRAPH_TCPMSS
options	NETGRAPH_VJC
options	NETGRAPH_UI
options	NETGRAPH_TTY
options	NETGRAPH_TEE
options	NETGRAPH_SOCKET
options	NETGRAPH_RFC1490
options	NETGRAPH_PPTPGRE
options	NETGRAPH_PPPOE
options	NETGRAPH_PPP
options	NETGRAPH_ONE2MANY
options	NETGRAPH_MPPC_ENCRYPTION
options	NETGRAPH_LMI
options	NETGRAPH_KSOCKET
options	NETGRAPH_EIFACE
options	NETGRAPH_IFACE
options	NETGRAPH_HOLE
options	NETGRAPH_FRAME_RELAY
options	NETGRAPH_ETHER
options	NETGRAPH_ECHO
options	NETGRAPH_CISCO
options	NETGRAPH_BRIDGE
options	NETGRAPH_BPF
options	NETGRAPH_ASYNC
options	NETGRAPH_L2TP
options	NETGRAPH_VLAN
options	NETGRAPH
options	IPSTEALTH
options	IPFIREWALL_VERBOSE_LIMIT
options	IPFIREWALL_VERBOSE
options	IPFIREWALL_FORWARD
options	IPFIREWALL_DEFAULT_TO_ACCEPT
options	UNIONFS
options	GEOM_BDE
options	GEOM_ELI
options	GEOM_UZIP
options	GEOM_MIRROR
options	DDB
options	KDB
options	USB_DEBUG
options	AH_SUPPORT_AR5416
options	IEEE80211_SUPPORT_MESH
options	IEEE80211_AMPDU_AGE
options	IEEE80211_DEBUG
options	AHD_REG_PRETTY_PRINT
options	AHC_REG_PRETTY_PRINT
options	ATA_STATIC_ID
options	SMP
options	INCLUDE_CONFIG_FILE
options	AUDIT
options	HWPMC_HOOKS
options	KBD_INSTALL_CDEV
options	PRINTF_BUFR_SIZE=128
options	_KPOSIX_PRIORITY_SCHEDULING
options	P1003_1B_SEMAPHORES
options	SYSVSEM
options	SYSVMSG
options	SYSVSHM
options	STACK
options	KTRACE
options	SCSI_DELAY=5000
options	COMPAT_FREEBSD7
options	GEOM_LABEL
options	GEOM_PART_GPT
options	PSEUDOFS
options	PROCFS
options	CD9660
options	MSDOSFS
options	NFS_ROOT
options	NFSLOCKD
options	NFSSERVER
options	NFSCLIENT
options	MD_ROOT
options	UFS_GJOURNAL
options	UFS_DIRHASH
options	UFS_ACL
options	SOFTUPDATES
options	FFS
options	SCTP
options	INET6
options	INET
options	PREEMPTION
options	SCHED_ULE
options	NATIVE
options	GEOM_PART_MBR
options	GEOM_PART_EBR_COMPAT
options	GEOM_PART_EBR
options	GEOM_PART_BSD
options	ISAPNP
device	isa
device	npx
device	mem
device	io
device	uart_ns8250
device	atpic
device	apic
device	cpufreq
device	acpi
device	eisa
device	pci
device	fdc
device	ata
device	atadisk
device	ataraid
device	atapicd
device	atapifd
device	atapist
device	ahb
device	ahc
device	ahd
device	amd
device	hptiop
device	isp
device	mpt
device	sym
device	trm
device	adv
device	adw
device	aha
device	aic
device	bt
device	ncv
device	nsp
device	stg
device	scbus
device	ch
device	da
device	sa
device	cd
device	pass
device	ses
device	amr
device	arcmsr
device	asr
device	ciss
device	dpt
device	hptmv
device	hptrr
device	iir
device	ips
device	mly
device	twa
device	aac
device	aacp
device	ida
device	mfi
device	mlx
device	pst
device	twe
device	atkbdc
device	atkbd
device	psm
device	kbdmux
device	vga
device	splash
device	sc
device	agp
device	pmtimer
device	cbb
device	pccard
device	cardbus
device	uart
device	ppc
device	ppbus
device	lpt
device	plip
device	ppi
device	de
device	em
device	igb
device	ixgb
device	le
device	ti
device	txp
device	vx
device	miibus
device	ae
device	age
device	alc
device	ale
device	bce
device	bfe
device	bge
device	dc
device	et
device	fxp
device	jme
device	lge
device	msk
device	nfe
device	nge
device	pcn
device	re
device	rl
device	sf
device	sge
device	sis
device	sk
device	ste
device	stge
device	tl
device	tx
device	vge
device	vr
device	wb
device	xl
device	cs
device	ed
device	ex
device	ep
device	fe
device	ie
device	sn
device	xe
device	wlan
device	wlan_wep
device	wlan_ccmp
device	wlan_tkip
device	wlan_amrr
device	an
device	ath
device	ath_hal
device	ath_rate_sample
device	ral
device	wi
device	loop
device	random
device	ether
device	vlan
device	tun
device	pty
device	md
device	gif
device	faith
device	firmware
device	bpf
device	uhci
device	ohci
device	ehci
device	usb
device	uhid
device	ukbd
device	ulpt
device	umass
device	ums
device	urio
device	u3g
device	uark
device	ubsa
device	uftdi
device	uipaq
device	uplcom
device	uslcom
device	uvisor
device	uvscom
device	aue
device	axe
device	cdce
device	cue
device	kue
device	rue
device	udav
device	rum
device	uath
device	ural
device	zyd
device	firewire
device	fwe
device	fwip
device	dcons
device	dcons_crom
device	enc
device	wlan_xauth
device	wlan_acl
device	iwi
device	ipw
device	ipwfw
device	wpi
device	wpifw
device	iwn
device	iwnfw
device	ralfw
device	urtw
device	mwl
device	mwlfw
device	malo
device	upgt
device	bwi
device	siba_bwn
device	bwn
device	run
device	carp
device	pf
device	pflog
device	pfsync
device	gre
device	ufoma
device	ucom
device	umct
device	umodem
device	if_bridge
device	tap
device	hme
device	speaker
device	lagg
device	bktr
device	safe
device	mxge
device	cxgb
device	ixgbe
device	cas
device	crypto
device	cryptodev
device	hifn
device	ubsec
device	padlock

ddb.txt

db:0:kdb.enter.default>  run lockinfo
db:1:lockinfo> show locks
No such command
db:1:locks>  show alllocks
No such command
db:1:alllocks>  show lockedvnods
Locked vnodes
db:0:kdb.enter.default>  show pcpu
cpuid        = 2
dynamic pcpu    = 0x467b880
curthread    = 0xc6318780: pid 87 "clog"
curpcb       = 0xc5ba9d90
fpcurthread  = none
idlethread   = 0xc5d8f780: pid 11 "idle: cpu2"
APIC ID      = 2
currentldt   = 0x50
db:0:kdb.enter.default>  bt
Tracing pid 87 tid 64083 td 0xc6318780
kdb_enter(c0ee668e,c0ee668e,c0f08858,c5ba9a68,2,...) at kdb_enter+0x3b
panic(c0f08858,c634653c,0,0,4000,...) at panic+0xff
ffs_write(c5ba9bbc,80400,c6346594,0,c5ba9b54,...) at ffs_write+0x7d
VOP_WRITE_APV(c12ebec0,c5ba9bbc,c0eee7f0,25f,0,...) at VOP_WRITE_APV+0x96
vn_write(c62b2700,c5ba9c54,c5d8a400,0,c6318780,...) at vn_write+0x166
dofilewrite(c5ba9c54,ffffffff,ffffffff,0,c62b2700,...) at dofilewrite+0x84
kern_writev(c6318780,3,c5ba9c54,c5ba9c74,1,...) at kern_writev+0x6a
write(c6318780,c5ba9cf8,c,c6318780,c,...) at write+0x50
syscall(c5ba9d38) at syscall+0x1ad
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (4, FreeBSD ELF32, write), eip = 0x28184573, esp = 0xbfbfad9c, ebp = 0xbfbfede8 ---
db:0:kdb.enter.default>  ps
  pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
   87    24    24     0  R+      CPU 2               clog
   39     0     0     0  SL      mdwait   0xc6250000 [md0]
   24     1    24     0  Ss+     wait     0xc5e997f8 sh
   23     0     0     0  SL      sdflush  0xc136ac60 [softdepflush]
   22     0     0     0  SL      syncer   0xc134fd94 [syncer]
   21     0     0     0  SL      vlruwt   0xc6204000 [vnlru]
   20     0     0     0  SL      psleep   0xc134fac8 [bufdaemon]
   19     0     0     0  SL      pollid   0xc133b77c [idlepoll]
   18     0     0     0  SL      pgzero   0xc136b914 [pagezero]
   17     0     0     0  SL      psleep   0xc136b53c [vmdaemon]
   16     0     0     0  SL      psleep   0xc136b504 [pagedaemon]
    9     0     0     0  SL      ccb_scan 0xc1306454 [xpt_thrd]
    8     0     0     0  SL      pftm     0xc04f6d80 [pfpurge]
    7     0     0     0  SL      waiting_ 0xc13571d8 [sctp_iterator]
   15     0     0     0  SL      (threaded)          usb
 64045                   D       -        0xc6004d0c [usbus1]
 64044                   D       -        0xc6004cdc [usbus1]
 64043                   D       -        0xc6004cac [usbus1]
 64042                   D       -        0xc6004c7c [usbus1]
 64039                   D       -        0xc5fd5d0c [usbus0]
 64038                   D       -        0xc5fd5cdc [usbus0]
 64037                   D       -        0xc5fd5cac [usbus0]
 64036                   D       -        0xc5fd5c7c [usbus0]
   14     0     0     0  SL      -        0xc133c084 [yarrow]
    6     0     0     0  SL      crypto_r 0xc136a28c [crypto returns]
    5     0     0     0  SL      crypto_w 0xc136a268 [crypto]
    4     0     0     0  SL      -        0xc1339924 [g_down]
    3     0     0     0  SL      -        0xc1339920 [g_up]
    2     0     0     0  SL      -        0xc1339918 [g_event]
   13     0     0     0  SL      (threaded)          ng_queue
 64017                   D       sleep    0xc12dcae0 [ng_queue3]
 64016                   D       sleep    0xc12dcae0 [ng_queue2]
 64015                   D       sleep    0xc12dcae0 [ng_queue1]
 64014                   D       sleep    0xc12dcae0 [ng_queue0]
   12     0     0     0  WL      (threaded)          intr
 64051                   I                           [irq1: atkbd0]
 64050                   I                           [irq15: ata1]
 64049                   I                           [irq14: ata0]
 64048                   I                           [swi0: uart]
 64047                   I                           [irq5: ppc1]
 64046                   I                           [irq20: atapci0+]
 64041                   I                           [irq17: rl1]
 64040                   I                           [irq16: rl0]
 64035                   I                           [irq23: ehci0 ehci1]
 64034                   I                           [irq9: acpi0]
 64030                   I                           [swi6: task queue]
 64029                   I                           [swi6: Giant taskq]
 64027                   I                           [swi5: +]
 64025                   I                           [swi2: cambio]
 64013                   I                           [swi4: clock]
 64012                   I                           [swi4: clock]
 64011                   I                           [swi4: clock]
 64010                   I                           [swi4: clock]
 64009                   I                           [swi1: netisr 0]
 64008                   I                           [swi3: vm]
   11     0     0     0  RL      (threaded)          idle
 64007                   Run     CPU 0               [idle: cpu0]
 64006                   Run     CPU 1               [idle: cpu1]
 64005                   CanRun                      [idle: cpu2]
 64004                   Run     CPU 3               [idle: cpu3]
    1     0     1     0  SLs     wait     0xc5d8dd48 [init]
   10     0     0     0  SL      audit_wo 0xc136a600 [audit]
    0     0     0     0  SLs     (threaded)          kernel
 64033                   D       -        0xc5f5f180 [acpi_task_2]
 64032                   D       -        0xc5f5f180 [acpi_task_1]
 64031                   D       -        0xc5f5f180 [acpi_task_0]
 64028                   D       -        0xc5f5f2c0 [thread]
 64026                   D       -        0xc5f5f680 [kqueue taskq]
 64018                   D       -        0xc5d73600 [firmware taskq]
 64001                   D       sched    0xc1339a00 [swapper]
db:0:kdb.enter.default>  alltrace

Tracing command clog pid 87 tid 64083 td 0xc6318780
kdb_enter(c0ee668e,c0ee668e,c0f08858,c5ba9a68,2,...) at kdb_enter+0x3b
panic(c0f08858,c634653c,0,0,4000,...) at panic+0xff
ffs_write(c5ba9bbc,80400,c6346594,0,c5ba9b54,...) at ffs_write+0x7d
VOP_WRITE_APV(c12ebec0,c5ba9bbc,c0eee7f0,25f,0,...) at VOP_WRITE_APV+0x96
vn_write(c62b2700,c5ba9c54,c5d8a400,0,c6318780,...) at vn_write+0x166
dofilewrite(c5ba9c54,ffffffff,ffffffff,0,c62b2700,...) at dofilewrite+0x84
kern_writev(c6318780,3,c5ba9c54,c5ba9c74,1,...) at kern_writev+0x6a
write(c6318780,c5ba9cf8,c,c6318780,c,...) at write+0x50
syscall(c5ba9d38) at syscall+0x1ad
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (4, FreeBSD ELF32, write), eip = 0x28184573, esp = 0xbfbfad9c, ebp = 0xbfbfede8 ---

Tracing command md0 pid 39 tid 64078 td 0xc6319000
sched_switch(c6319000,0,104,eb4aa2fe,21,...) at sched_switch+0xb9
mi_switch(104,0,4c,c6250000,c5b96c84,...) at mi_switch+0xd6
sleepq_switch(c6319000,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c6250000,4c,c0eb850c,0,0,...) at sleepq_wait+0x45
_sleep(c6250000,c6250020,24c,c0eb850c,0,...) at _sleep+0x33e
md_kthread(c6250000,c5b96d38,0,0,0,...) at md_kthread+0x1f6
fork_exit(c07eac30,c6250000,c5b96d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5b96d70, ebp = 0 ---

Tracing command sh pid 24 tid 64063 td 0xc6207c80
sched_switch(c6207c80,0,104,5cff3608,22,...) at sched_switch+0xb9
mi_switch(104,0,0,c5e99880,c5b5ab7c,...) at mi_switch+0xd6
sleepq_switch(c6207c80,0,c0ee9712,1a0,0,...) at sleepq_switch+0xae
sleepq_catch_signals(0,c5e99880,c5b5abc8,c0a448a9,c5e997f8,...) at sleepq_catch_signals+0x33b
sleepq_wait_sig(c5e997f8,5c,c0efafc7,100,0,...) at sleepq_wait_sig+0x18
_sleep(c5e997f8,c5e99880,15c,c0efafc7,0,...) at _sleep+0x279
kern_wait(c6207c80,ffffffff,c5b5ac70,2,0,...) at kern_wait+0x747
wait4(c6207c80,c5b5acf8,c,c6207c80,c,...) at wait4+0x3b
syscall(c5b5ad38) at syscall+0x1ad
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x2816469b, esp = 0xbfbfe52c, ebp = 0xbfbfe548 ---

Tracing command softdepflush pid 23 tid 64062 td 0xc6209000
sched_switch(c6209000,0,104,a1befeb5,21,...) at sched_switch+0xb9
mi_switch(104,0,c6209000,c136ac04,c5b56c88,...) at mi_switch+0xd6
sleepq_switch(c6209000,0,c0ee9712,283,0,...) at sleepq_switch+0xae
sleepq_timedwait(c136ac60,44,c0f07913,0,0,...) at sleepq_timedwait+0x3e
_sleep(c136ac60,c136ac04,44,c0f07913,3e8,...) at _sleep+0x323
softdep_flush(0,c5b56d38,d141a87b,3e8c1ea,44f78a45,...) at softdep_flush+0x3b1
fork_exit(c0cd9090,0,c5b56d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5b56d70, ebp = 0 ---

Tracing command syncer pid 22 tid 64061 td 0xc6209280
sched_switch(c6209280,0,104,a1efb2cd,21,...) at sched_switch+0xb9
mi_switch(104,0,c6209280,c6209280,c5b53c64,...) at mi_switch+0xd6
sleepq_switch(c6209280,0,c0ee9712,283,0,...) at sleepq_switch+0xae
sleepq_timedwait(c134fd94,0,c0eedc47,1,0,...) at sleepq_timedwait+0x3e
_cv_timedwait(c134fd94,c134fd80,3e8,c5e99d48,4e20,...) at _cv_timedwait+0x133
sched_sync(0,c5b53d38,ffffffff,535f2e5c,53445f42,...) at sched_sync+0x492
fork_exit(c0ad06e0,0,c5b53d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5b53d70, ebp = 0 ---

Tracing command vnlru pid 21 tid 64060 td 0xc6209500
sched_switch(c6209500,0,104,a1da8421,21,...) at sched_switch+0xb9
mi_switch(104,0,c6209500,c134fd54,c5b50c58,...) at mi_switch+0xd6
sleepq_switch(c6209500,0,c0ee9712,283,0,...) at sleepq_switch+0xae
sleepq_timedwait(c6204000,50,c0eee420,0,0,...) at sleepq_timedwait+0x3e
_sleep(c6204000,c134fd54,250,c0eee420,3e8,...) at _sleep+0x323
vnlru_proc(0,c5b50d38,a305845,2f5c8602,42535f03,...) at vnlru_proc+0x659
fork_exit(c0aceb80,0,c5b50d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5b50d70, ebp = 0 ---

Tracing command bufdaemon pid 20 tid 64059 td 0xc6209780
sched_switch(c6209780,0,104,a1d28222,21,...) at sched_switch+0xb9
mi_switch(104,0,c6209780,c134facc,c5b4dc98,...) at mi_switch+0xd6
sleepq_switch(c6209780,0,c0ee9712,283,0,...) at sleepq_switch+0xae
sleepq_timedwait(c134fac8,44,c0eec633,0,0,...) at sleepq_timedwait+0x3e
_sleep(c134fac8,c134facc,44,c0eec633,3e8,...) at _sleep+0x323
buf_daemon(0,c5b4dd38,0,1ee10000,0,...) at buf_daemon+0x1bd
fork_exit(c0ab42f0,0,c5b4dd38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5b4dd70, ebp = 0 ---

Tracing command idlepoll pid 19 tid 64058 td 0xc5fee780
sched_switch(c5fee780,0,104,a1d26f4c,21,...) at sched_switch+0xb9
mi_switch(104,0,c5fee780,0,c5b4ac88,...) at mi_switch+0xd6
sleepq_switch(c5fee780,0,c0ee9712,283,0,...) at sleepq_switch+0xae
sleepq_timedwait(c133b77c,0,c0ee5ab1,0,0,...) at sleepq_timedwait+0x3e
_sleep(c133b77c,0,0,c0ee5ab1,bb8,...) at _sleep+0x323
poll_idle(0,c5b4ad38,1000986,fee00000,10000,...) at poll_idle+0x1aa
fork_exit(c0a2dd30,0,c5b4ad38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5b4ad70, ebp = 0 ---

Tracing command pagezero pid 18 tid 64057 td 0xc5feea00
sched_switch(c5feea00,0,104,66f96d29,1f,...) at sched_switch+0xb9
mi_switch(104,0,c5feea00,c136b4d0,c5b47c98,...) at mi_switch+0xd6
sleepq_switch(c5feea00,0,c0ee9712,283,0,...) at sleepq_switch+0xae
sleepq_timedwait(c136b914,0,c0f0b4d5,0,0,...) at sleepq_timedwait+0x3e
_sleep(c136b914,c136b4d0,0,c0f0b4d5,493e0,...) at _sleep+0x323
vm_pagezero(0,c5b47d38,9410046e,45bc1d26,cade58b6,...) at vm_pagezero+0x7c
fork_exit(c0d1d900,0,c5b47d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5b47d70, ebp = 0 ---

Tracing command vmdaemon pid 17 tid 64056 td 0xc5feec80
sched_switch(c5feec80,0,104,66f8815f,1f,...) at sched_switch+0xb9
mi_switch(104,0,68,c136b53c,c5b44c5c,...) at mi_switch+0xd6
sleepq_switch(c5feec80,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c136b53c,68,c0eec633,0,0,...) at sleepq_wait+0x45
_sleep(c136b53c,c136b540,68,c0eec633,0,...) at _sleep+0x33e
vm_daemon(0,c5b44d38,5824448b,550d8d4c,48000027,...) at vm_daemon+0x56
fork_exit(c0d1a1d0,0,c5b44d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5b44d70, ebp = 0 ---

Tracing command pagedaemon pid 16 tid 64055 td 0xc6027000
sched_switch(c6027000,0,104,66fb45eb,1f,...) at sched_switch+0xb9
mi_switch(104,0,c6027000,c136b4d0,c5b41bf4,...) at mi_switch+0xd6
sleepq_switch(c6027000,0,c0ee9712,283,0,...) at sleepq_switch+0xae
sleepq_timedwait(c136b504,44,c0eec633,0,0,...) at sleepq_timedwait+0x3e
_sleep(c136b504,c136b4d0,44,c0eec633,1388,...) at _sleep+0x323
vm_pageout(0,c5b41d38,8d487974,802494,cb8b0000,...) at vm_pageout+0x823
fork_exit(c0d18bc0,0,c5b41d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5b41d70, ebp = 0 ---

Tracing command xpt_thrd pid 9 tid 64054 td 0xc6027280
sched_switch(c6027280,0,104,654b4db1,1f,...) at sched_switch+0xb9
mi_switch(104,0,4c,c1306454,c5b3ec88,...) at mi_switch+0xd6
sleepq_switch(c6027280,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c1306454,4c,c0e752ac,0,0,...) at sleepq_wait+0x45
_sleep(c1306454,c1306498,4c,c0e752ac,0,...) at _sleep+0x33e
xpt_scanner_thread(0,c5b3ed38,48382444,4024548d,248c8d48,...) at xpt_scanner_thread+0x11c
fork_exit(c0494640,0,c5b3ed38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5b3ed70, ebp = 0 ---

Tracing command pfpurge pid 8 tid 64053 td 0xc6027500
sched_switch(c6027500,0,104,4187c5c3,22,...) at sched_switch+0xb9
mi_switch(104,0,c6027500,0,c5b3bc8c,...) at mi_switch+0xd6
sleepq_switch(c6027500,0,c0ee9712,283,0,...) at sleepq_switch+0xae
sleepq_timedwait(c04f6d80,5c,c0e83c62,0,0,...) at sleepq_timedwait+0x3e
_sleep(c04f6d80,0,5c,c0e83c62,3e8,...) at _sleep+0x323
pf_purge_thread(0,c5b3bd38,0,0,0,...) at pf_purge_thread+0x3d
fork_exit(c04f6d80,0,c5b3bd38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5b3bd70, ebp = 0 ---

Tracing command sctp_iterator pid 7 tid 64052 td 0xc6027780
sched_switch(c6027780,0,104,486d9921,1f,...) at sched_switch+0xb9
mi_switch(104,0,0,c13571d8,c5b38c94,...) at mi_switch+0xd6
sleepq_switch(c6027780,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c13571d8,0,c0efa516,0,0,...) at sleepq_wait+0x45
_sleep(c13571d8,c13571a0,0,c0efa516,0,...) at _sleep+0x33e
sctp_iterator_thread(0,c5b38d38,48502444,20244489,1b8,...) at sctp_iterator_thread+0x58
fork_exit(c0ba7170,0,c5b38d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5b38d70, ebp = 0 ---

Tracing command usb pid 15 tid 64045 td 0xc5f7cc80
sched_switch(c5f7cc80,0,104,5ddbcb56,21,...) at sched_switch+0xb9
mi_switch(104,0,0,c6004d0c,c5a7fc90,...) at mi_switch+0xd6
sleepq_switch(c5f7cc80,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c6004d0c,0,c0edc643,1,0,...) at sleepq_wait+0x45
_cv_wait(c6004d0c,c6004dac,c0ed4bc9,68,0,...) at _cv_wait+0x11b
usb_process(c6004d04,c5a7fd38,0,0,0,...) at usb_process+0x15f
fork_exit(c09297d0,c6004d04,c5a7fd38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5a7fd70, ebp = 0 ---

Tracing command usb pid 15 tid 64044 td 0xc5fed000
sched_switch(c5fed000,0,104,5ddbecaa,21,...) at sched_switch+0xb9
mi_switch(104,0,0,c6004cdc,c5a7cc90,...) at mi_switch+0xd6
sleepq_switch(c5fed000,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c6004cdc,0,c0edc643,1,0,...) at sleepq_wait+0x45
_cv_wait(c6004cdc,c6004dac,c0ed4bc9,68,0,...) at _cv_wait+0x11b
usb_process(c6004cd4,c5a7cd38,0,0,0,...) at usb_process+0x15f
fork_exit(c09297d0,c6004cd4,c5a7cd38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5a7cd70, ebp = 0 ---

Tracing command usb pid 15 tid 64043 td 0xc5fed280
sched_switch(c5fed280,0,104,5d4d3a60,21,...) at sched_switch+0xb9
mi_switch(104,0,0,c6004cac,c5a79c90,...) at mi_switch+0xd6
sleepq_switch(c5fed280,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c6004cac,0,c0edc643,1,0,...) at sleepq_wait+0x45
_cv_wait(c6004cac,c6004dac,c0ed4bc9,68,0,...) at _cv_wait+0x11b
usb_process(c6004ca4,c5a79d38,0,0,0,...) at usb_process+0x15f
fork_exit(c09297d0,c6004ca4,c5a79d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5a79d70, ebp = 0 ---

Tracing command usb pid 15 tid 64042 td 0xc5fed500
sched_switch(c5fed500,0,104,485e5bbd,1f,...) at sched_switch+0xb9
mi_switch(104,0,0,c6004c7c,c5a76c90,...) at mi_switch+0xd6
sleepq_switch(c5fed500,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c6004c7c,0,c0edc643,1,0,...) at sleepq_wait+0x45
_cv_wait(c6004c7c,c6004dac,c0ed4bc9,68,0,...) at _cv_wait+0x11b
usb_process(c6004c74,c5a76d38,0,0,0,...) at usb_process+0x15f
fork_exit(c09297d0,c6004c74,c5a76d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5a76d70, ebp = 0 ---

Tracing command usb pid 15 tid 64039 td 0xc5fedc80
sched_switch(c5fedc80,0,104,30163b59,21,...) at sched_switch+0xb9
mi_switch(104,0,0,c5fd5d0c,c5a41c90,...) at mi_switch+0xd6
sleepq_switch(c5fedc80,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c5fd5d0c,0,c0edc643,1,0,...) at sleepq_wait+0x45
_cv_wait(c5fd5d0c,c5fd5dac,c0ed4bc9,68,488d96cb,...) at _cv_wait+0x11b
usb_process(c5fd5d04,c5a41d38,66cde79d,42820325,f537c069,...) at usb_process+0x15f
fork_exit(c09297d0,c5fd5d04,c5a41d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5a41d70, ebp = 0 ---

Tracing command usb pid 15 tid 64038 td 0xc5fee000
sched_switch(c5fee000,0,104,30166147,21,...) at sched_switch+0xb9
mi_switch(104,0,0,c5fd5cdc,c5a3ec90,...) at mi_switch+0xd6
sleepq_switch(c5fee000,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c5fd5cdc,0,c0edc643,1,0,...) at sleepq_wait+0x45
_cv_wait(c5fd5cdc,c5fd5dac,c0ed4bc9,68,5371a133,...) at _cv_wait+0x11b
usb_process(c5fd5cd4,c5a3ed38,a2b3e50,d9bb3556,99d319a1,...) at usb_process+0x15f
fork_exit(c09297d0,c5fd5cd4,c5a3ed38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5a3ed70, ebp = 0 ---

Tracing command usb pid 15 tid 64037 td 0xc5fee280
sched_switch(c5fee280,0,104,2fb48f4c,21,...) at sched_switch+0xb9
mi_switch(104,0,0,c5fd5cac,c5a3bc90,...) at mi_switch+0xd6
sleepq_switch(c5fee280,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c5fd5cac,0,c0edc643,1,0,...) at sleepq_wait+0x45
_cv_wait(c5fd5cac,c5fd5dac,c0ed4bc9,68,5371a133,...) at _cv_wait+0x11b
usb_process(c5fd5ca4,c5a3bd38,a2b3e50,d9bb3556,99d319a1,...) at usb_process+0x15f
fork_exit(c09297d0,c5fd5ca4,c5a3bd38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5a3bd70, ebp = 0 ---

Tracing command usb pid 15 tid 64036 td 0xc5fee500
sched_switch(c5fee500,0,104,484edb69,1f,...) at sched_switch+0xb9
mi_switch(104,0,0,c5fd5c7c,c5a38c90,...) at mi_switch+0xd6
sleepq_switch(c5fee500,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c5fd5c7c,0,c0edc643,1,0,...) at sleepq_wait+0x45
_cv_wait(c5fd5c7c,c5fd5dac,c0ed4bc9,68,2946555c,...) at _cv_wait+0x11b
usb_process(c5fd5c74,c5a38d38,23606cc6,93e9a1f3,f091cb24,...) at usb_process+0x15f
fork_exit(c09297d0,c5fd5c74,c5a38d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5a38d70, ebp = 0 ---

Tracing command yarrow pid 14 tid 64024 td 0xc5dd4a00
sched_switch(c5dd4a00,0,104,5457fd87,22,...) at sched_switch+0xb9
mi_switch(104,0,c5dd4a00,0,c59dfc64,...) at mi_switch+0xd6
sleepq_switch(c5dd4a00,0,c0ee9712,283,0,...) at sleepq_switch+0xae
sleepq_timedwait(c133c084,0,c0edc643,2,0,...) at sleepq_timedwait+0x3e
_sleep(c133c084,0,0,c0edc643,64,...) at _sleep+0x323
pause(c0edc643,64,83e17ac,e5083a36,1c80440,...) at pause+0x30
random_kthread(0,c59dfd38,b0794af4,f72f687f,638aed13,...) at random_kthread+0x1b8
fork_exit(c08867f0,0,c59dfd38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59dfd70, ebp = 0 ---

Tracing command crypto returns pid 6 tid 64023 td 0xc5dd4c80
sched_switch(c5dd4c80,0,104,484ebec1,1f,...) at sched_switch+0xb9
mi_switch(104,0,5c,c136a28c,c59d8c80,...) at mi_switch+0xd6
sleepq_switch(c5dd4c80,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c136a28c,5c,c0f052d4,0,0,...) at sleepq_wait+0x45
_sleep(c136a28c,c136a29c,5c,c0f052d4,0,...) at _sleep+0x33e
crypto_ret_proc(0,c59d8d38,f5d4c1af,2644caa9,662ce65e,...) at crypto_ret_proc+0x19c
fork_exit(c0c97320,0,c59d8d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59d8d70, ebp = 0 ---

Tracing command crypto pid 5 tid 64022 td 0xc5e95000
sched_switch(c5e95000,0,104,484eb1ed,1f,...) at sched_switch+0xb9
mi_switch(104,0,5c,c136a268,c59d5c8c,...) at mi_switch+0xd6
sleepq_switch(c5e95000,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c136a268,5c,c0f052c8,0,0,...) at sleepq_wait+0x45
_sleep(c136a268,c136a278,5c,c0f052c8,0,...) at _sleep+0x33e
crypto_proc(0,c59d5d38,f5d4c1af,2644caa9,662ce65e,...) at crypto_proc+0x160
fork_exit(c0c96be0,0,c59d5d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59d5d70, ebp = 0 ---

Tracing command g_down pid 4 tid 64021 td 0xc5e95280
sched_switch(c5e95280,0,104,5cfb5bd8,22,...) at sched_switch+0xb9
mi_switch(104,0,4c,c1339924,c59d2c50,...) at mi_switch+0xd6
sleepq_switch(c5e95280,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c1339924,4c,c0edc643,0,0,...) at sleepq_wait+0x45
_sleep(c1339924,c1339888,24c,c0edc643,0,...) at _sleep+0x33e
g_io_schedule_down(c5e95280,4c,c0ede69d,72,0,...) at g_io_schedule_down+0x273
g_down_procbody(0,c59d2d38,0,0,0,...) at g_down_procbody+0x5b
fork_exit(c09c58a0,0,c59d2d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59d2d70, ebp = 0 ---

Tracing command g_up pid 3 tid 64020 td 0xc5e95500
sched_switch(c5e95500,0,104,5d1e00e8,22,...) at sched_switch+0xb9
mi_switch(104,0,4c,c1339920,c59cfc78,...) at mi_switch+0xd6
sleepq_switch(c5e95500,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c1339920,4c,c0edc643,0,0,...) at sleepq_wait+0x45
_sleep(c1339920,c13398a8,24c,c0edc643,0,...) at _sleep+0x33e
g_io_schedule_up(c5e95500,4c,c0ede69d,5b,0,...) at g_io_schedule_up+0xbd
g_up_procbody(0,c59cfd38,0,0,0,...) at g_up_procbody+0x5b
fork_exit(c09c5910,0,c59cfd38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59cfd70, ebp = 0 ---

Tracing command g_event pid 2 tid 64019 td 0xc5e95780
sched_switch(c5e95780,0,104,4d7d20d5,22,...) at sched_switch+0xb9
mi_switch(104,0,c5e95780,0,c59ccc94,...) at mi_switch+0xd6
sleepq_switch(c5e95780,0,c0ee9712,283,0,...) at sleepq_switch+0xae
sleepq_timedwait(c1339918,4c,c0edc643,0,0,...) at sleepq_timedwait+0x3e
_sleep(c1339918,0,4c,c0edc643,64,...) at _sleep+0x323
g_event_procbody(0,c59ccd38,0,0,0,...) at g_event_procbody+0x97
fork_exit(c09c5980,0,c59ccd38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59ccd70, ebp = 0 ---

Tracing command ng_queue pid 13 tid 64017 td 0xc5e95c80
sched_switch(c5e95c80,0,104,484df2cd,1f,...) at sched_switch+0xb9
mi_switch(104,0,10,c12dcae0,c59c6c8c,...) at mi_switch+0xd6
sleepq_switch(c5e95c80,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c12dcae0,10,c0eecb1b,0,0,...) at sleepq_wait+0x45
_sleep(c12dcae0,c1352924,10,c0eecb1b,0,...) at _sleep+0x33e
ngthread(0,c59c6d38,0,0,0,...) at ngthread+0x5c
fork_exit(c0b5b350,0,c59c6d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59c6d70, ebp = 0 ---

Tracing command ng_queue pid 13 tid 64016 td 0xc5d91280
sched_switch(c5d91280,0,104,484de859,1f,...) at sched_switch+0xb9
mi_switch(104,0,10,c12dcae0,c59c3c8c,...) at mi_switch+0xd6
sleepq_switch(c5d91280,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c12dcae0,10,c0eecb1b,0,0,...) at sleepq_wait+0x45
_sleep(c12dcae0,c1352924,10,c0eecb1b,0,...) at _sleep+0x33e
ngthread(0,c59c3d38,0,0,0,...) at ngthread+0x5c
fork_exit(c0b5b350,0,c59c3d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59c3d70, ebp = 0 ---

Tracing command ng_queue pid 13 tid 64015 td 0xc5d91500
sched_switch(c5d91500,0,104,484dde65,1f,...) at sched_switch+0xb9
mi_switch(104,0,10,c12dcae0,c59c0c8c,...) at mi_switch+0xd6
sleepq_switch(c5d91500,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c12dcae0,10,c0eecb1b,0,0,...) at sleepq_wait+0x45
_sleep(c12dcae0,c1352924,10,c0eecb1b,0,...) at _sleep+0x33e
ngthread(0,c59c0d38,0,0,0,...) at ngthread+0x5c
fork_exit(c0b5b350,0,c59c0d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59c0d70, ebp = 0 ---

Tracing command ng_queue pid 13 tid 64014 td 0xc5d91780
sched_switch(c5d91780,0,104,484dd2a1,1f,...) at sched_switch+0xb9
mi_switch(104,0,10,c12dcae0,c59bdc8c,...) at mi_switch+0xd6
sleepq_switch(c5d91780,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c12dcae0,10,c0eecb1b,0,0,...) at sleepq_wait+0x45
_sleep(c12dcae0,c1352924,10,c0eecb1b,0,...) at _sleep+0x33e
ngthread(0,c59bdd38,0,0,0,...) at ngthread+0x5c
fork_exit(c0b5b350,0,c59bdd38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59bdd70, ebp = 0 ---

Tracing command intr pid 12 tid 64051 td 0xc6027a00
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 64050 td 0xc6027c80
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 64049 td 0xc6028000
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 64048 td 0xc6028280
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 64047 td 0xc6028500
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 64046 td 0xc5f7ca00
sched_switch(c5f7ca00,0,109,5d1af8f0,22,...) at sched_switch+0xb9
mi_switch(109,0,c0ee305c,52d,c5dd34f0,...) at mi_switch+0xd6
ithread_loop(c601c0c0,c5a82d38,0,0,0,...) at ithread_loop+0x1a1
fork_exit(c0a11aa0,c601c0c0,c5a82d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5a82d70, ebp = 0 ---

Tracing command intr pid 12 tid 64041 td 0xc5fed780
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 64040 td 0xc5feda00
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 64035 td 0xc5f7b000
sched_switch(c5f7b000,0,109,5ddbadc3,21,...) at sched_switch+0xb9
mi_switch(109,0,c0ee305c,52d,c5dd3370,...) at mi_switch+0xd6
ithread_loop(c5fcf290,c5a35d38,23606cc6,93e9a1f3,f091cb24,...) at ithread_loop+0x1a1
fork_exit(c0a11aa0,c5fcf290,c5a35d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5a35d70, ebp = 0 ---

Tracing command intr pid 12 tid 64034 td 0xc5f7b280
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 64030 td 0xc5f7bc80
sched_switch(c5f7bc80,0,109,654a56a1,1f,...) at sched_switch+0xb9
mi_switch(109,0,c0ee305c,52d,c5f7a8f0,...) at mi_switch+0xd6
ithread_loop(c5d8c430,c59f1d38,fabadf73,259fcd1e,f4b6c36a,...) at ithread_loop+0x1a1
fork_exit(c0a11aa0,c5d8c430,c59f1d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59f1d70, ebp = 0 ---

Tracing command intr pid 12 tid 64029 td 0xc5f7c000
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 64027 td 0xc5f7c500
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 64025 td 0xc5dd4780
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 64013 td 0xc5d91a00
sched_switch(c5d91a00,0,109,59057ce2,22,...) at sched_switch+0xb9
mi_switch(109,0,c0ee305c,52d,c5dd3070,...) at mi_switch+0xd6
ithread_loop(c5d8c0d0,c59bad38,0,0,0,...) at ithread_loop+0x1a1
fork_exit(c0a11aa0,c5d8c0d0,c59bad38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59bad70, ebp = 0 ---

Tracing command intr pid 12 tid 64012 td 0xc5d91c80
sched_switch(c5d91c80,0,109,5456efd3,22,...) at sched_switch+0xb9
mi_switch(109,0,c0ee305c,52d,c5dd30f0,...) at mi_switch+0xd6
ithread_loop(c5d8c0e0,c59b7d38,0,0,0,...) at ithread_loop+0x1a1
fork_exit(c0a11aa0,c5d8c0e0,c59b7d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59b7d70, ebp = 0 ---

Tracing command intr pid 12 tid 64011 td 0xc5dd4000
sched_switch(c5dd4000,0,109,4be245cc,22,...) at sched_switch+0xb9
mi_switch(109,0,c0ee305c,52d,c5dd3170,...) at mi_switch+0xd6
ithread_loop(c5d8c0f0,c59b4d38,0,0,0,...) at ithread_loop+0x1a1
fork_exit(c0a11aa0,c5d8c0f0,c59b4d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59b4d70, ebp = 0 ---

Tracing command intr pid 12 tid 64010 td 0xc5dd4280
sched_switch(c5dd4280,0,109,5d204c88,22,...) at sched_switch+0xb9
mi_switch(109,0,c0ee305c,52d,c5dd31f0,...) at mi_switch+0xd6
ithread_loop(c5d8c100,c59b1d38,0,0,0,...) at ithread_loop+0x1a1
fork_exit(c0a11aa0,c5d8c100,c59b1d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59b1d70, ebp = 0 ---

Tracing command intr pid 12 tid 64009 td 0xc5dd4500
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 64008 td 0xc5d8f000
fork_trampoline() at fork_trampoline

Tracing command idle pid 11 tid 64007 td 0xc5d8f280
cpustop_handler(1,c59a7c34,c0def345,369e99,0,...) at cpustop_handler+0x32
ipi_nmi_handler(369e99,0,0,c59a7bfc,c5d8daa0,...) at ipi_nmi_handler+0x2c
trap(c59a7c40) at trap+0x255
calltrap() at calltrap+0x6
--- trap 0x13, eip = 0xc0dc4bf5, esp = 0xc59a7c80, ebp = 0xc59a7c80 ---
acpi_cpu_c1(1,c59a7c94,0,0,ffffffff,1) at acpi_cpu_c1+0x5
acpi_cpu_idle(1,c59a7ccc,c0ee77d5,9f9,c5d8f280,...) at acpi_cpu_idle+0x1b9
sched_idletd(0,c59a7d38,0,0,0,...) at sched_idletd+0x117
fork_exit(c0a5e060,0,c59a7d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59a7d70, ebp = 0 ---

Tracing command idle pid 11 tid 64006 td 0xc5d8f500
cpustop_handler(2,c59a4c34,c0def345,c5e95500,c1340680,...) at cpustop_handler+0x32
ipi_nmi_handler(c5e95500,c1340680,46,c133c29c,c5d8daa0,...) at ipi_nmi_handler+0x2c
trap(c59a4c40) at trap+0x255
calltrap() at calltrap+0x6
--- trap 0x13, eip = 0xc0dc4bf5, esp = 0xc59a4c80, ebp = 0xc59a4c80 ---
acpi_cpu_c1(1,c59a4c94,1,0,ffffffff,2) at acpi_cpu_c1+0x5
acpi_cpu_idle(1,c59a4ccc,c0ee77d5,9f9,c5d8f500,...) at acpi_cpu_idle+0x1b9
sched_idletd(0,c59a4d38,0,0,0,...) at sched_idletd+0x117
fork_exit(c0a5e060,0,c59a4d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59a4d70, ebp = 0 ---

Tracing command idle pid 11 tid 64005 td 0xc5d8f780
sched_switch(c5d8f780,0,60c,5cfeb0b8,22,...) at sched_switch+0xb9
mi_switch(60c,0,c0ee77d5,805,2,...) at mi_switch+0xd6
sched_preempt(c5d8f780,2,1f4,c59a1c80,c0dd10ae,...) at sched_preempt+0xaf
ipi_bitmap_handler(c0f20008,28,c5ea0028,c5ea1800,c5ea1818,...) at ipi_bitmap_handler+0x6e
Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x2e
--- interrupt, eip = 0xc0dc4bf5, esp = 0xc59a1c80, ebp = 0xc59a1c80 ---
acpi_cpu_c1(1,c59a1c94,2,0,ffffffff,4) at acpi_cpu_c1+0x5
acpi_cpu_idle(1,c59a1ccc,c0ee77d5,9f9,c5d8f780,...) at acpi_cpu_idle+0x1b9
sched_idletd(0,c59a1d38,0,0,0,...) at sched_idletd+0x117
fork_exit(c0a5e060,0,c59a1d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59a1d70, ebp = 0 ---

Tracing command idle pid 11 tid 64004 td 0xc5d8fa00
cpustop_handler(8,c599ec34,c0def345,c5dd4280,c1341380,...) at cpustop_handler+0x32
ipi_nmi_handler(c5dd4280,c1341380,46,c133c314,c5d8daa0,...) at ipi_nmi_handler+0x2c
trap(c599ec40) at trap+0x255
calltrap() at calltrap+0x6
--- trap 0x13, eip = 0xc0dc4bf5, esp = 0xc599ec80, ebp = 0xc599ec80 ---
acpi_cpu_c1(1,c599ec94,3,0,ffffffff,8) at acpi_cpu_c1+0x5
acpi_cpu_idle(1,c599eccc,c0ee77d5,9f9,c5d8fa00,...) at acpi_cpu_idle+0x1b9
sched_idletd(0,c599ed38,0,0,0,...) at sched_idletd+0x117
fork_exit(c0a5e060,0,c599ed38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc599ed70, ebp = 0 ---

Tracing command init pid 1 tid 64003 td 0xc5d8fc80
sched_switch(c5d8fc80,0,104,6f9ee5e0,21,...) at sched_switch+0xb9
mi_switch(104,0,0,c5d8ddd0,c599ab7c,...) at mi_switch+0xd6
sleepq_switch(c5d8fc80,0,c0ee9712,1a0,0,...) at sleepq_switch+0xae
sleepq_catch_signals(0,c5d8ddd0,c599abc8,c0a448a9,c5d8dd48,...) at sleepq_catch_signals+0x33b
sleepq_wait_sig(c5d8dd48,5c,c0efafc7,100,0,...) at sleepq_wait_sig+0x18
_sleep(c5d8dd48,c5d8ddd0,15c,c0efafc7,0,...) at _sleep+0x279
kern_wait(c5d8fc80,ffffffff,c599ac70,2,0,...) at kern_wait+0x747
wait4(c5d8fc80,c599acf8,c,c5d8fc80,c,...) at wait4+0x3b
syscall(c599ad38) at syscall+0x1ad
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x8054b7f, esp = 0xbfbfe8cc, ebp = 0xbfbfe8e8 ---

Tracing command audit pid 10 tid 64002 td 0xc5d91000
sched_switch(c5d91000,0,104,484dc531,1f,...) at sched_switch+0xb9
mi_switch(104,0,0,c136a600,c5997c70,...) at mi_switch+0xd6
sleepq_switch(c5d91000,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c136a600,0,c0f05e3f,1,0,...) at sleepq_wait+0x45
_cv_wait(c136a600,c136a5e4,0,0,0,...) at _cv_wait+0x11b
audit_worker(0,c5997d38,0,0,0,...) at audit_worker+0x362
fork_exit(c0cbeba0,0,c5997d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc5997d70, ebp = 0 ---

Tracing command kernel pid 0 tid 64033 td 0xc5f7b500
sched_switch(c5f7b500,0,104,48a92955,1f,...) at sched_switch+0xb9
mi_switch(104,0,0,c5f5f180,c59faca0,...) at mi_switch+0xd6
sleepq_switch(c5f7b500,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c5f5f180,0,c0edc643,0,0,...) at sleepq_wait+0x45
msleep_spin(c5f5f180,c5f5f198,c0edc643,0,20,...) at msleep_spin+0x1ad
taskqueue_thread_loop(c130aa20,c59fad38,965c06be,bea6fb6d,98521aa2,...) at taskqueue_thread_loop+0x6e
fork_exit(c0a77de0,c130aa20,c59fad38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59fad70, ebp = 0 ---

Tracing command kernel pid 0 tid 64032 td 0xc5f7b780
sched_switch(c5f7b780,0,104,48a33539,1f,...) at sched_switch+0xb9
mi_switch(104,0,0,c5f5f180,c59f7ca0,...) at mi_switch+0xd6
sleepq_switch(c5f7b780,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c5f5f180,0,c0edc643,0,0,...) at sleepq_wait+0x45
msleep_spin(c5f5f180,c5f5f198,c0edc643,0,c59f7cf0,...) at msleep_spin+0x1ad
taskqueue_thread_loop(c130aa20,c59f7d38,965c06be,bea6fb6d,98521aa2,...) at taskqueue_thread_loop+0x6e
fork_exit(c0a77de0,c130aa20,c59f7d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59f7d70, ebp = 0 ---

Tracing command kernel pid 0 tid 64031 td 0xc5f7ba00
sched_switch(c5f7ba00,0,104,48a32b01,1f,...) at sched_switch+0xb9
mi_switch(104,0,0,c5f5f180,c59f4ca0,...) at mi_switch+0xd6
sleepq_switch(c5f7ba00,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c5f5f180,0,c0edc643,0,0,...) at sleepq_wait+0x45
msleep_spin(c5f5f180,c5f5f198,c0edc643,0,c59f4cf0,...) at msleep_spin+0x1ad
taskqueue_thread_loop(c130aa20,c59f4d38,fabadf73,259fcd1e,f4b6c36a,...) at taskqueue_thread_loop+0x6e
fork_exit(c0a77de0,c130aa20,c59f4d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59f4d70, ebp = 0 ---

Tracing command kernel pid 0 tid 64028 td 0xc5f7c280
sched_switch(c5f7c280,0,104,6560e7f9,1f,...) at sched_switch+0xb9
mi_switch(104,0,0,c5f5f2c0,c59ebc90,...) at mi_switch+0xd6
sleepq_switch(c5f7c280,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c5f5f2c0,0,c0edc643,0,0,...) at sleepq_wait+0x45
_sleep(c5f5f2c0,c5f5f2d8,0,c0edc643,0,...) at _sleep+0x33e
taskqueue_thread_loop(c134ea88,c59ebd38,d3f18de5,6fcd59bb,9df411ef,...) at taskqueue_thread_loop+0xc7
fork_exit(c0a77de0,c134ea88,c59ebd38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59ebd70, ebp = 0 ---

Tracing command kernel pid 0 tid 64026 td 0xc5f7c780
sched_switch(c5f7c780,0,104,486dc071,1f,...) at sched_switch+0xb9
mi_switch(104,0,0,c5f5f680,c59e5c90,...) at mi_switch+0xd6
sleepq_switch(c5f7c780,0,c0ee9712,260,0,...) at sleepq_switch+0xae
sleepq_wait(c5f5f680,0,c0edc643,0,0,...) at sleepq_wait+0x45
_sleep(c5f5f680,c5f5f698,0,c0edc643,0,...) at _sleep+0x33e
taskqueue_thread_loop(c133a2e4,c59e5d38,99321862,bd7dfcda,ac83f96,...) at taskqueue_thread_loop+0xc7
fork_exit(c0a77de0,c133a2e4,c59e5d38) at fork_exit+0x88
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc59e5d70, ebp = 0 ---

Tracing command kernel pid 0 tid 64018 td 0xc5e95a00
sched_switch(c5e95a00,0,104,608bbc92,21,...) at <[/thread]

[werter]

Бэкап конфига есть? Сносим полностью , ставим по-новой и заливаем конфиг. На разбор полетов вы времени больше потратите , чем на восстановление из бэкапа.
На себе проверено не один десяток раз.

[1qoot1]

@werter:

Бэкап конфига есть? Сносим полностью , ставим по-новой и заливаем конфиг. На разбор полетов вы времени больше потратите , чем на восстановление из бэкапа.
На себе проверено не один десяток раз.

Да бэкапы то есть, но все же как то странно у меня эта машина с января стояла.. из серии 1 раз настроил и забыл… так по мелочам гайки подкручивались на проксе, и подкручивались правила фаерволе

[dr.gopher]

@1qoot1:

Да бэкапы то есть, но все же как то странно у меня эта машина с января стояла.. из серии 1 раз настроил и забыл… так по мелочам гайки подкручивались на проксе, и подкручивались правила фаерволе

И у меня такаж хрень была. Переустановил и всё ок. Но осадочек остался. :-(

[1qoot1]

@dr.gopher:

@1qoot1:

Да бэкапы то есть, но все же как то странно у меня эта машина с января стояла.. из серии 1 раз настроил и забыл… так по мелочам гайки подкручивались на проксе, и подкручивались правила фаерволе

И у меня такаж хрень была. Переустановил и всё ок. Но осадочек остался. :-(

Согласен что осадок есть, но у меня не одна така машинка стоит…. около 16 =(

Вопрос чего ждать от других?

Есть и критически важные точки за которыми стоит и телефон и всякие пользовательские приблуды типа 1с и тд...

Грубо говоря у меня гибридная схема сети звезда-звезда.... т.е. первая звезда на pf вертится вторая на centos+vz.

Но как бы меня не поймут если вдруг первая звезда неожиданно отвалится c kernel panic...

[werter]

В 99 % случаев - проблема в железе. ИБП есть? Свет случаем резко и надолго не выключали?

[1qoot1]

@werter:

В 99 % случаев - проблема в железе. ИБП есть? Свет случаем резко и надолго не выключали?

Угу, куда ж без него родного…
А на счет света вроде бы нет...