Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Constant kernel panics

    Scheduled Pinned Locked Moved Hardware
    6 Posts 4 Posters 2.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      ajm786
      last edited by

      I upgraded from 2.0.1 to 2.0.2 on a new computer. I am getting constant kernel panics, and I have no idea how to troubleshoot them. I would greatly appreciate any help that you guys could give me on what is causing this. Here is the kernel panic info below.

      Crash report begins.  Anonymous machine information:
      
      amd64
      8.1-RELEASE-p13
      FreeBSD 8.1-RELEASE-p13 #0: Fri Dec  7 23:04:01 EST 2012     root@snapshots-8_1-amd64.builders.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8
      
      Crash report details:
      
      Filename: /var/crash/bounds
      1
      
      Filename: /var/crash/info.0
      Dump header from device /dev/ad0s1b
        Architecture: amd64
        Architecture Version: 1
        Dump Length: 123392B (0 MB)
        Blocksize: 512
        Dumptime: Sat Jan 26 11:45:41 2013
        Hostname: ajm786.localdomain
        Magic: FreeBSD Text Dump
        Version String: FreeBSD 8.1-RELEASE-p13 #0: Fri Dec  7 23:04:01 EST 2012
          root@snapshots-8_1-amd64.builders.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8
        Panic String: ffs_valloc: dup alloc
        Dump Parity: 785600318
        Bounds: 0
        Dump Status: good
      
      Filename: /var/crash/textdump.tar.0
      ddb.txt���������������������������������������������������������������������������������������������0600����0�������0�������140000������12101004065�  7057� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������db:0:kdb.enter.default>  run lockinfo
      db:1:lockinfo> show locks
      No such command
      db:1:locks>  show alllocks
      No such command
      db:1:alllocks>  show lockedvnods
      Locked vnodes
      db:0:kdb.enter.default>  show pcpu
      cpuid        = 1
      dynamic pcpu    = 0xffffff807ef07080
      curthread    = 0xffffff009cef0000: pid 18121 "initial thread"
      curpcb       = 0xffffff803cf48d40
      fpcurthread  = 0xffffff009cef0000: pid 18121 "initial thread"
      idlethread   = 0xffffff000235a7c0: pid 11 "idle: cpu1"
      curpmap         = 0
      tssp            = 0xffffffff811ddde8
      commontssp      = 0xffffffff811ddde8
      rsp0            = 0xffffff803cf48d40
      gs32p           = 0xffffffff811dcc20
      ldt             = 0xffffffff811dcc60
      tss             = 0xffffffff811dcc50
      db:0:kdb.enter.default>  bt
      Tracing pid 18121 tid 64176 td 0xffffff009cef0000
      kdb_enter() at kdb_enter+0x3d
      panic() at panic+0x17b
      ffs_valloc() at ffs_valloc+0x5bc
      ufs_makeinode() at ufs_makeinode+0x74
      VOP_CREATE_APV() at VOP_CREATE_APV+0x31
      vn_open_cred() at vn_open_cred+0x485
      kern_openat() at kern_openat+0x179
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (5, FreeBSD ELF64, open), rip = 0x800f3b24c, rsp = 0x7fffffffe7f8, rbp = 0x8012041c0 ---
      db:0:kdb.enter.default>  ps
        pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
      10941 22043    24     0  S       nanslp   0xffffffff8115cc28 sleep
       8307 61939    24     0  S       nanslp   0xffffffff8115cc28 sleep
      18268 18121 50311    62  S       piperd   0xffffff000482b5b0 unlinkd
      18121 50311 50311    62  R       CPU 1                       initial thread
      49493 36156 35959     0  S       accept   0xffffff013527bb06 initial thread
      35966     1 35959     0  S       bpf      0xffffff007df05000 arpwatch
      29100     1 29100     0  Ss+     ttyin    0xffffff0002f848a8 getty
      28775     1 28775     0  Ss+     ttyin    0xffffff0002f8a0a8 getty
      25538 25471 25471     0  S       nanslp   0xffffffff8115cc28 minicron
      25471     1 25471     0  Ss      wait     0xffffff009cef2460 minicron
      25269 24904 24904     0  S       nanslp   0xffffffff8115cc28 minicron
      24904     1 24904     0  Ss      wait     0xffffff000498d000 minicron
      24627 24233 24233     0  S       nanslp   0xffffffff8115cc28 minicron
      24233     1 24233     0  Ss      wait     0xffffff0004822460 minicron
      22043     1    24     0  S+      wait     0xffffff009cef38c0 sh
      50311     1 50311     0  Ss      wait     0xffffff0004996000 squid
        773 31001 31001     0  S       piperd   0xffffff0004db0888 rrdtool
      34745 33809 33253   106  S       piperd   0xffffff00040c7000 havp
      34495 33407 33253   106  S       piperd   0xffffff0004d3d5b0 havp
      34133 33492 33253   106  S       piperd   0xffffff0004d3db60 havp
      33809 33253 33253   106  S       accept   0xffffff0004aef066 havp
      33492 33253 33253   106  S       accept   0xffffff0004aef066 havp
      33407 33253 33253   106  S       accept   0xffffff0004aef066 havp
      33253     1 33253   106  Ss      nanslp   0xffffffff8115cc28 havp
      31591     1 31591     0  Ss      (threaded)                  clamd
       64161                   S       select   0xffffff00040729c0 clamd
       64110                   S       select   0xffffff000406f2c0 clamd
      15607     0     0     0  SL      mdwait   0xffffff0004beb000 [md10]
       7484     1  7484     0  Ss      nanslp   0xffffffff8115cc28 cron
       7196     1  7196     0  Ss      select   0xffffff00049fa040 powerd
       6664     1  6664     0  Ss      select   0xffffff0004959840 miniupnpd
      61939     1    24     0  S+      wait     0xffffff00041d7460 sh
      51874 19631 51874     0  Ss      (threaded)                  sshlockout_pf
       64146                   S       nanslp   0xffffffff8115cc28 sshlockout_pf
       64132                   S       piperd   0xffffff00041d42d8 initial thread
      47379     1 47379     0  Ss      select   0xffffff000403d8c0 ntpd
      45642     1 45599 65534  S       select   0xffffff000403d740 dnsmasq
      44471     1 44471     0  Ss      kqread   0xffffff0135cb0d00 dhcpleases
      43519     1 43519  1002  Ss      select   0xffffff000403db40 dhcpd
      36832 36156 35959     0  S       accept   0xffffff00043c7066 initial thread
      36156     1 35959     0  S       kqread   0xffffff000442a900 lighttpd
      31001     1 31001     0  Ss      select   0xffffff000403cb40 apinger
      23017     1 23017     0  Ss      select   0xffffff000406e440 inetd
      19631     1 19631     0  Ss      select   0xffffff000406f540 syslogd
      15384     1 15384    65  Ss      select   0xffffff000403c6c0 dhclient
       8640     1  8640     0  Ss      select   0xffffff000403c540 dhclient
       7942     1  7942     0  Ss      select   0xffffff00040706c0 sshd
        259     1   259     0  Ss      select   0xffffff000403b0c0 devd
        249   247   247     0  S       kqread   0xffffff0004028900 check_reload_status
        247     1   247     0  Ss      kqread   0xffffff0004028b00 check_reload_status
         39     0     0     0  SL      mdwait   0xffffff000400e800 [md0]
         23     0     0     0  SL      sdflush  0xffffffff811a3738 [softdepflush]
         22     0     0     0  SL      vlruwt   0xffffff0004040000 [vnlru]
         21     0     0     0  SL      syncer   0xffffffff81180e40 [syncer]
         20     0     0     0  SL      psleep   0xffffffff81180968 [bufdaemon]
         19     0     0     0  RL      CPU 0                       [idlepoll]
         18     0     0     0  SL      pgzero   0xffffffff811a51cc [pagezero]
         17     0     0     0  SL      psleep   0xffffffff811a4568 [vmdaemon]
         16     0     0     0  SL      psleep   0xffffffff811a452c [pagedaemon]
          9     0     0     0  SL      ccb_scan 0xffffffff811229e0 [xpt_thrd]
          8     0     0     0  SL      pftm     0xffffffff80204dd0 [pfpurge]
          7     0     0     0  SL      waiting_ 0xffffffff8118ce00 [sctp_iterator]
         15     0     0     0  SL      (threaded)                  usb
       64068                   D       -        0xffffff8000306dd0 [usbus6]
       64067                   D       -        0xffffff8000306d78 [usbus6]
       64066                   D       -        0xffffff8000306d20 [usbus6]
       64065                   D       -        0xffffff8000306cc8 [usbus6]
       64064                   D       -        0xffffff80002fdef0 [usbus5]
       64063                   D       -        0xffffff80002fde98 [usbus5]
       64062                   D       -        0xffffff80002fde40 [usbus5]
       64061                   D       -        0xffffff80002fdde8 [usbus5]
       64060                   D       -        0xffffff80002f4ef0 [usbus4]
       64059                   D       -        0xffffff80002f4e98 [usbus4]
       64058                   D       -        0xffffff80002f4e40 [usbus4]
       64057                   D       -        0xffffff80002f4de8 [usbus4]
       64055                   D       -        0xffffff80002ebef0 [usbus3]
       64054                   D       -        0xffffff80002ebe98 [usbus3]
       64053                   D       -        0xffffff80002ebe40 [usbus3]
       64052                   D       -        0xffffff80002ebde8 [usbus3]
       64043                   D       -        0xffffff80002c4dd0 [usbus2]
       64042                   D       -        0xffffff80002c4d78 [usbus2]
       64041                   D       -        0xffffff80002c4d20 [usbus2]
       64040                   D       -        0xffffff80002c4cc8 [usbus2]
       64038                   D       -        0xffffff80002bbef0 [usbus1]
       64037                   D       -        0xffffff80002bbe98 [usbus1]
       64036                   D       -        0xffffff80002bbe40 [usbus1]
       64035                   D       -        0xffffff80002bbde8 [usbus1]
       64033                   D       -        0xffffff80002b2ef0 [usbus0]
       64032                   D       -        0xffffff80002b2e98 [usbus0]
       64031                   D       -        0xffffff80002b2e40 [usbus0]
       64030                   D       -        0xffffff80002b2de8 [usbus0]
         14     0     0     0  SL      -        0xffffffff8115c904 [yarrow]
          6     0     0     0  SL      crypto_r 0xffffffff811a2660 [crypto returns]
          5     0     0     0  SL      crypto_w 0xffffffff811a2620 [crypto]
          4     0     0     0  RL                                  [g_down]
          3     0     0     0  SL      -        0xffffffff811586e0 [g_up]
          2     0     0     0  SL      -        0xffffffff811586d0 [g_event]
         13     0     0     0  SL      (threaded)                  ng_queue
       64011                   D       sleep    0xffffffff810d19d0 [ng_queue1]
       64010                   D       sleep    0xffffffff810d19d0 [ng_queue0]
         12     0     0     0  WL      (threaded)                  intr
       64083                   I                                   [swi1: netisr 1]
       64071                   I                                   [irq1: atkbd0]
       64070                   I                                   [irq15: ata1]
       64069                   I                                   [irq14: ata0]
       64056                   I                                   [irq20: uhci3]
       64051                   I                                   [irq23: uhci2 ehci1]
       64050                   I                                   [irq259: em0:link]
       64048                   I                                   [irq258: em0:tx 0]
       64046                   I                                   [irq257: em0:rx 0]
       64045                   I                                   [irq17: atapci0]
       64039                   I                                   [irq18: ehci0+]
       64034                   I                                   [irq21: uhci1 uhci4]
       64029                   I                                   [irq22: uhci0]
       64028                   I                                   [irq9: acpi0]
       64026                   I                                   [swi5: +]
       64024                   I                                   [swi2: cambio]
       64020                   I                                   [swi6: task queue]
       64019                   I                                   [swi6: Giant taskq]
       64009                   I                                   [swi3: vm]
       64008                   I                                   [swi4: clock]
       64007                   I                                   [swi4: clock]
       64006                   I                                   [swi1: netisr 0]
         11     0     0     0  RL      (threaded)                  idle
       64005                   CanRun                              [idle: cpu0]
       64004                   CanRun                              [idle: cpu1]
          1     0     1     0  SLs     wait     0xffffff00023578c0 [init]
         10     0     0     0  SL      audit_wo 0xffffffff811a2b90 [audit]
          0     0     0     0  SLs     (threaded)                  kernel
       64049                   D       -        0xffffff0002831900 [em0 txq]
       64047                   D       -        0xffffff0002832000 [em0 rxq]
       64044                   D       -        0xffffff0002703780 [age0 taskq]
       64027                   D       -        0xffffff000249fd80 [thread]
       64025                   D       -        0xffffff0002502080 [kqueue taskq]
       64023                   D       -        0xffffff0002502380 [acpi_task_2]
       64022                   D       -        0xffffff0002502380 [acpi_task_1]
       64021                   D       -        0xffffff0002502380 [acpi_task_0]
       64012                   D       -        0xffffff0002367a80 [firmware taskq]
       64001                   D       sched    0xffffffff81158840 [swapper]
      db:0:kdb.enter.default>  alltrace
      
      Tracing command sleep pid 10941 tid 64163 td 0xffffff00040d23e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
      _sleep() at _sleep+0x1a3
      kern_nanosleep() at kern_nanosleep+0x118
      nanosleep() at nanosleep+0x6e
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80071e66c, rsp = 0x7fffffffed28, rbp = 0x7fffffffed80 ---
      
      Tracing command sleep pid 8307 tid 64184 td 0xffffff009cef8ba0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
      _sleep() at _sleep+0x1a3
      kern_nanosleep() at kern_nanosleep+0x118
      nanosleep() at nanosleep+0x6e
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80071e66c, rsp = 0x7fffffffed28, rbp = 0x7fffffffed80 ---
      
      Tracing command unlinkd pid 18268 tid 64170 td 0xffffff009ce287c0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      pipe_read() at pipe_read+0x432
      dofileread() at dofileread+0xa1
      kern_readv() at kern_readv+0x60
      read() at read+0x55
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (3, FreeBSD ELF64, read), rip = 0x8007394ac, rsp = 0x7fffffffe7f8, rbp = 0x7fffffffeca0 ---
      
      Tracing command squid pid 18121 tid 64176 td 0xffffff009cef0000
      kdb_enter() at kdb_enter+0x3d
      panic() at panic+0x17b
      ffs_valloc() at ffs_valloc+0x5bc
      ufs_makeinode() at ufs_makeinode+0x74
      VOP_CREATE_APV() at VOP_CREATE_APV+0x31
      vn_open_cred() at vn_open_cred+0x485
      kern_openat() at kern_openat+0x179
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (5, FreeBSD ELF64, open), rip = 0x800f3b24c, rsp = 0x7fffffffe7f8, rbp = 0x8012041c0 ---
      
      Tracing command php pid 49493 tid 64118 td 0xffffff00041133e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      kern_accept() at kern_accept+0x1cc
      accept() at accept+0x75
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (30, FreeBSD ELF64, accept), rip = 0x8012d357c, rsp = 0x7ffffffec7e8, rbp = 0 ---
      
      Tracing command arpwatch pid 35966 tid 64098 td 0xffffff00040d87c0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
      _sleep() at _sleep+0x1a3
      bpfread() at bpfread+0x28b
      devfs_read_f() at devfs_read_f+0x86
      dofileread() at dofileread+0xa1
      kern_readv() at kern_readv+0x60
      read() at read+0x55
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (3, FreeBSD ELF64, read), rip = 0x8008704ac, rsp = 0x7fffffffe968, rbp = 0x800a04400 ---
      
      Tracing command getty pid 29100 tid 64090 td 0xffffff00040493e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _cv_wait_sig() at _cv_wait_sig+0x120
      tty_wait() at tty_wait+0x25
      ttydisc_read() at ttydisc_read+0x2b1
      ttydev_read() at ttydev_read+0x10e
      devfs_read_f() at devfs_read_f+0x86
      dofileread() at dofileread+0xa1
      kern_readv() at kern_readv+0x60
      read() at read+0x55
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (3, FreeBSD ELF64, read), rip = 0x80084e4ac, rsp = 0x7fffffffecc8, rbp = 0 ---
      
      Tracing command getty pid 28775 tid 64084 td 0xffffff000404c3e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _cv_wait_sig() at _cv_wait_sig+0x120
      tty_wait() at tty_wait+0x25
      ttydisc_read() at ttydisc_read+0x2b1
      ttydev_read() at ttydev_read+0x10e
      devfs_read_f() at devfs_read_f+0x86
      dofileread() at dofileread+0xa1
      kern_readv() at kern_readv+0x60
      read() at read+0x55
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (3, FreeBSD ELF64, read), rip = 0x80084e4ac, rsp = 0x7fffffffecc8, rbp = 0 ---
      
      Tracing command minicron pid 25538 tid 64099 td 0xffffff00040d83e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
      _sleep() at _sleep+0x1a3
      kern_nanosleep() at kern_nanosleep+0x118
      nanosleep() at nanosleep+0x6e
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80071f66c, rsp = 0x7fffffffec18, rbp = 0x800a0c040 ---
      
      Tracing command minicron pid 25471 tid 64173 td 0xffffff009cef0ba0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      kern_wait() at kern_wait+0x77e
      wait4() at wait4+0x35
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8006b2f8c, rsp = 0x7fffffffec48, rbp = 0x4 ---
      
      Tracing command minicron pid 25269 tid 64109 td 0xffffff00040e47c0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
      _sleep() at _sleep+0x1a3
      kern_nanosleep() at kern_nanosleep+0x118
      nanosleep() at nanosleep+0x6e
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80071f66c, rsp = 0x7fffffffec28, rbp = 0x800a0c040 ---
      
      Tracing command minicron pid 24904 tid 64142 td 0xffffff0004ac47c0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      kern_wait() at kern_wait+0x77e
      wait4() at wait4+0x35
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8006b2f8c, rsp = 0x7fffffffec58, rbp = 0x4 ---
      
      Tracing command minicron pid 24627 tid 64111 td 0xffffff00040e4000
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
      _sleep() at _sleep+0x1a3
      kern_nanosleep() at kern_nanosleep+0x118
      nanosleep() at nanosleep+0x6e
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80071f66c, rsp = 0x7fffffffec28, rbp = 0x800a0c040 ---
      
      Tracing command minicron pid 24233 tid 64131 td 0xffffff000482e3e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      kern_wait() at kern_wait+0x77e
      wait4() at wait4+0x35
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8006b2f8c, rsp = 0x7fffffffec58, rbp = 0x4 ---
      
      Tracing command sh pid 22043 tid 64169 td 0xffffff009ce28ba0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      kern_wait() at kern_wait+0x77e
      wait4() at wait4+0x35
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800934f8c, rsp = 0x7fffffffe888, rbp = 0x18 ---
      
      Tracing command squid pid 50311 tid 64145 td 0xffffff00049d4ba0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      kern_wait() at kern_wait+0x77e
      wait4() at wait4+0x35
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800ec2f8c, rsp = 0x7fffffffeb48, rbp = 0x4c81be ---
      
      Tracing command rrdtool pid 773 tid 64147 td 0xffffff00049d43e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      pipe_read() at pipe_read+0x432
      dofileread() at dofileread+0xa1
      kern_readv() at kern_readv+0x60
      read() at read+0x55
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (3, FreeBSD ELF64, read), rip = 0x800f8b4ac, rsp = 0x7fffffffeb88, rbp = 0x801204000 ---
      
      Tracing command havp pid 34745 tid 64116 td 0xffffff0004113ba0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      pipe_read() at pipe_read+0x432
      dofileread() at dofileread+0xa1
      kern_readv() at kern_readv+0x60
      read() at read+0x55
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (3, FreeBSD ELF64, read), rip = 0x8017ff4ac, rsp = 0x7fffffffde58, rbp = 0x7fffffffe530 ---
      
      Tracing command havp pid 34495 tid 64117 td 0xffffff00041137c0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      pipe_read() at pipe_read+0x432
      dofileread() at dofileread+0xa1
      kern_readv() at kern_readv+0x60
      read() at read+0x55
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (3, FreeBSD ELF64, read), rip = 0x8017ff4ac, rsp = 0x7fffffffde58, rbp = 0x7fffffffe530 ---
      
      Tracing command havp pid 34133 tid 64115 td 0xffffff00041bf000
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      pipe_read() at pipe_read+0x432
      dofileread() at dofileread+0xa1
      kern_readv() at kern_readv+0x60
      read() at read+0x55
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (3, FreeBSD ELF64, read), rip = 0x8017ff4ac, rsp = 0x7fffffffde58, rbp = 0x7fffffffe530 ---
      
      Tracing command havp pid 33809 tid 64086 td 0xffffff00040cc3e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      kern_accept() at kern_accept+0x1cc
      accept() at accept+0x75
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (30, FreeBSD ELF64, accept), rip = 0x80179b57c, rsp = 0x7fffffffdd28, rbp = 0x7fffffffe030 ---
      
      Tracing command havp pid 33492 tid 64103 td 0xffffff0002f23ba0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      kern_accept() at kern_accept+0x1cc
      accept() at accept+0x75
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (30, FreeBSD ELF64, accept), rip = 0x80179b57c, rsp = 0x7fffffffdd28, rbp = 0x7fffffffe030 ---
      
      Tracing command havp pid 33407 tid 64156 td 0xffffff0004c9d000
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      kern_accept() at kern_accept+0x1cc
      accept() at accept+0x75
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (30, FreeBSD ELF64, accept), rip = 0x80179b57c, rsp = 0x7fffffffdd28, rbp = 0x7fffffffe030 ---
      
      Tracing command havp pid 33253 tid 64127 td 0xffffff000482f3e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
      _sleep() at _sleep+0x1a3
      kern_nanosleep() at kern_nanosleep+0x118
      nanosleep() at nanosleep+0x6e
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x8017e466c, rsp = 0x7fffffffdfa8, rbp = 0x5103e326 ---
      
      Tracing command clamd pid 31591 tid 64161 td 0xffffff00040d2ba0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _cv_wait_sig() at _cv_wait_sig+0x120
      seltdwait() at seltdwait+0xf6
      poll() at poll+0x2f8
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (209, FreeBSD ELF64, poll), rip = 0x801aa684c, rsp = 0x7fffffbfe9d8, rbp = 0xffffffff ---
      
      Tracing command clamd pid 31591 tid 64110 td 0xffffff00040e43e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
      _cv_timedwait_sig() at _cv_timedwait_sig+0x129
      seltdwait() at seltdwait+0x8d
      poll() at poll+0x2f8
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (209, FreeBSD ELF64, poll), rip = 0x801aa684c, rsp = 0x7fffffffe3f8, rbp = 0xe10 ---
      
      Tracing command md10 pid 15607 tid 64158 td 0xffffff0004c9c7c0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_wait() at sleepq_wait+0x42
      _sleep() at _sleep+0x31c
      md_kthread() at md_kthread+0x1e0
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803ceeed30, rbp = 0 ---
      
      Tracing command cron pid 7484 tid 64141 td 0xffffff0004ac4ba0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
      _sleep() at _sleep+0x1a3
      kern_nanosleep() at kern_nanosleep+0x118
      nanosleep() at nanosleep+0x6e
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093e66c, rsp = 0x7fffffffeb08, rbp = 0x3c ---
      
      Tracing command powerd pid 7196 tid 64139 td 0xffffff00049493e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
      _cv_timedwait_sig() at _cv_timedwait_sig+0x129
      seltdwait() at seltdwait+0x8d
      kern_select() at kern_select+0x64d
      select() at select+0x5d
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (93, FreeBSD ELF64, select), rip = 0x80084b42c, rsp = 0x7fffffffe728, rbp = 0x7d0 ---
      
      Tracing command miniupnpd pid 6664 tid 64113 td 0xffffff00041bf7c0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
      _cv_timedwait_sig() at _cv_timedwait_sig+0x129
      seltdwait() at seltdwait+0x8d
      kern_select() at kern_select+0x64d
      select() at select+0x5d
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (93, FreeBSD ELF64, select), rip = 0x80085442c, rsp = 0x7fffffffea18, rbp = 0xb ---
      
      Tracing command sh pid 61939 tid 64114 td 0xffffff00041bf3e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      kern_wait() at kern_wait+0x77e
      wait4() at wait4+0x35
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800934f8c, rsp = 0x7fffffffe8a8, rbp = 0x18 ---
      
      Tracing command sshlockout_pf pid 51874 tid 64146 td 0xffffff00049d47c0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
      _sleep() at _sleep+0x1a3
      kern_nanosleep() at kern_nanosleep+0x118
      nanosleep() at nanosleep+0x6e
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80083866c, rsp = 0x7fffffbfef58, rbp = 0 ---
      
      Tracing command sshlockout_pf pid 51874 tid 64132 td 0xffffff000482e000
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      pipe_read() at pipe_read+0x432
      dofileread() at dofileread+0xa1
      kern_readv() at kern_readv+0x60
      read() at read+0x55
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (3, FreeBSD ELF64, read), rip = 0x8008534ac, rsp = 0x7fffffffe808, rbp = 0x7fffffffecd0 ---
      
      Tracing command ntpd pid 47379 tid 64137 td 0xffffff0004158ba0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _cv_wait_sig() at _cv_wait_sig+0x120
      seltdwait() at seltdwait+0xf6
      kern_select() at kern_select+0x64d
      select() at select+0x5d
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (93, FreeBSD ELF64, select), rip = 0x800fa242c, rsp = 0x7fffffffeb08, rbp = 0x7fffffffeba0 ---
      
      Tracing command dnsmasq pid 45642 tid 64112 td 0xffffff00041bfba0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _cv_wait_sig() at _cv_wait_sig+0x120
      seltdwait() at seltdwait+0xf6
      kern_select() at kern_select+0x64d
      select() at select+0x5d
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (93, FreeBSD ELF64, select), rip = 0x80089d42c, rsp = 0x7fffffffe9a8, rbp = 0x7fffffffea70 ---
      
      Tracing command dhcpleases pid 44471 tid 64124 td 0xffffff0004830000
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      kern_kevent() at kern_kevent+0x365
      kevent() at kevent+0x90
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (363, FreeBSD ELF64, kevent), rip = 0x80072169c, rsp = 0x7fffffffeb88, rbp = 0x3 ---
      
      Tracing command dhcpd pid 43519 tid 64093 td 0xffffff00040d97c0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
      _cv_timedwait_sig() at _cv_timedwait_sig+0x129
      seltdwait() at seltdwait+0x8d
      kern_select() at kern_select+0x64d
      select() at select+0x5d
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (93, FreeBSD ELF64, select), rip = 0x8009de42c, rsp = 0x7fffffffea98, rbp = 0x7fffffffeae8 ---
      
      Tracing command php pid 36832 tid 64128 td 0xffffff000482f000
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      kern_accept() at kern_accept+0x1cc
      accept() at accept+0x75
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (30, FreeBSD ELF64, accept), rip = 0x8012d357c, rsp = 0x7ffffffec7e8, rbp = 0 ---
      
      Tracing command lighttpd pid 36156 tid 64123 td 0xffffff00048303e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
      _sleep() at _sleep+0x1a3
      kern_kevent() at kern_kevent+0x365
      kevent() at kevent+0x90
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800c8969c, rsp = 0x7fffffffe998, rbp = 0x800e45800 ---
      
      Tracing command apinger pid 31001 tid 64126 td 0xffffff000482f7c0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
      _cv_timedwait_sig() at _cv_timedwait_sig+0x129
      seltdwait() at seltdwait+0x8d
      poll() at poll+0x2f8
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006f084c, rsp = 0x7fffffffcb08, rbp = 0x1 ---
      
      Tracing command inetd pid 23017 tid 64100 td 0xffffff00040487c0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _cv_wait_sig() at _cv_wait_sig+0x120
      seltdwait() at seltdwait+0xf6
      kern_select() at kern_select+0x64d
      select() at select+0x5d
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (93, FreeBSD ELF64, select), rip = 0x800a6342c, rsp = 0x7fffffffdd58, rbp = 0 ---
      
      Tracing command syslogd pid 19631 tid 64104 td 0xffffff0002f237c0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _cv_wait_sig() at _cv_wait_sig+0x120
      seltdwait() at seltdwait+0xf6
      kern_select() at kern_select+0x64d
      select() at select+0x5d
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (93, FreeBSD ELF64, select), rip = 0x800dc042c, rsp = 0x7fffffffe068, rbp = 0xf ---
      
      Tracing command dhclient pid 15384 tid 64108 td 0xffffff00040e4ba0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
      _cv_timedwait_sig() at _cv_timedwait_sig+0x129
      seltdwait() at seltdwait+0x8d
      poll() at poll+0x2f8
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006f684c, rsp = 0x7fffffffec98, rbp = 0x36ee80 ---
      
      Tracing command dhclient pid 8640 tid 64088 td 0xffffff0004049ba0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _cv_wait_sig() at _cv_wait_sig+0x120
      seltdwait() at seltdwait+0xf6
      poll() at poll+0x2f8
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006f684c, rsp = 0x7fffffffec98, rbp = 0xa ---
      
      Tracing command sshd pid 7942 tid 64107 td 0xffffff0004158000
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _cv_wait_sig() at _cv_wait_sig+0x120
      seltdwait() at seltdwait+0xf6
      kern_select() at kern_select+0x64d
      select() at select+0x5d
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (93, FreeBSD ELF64, select), rip = 0x80109f42c, rsp = 0x7fffffffddb8, rbp = 0x2 ---
      
      Tracing command devd pid 259 tid 64120 td 0xffffff00040d8000
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _cv_wait_sig() at _cv_wait_sig+0x120
      seltdwait() at seltdwait+0xf6
      kern_select() at kern_select+0x64d
      select() at select+0x5d
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (93, FreeBSD ELF64, select), rip = 0x440f0c, rsp = 0x7fffffffe878, rbp = 0x7fffffffe890 ---
      
      Tracing command check_reload_status pid 249 tid 64092 td 0xffffff0004049000
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      kern_kevent() at kern_kevent+0x365
      kevent() at kevent+0x90
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (363, FreeBSD ELF64, kevent), rip = 0x80083b69c, rsp = 0x7fffffffeb08, rbp = 0x7fffffffec00 ---
      
      Tracing command check_reload_status pid 247 tid 64094 td 0xffffff00040d93e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_catch_signals() at sleepq_catch_signals+0x31f
      sleepq_wait_sig() at sleepq_wait_sig+0xc
      _sleep() at _sleep+0x26b
      kern_kevent() at kern_kevent+0x365
      kevent() at kevent+0x90
      syscall() at syscall+0x1e7
      Xfast_syscall() at Xfast_syscall+0xe1
      --- syscall (363, FreeBSD ELF64, kevent), rip = 0x80083b69c, rsp = 0x7fffffffea18, rbp = 0x800a1a000 ---
      
      Tracing command md0 pid 39 tid 64087 td 0xffffff00040cc000
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_wait() at sleepq_wait+0x42
      _sleep() at _sleep+0x31c
      md_kthread() at md_kthread+0x1e0
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803cd8bd30, rbp = 0 ---
      
      Tracing command softdepflush pid 23 tid 64082 td 0xffffff0002f20000
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_timedwait() at sleepq_timedwait+0x42
      _sleep() at _sleep+0x308
      softdep_flush() at softdep_flush+0x34d
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803aa6fd30, rbp = 0 ---
      
      Tracing command vnlru pid 22 tid 64081 td 0xffffff0002f203e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_timedwait() at sleepq_timedwait+0x42
      _sleep() at _sleep+0x308
      vnlru_proc() at vnlru_proc+0x5fd
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803aa6ad30, rbp = 0 ---
      
      Tracing command syncer pid 21 tid 64080 td 0xffffff0002f207c0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_timedwait() at sleepq_timedwait+0x42
      _cv_timedwait() at _cv_timedwait+0x129
      sched_sync() at sched_sync+0x4e6
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803aa65d30, rbp = 0 ---
      
      Tracing command bufdaemon pid 20 tid 64079 td 0xffffff0002f20ba0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_timedwait() at sleepq_timedwait+0x42
      _sleep() at _sleep+0x308
      buf_daemon() at buf_daemon+0x192
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803aa60d30, rbp = 0 ---
      
      Tracing command idlepoll pid 19 tid 64078 td 0xffffff0002f21000
      cpustop_handler() at cpustop_handler+0x40
      ipi_nmi_handler() at ipi_nmi_handler+0x30
      trap() at trap+0x175
      nmi_calltrap() at nmi_calltrap+0x8
      --- trap 0x13, rip = 0xffffffff80a4aa36, rsp = 0xffffffff811e5b40, rbp = 0xffffff803aa5bb90 ---
      rdtsc() at rdtsc+0x6
      mi_switch() at mi_switch+0x4d
      poll_idle() at poll_idle+0x15a
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803aa5bd30, rbp = 0 ---
      
      Tracing command pagezero pid 18 tid 64077 td 0xffffff0002f213e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_timedwait() at sleepq_timedwait+0x42
      _sleep() at _sleep+0x308
      vm_pagezero() at vm_pagezero+0x83
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803aa56d30, rbp = 0 ---
      
      Tracing command vmdaemon pid 17 tid 64076 td 0xffffff0002f217c0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_wait() at sleepq_wait+0x42
      _sleep() at _sleep+0x31c
      vm_daemon() at vm_daemon+0x58
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803aa51d30, rbp = 0 ---
      
      Tracing command pagedaemon pid 16 tid 64075 td 0xffffff0002f21ba0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_timedwait() at sleepq_timedwait+0x42
      _sleep() at _sleep+0x308
      vm_pageout() at vm_pageout+0x877
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803aa4cd30, rbp = 0 ---
      
      Tracing command xpt_thrd pid 9 tid 64074 td 0xffffff0002f22000
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_wait() at sleepq_wait+0x42
      _sleep() at _sleep+0x31c
      xpt_scanner_thread() at xpt_scanner_thread+0xff
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803aa47d30, rbp = 0 ---
      
      Tracing command pfpurge pid 8 tid 64073 td 0xffffff0002f223e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_timedwait() at sleepq_timedwait+0x42
      _sleep() at _sleep+0x308
      pf_purge_thread() at pf_purge_thread+0x31
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803aa42d30, rbp = 0 ---
      
      Tracing command sctp_iterator pid 7 tid 64072 td 0xffffff0002f227c0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_wait() at sleepq_wait+0x42
      _sleep() at _sleep+0x31c
      sctp_iterator_thread() at sctp_iterator_thread+0x54
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803aa3dd30, rbp = 0 ---
      
      Tracing command usb pid 15 tid 64068 td 0xffffff0002c403e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_wait() at sleepq_wait+0x42
      _cv_wait() at _cv_wait+0x111
      usb_process() at usb_process+0x18b
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803a99dd30, rbp = 0 ---
      
      Tracing command usb pid 15 tid 64067 td 0xffffff0002c407c0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_wait() at sleepq_wait+0x42
      _cv_wait() at _cv_wait+0x111
      usb_process() at usb_process+0x18b
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803a998d30, rbp = 0 ---
      
      Tracing command usb pid 15 tid 64066 td 0xffffff0002c40ba0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_wait() at sleepq_wait+0x42
      _cv_wait() at _cv_wait+0x111
      usb_process() at usb_process+0x18b
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803a993d30, rbp = 0 ---
      
      Tracing command usb pid 15 tid 64065 td 0xffffff0002c41000
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_wait() at sleepq_wait+0x42
      _cv_wait() at _cv_wait+0x111
      usb_process() at usb_process+0x18b
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803a98ed30, rbp = 0 ---
      
      Tracing command usb pid 15 tid 64064 td 0xffffff0002c413e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_wait() at sleepq_wait+0x42
      _cv_wait() at _cv_wait+0x111
      usb_process() at usb_process+0x18b
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803a807d30, rbp = 0 ---
      
      Tracing command usb pid 15 tid 64063 td 0xffffff0002c417c0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_wait() at sleepq_wait+0x42
      _cv_wait() at _cv_wait+0x111
      usb_process() at usb_process+0x18b
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803a802d30, rbp = 0 ---
      
      Tracing command usb pid 15 tid 64062 td 0xffffff0002c41ba0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_wait() at sleepq_wait+0x42
      _cv_wait() at _cv_wait+0x111
      usb_process() at usb_process+0x18b
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803a7fdd30, rbp = 0 ---
      
      Tracing command usb pid 15 tid 64061 td 0xffffff0002c43000
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_wait() at sleepq_wait+0x42
      _cv_wait() at _cv_wait+0x111
      usb_process() at usb_process+0x18b
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803a7f8d30, rbp = 0 ---
      
      Tracing command usb pid 15 tid 64060 td 0xffffff0002c433e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_wait() at sleepq_wait+0x42
      _cv_wait() at _cv_wait+0x111
      usb_process() at usb_process+0x18b
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803a66dd30, rbp = 0 ---
      
      Tracing command usb pid 15 tid 64059 td 0xffffff0002c437c0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_wait() at sleepq_wait+0x42
      _cv_wait() at _cv_wait+0x111
      usb_process() at usb_process+0x18b
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803a668d30, rbp = 0 ---
      
      Tracing command usb pid 15 tid 64058 td 0xffffff0002c43ba0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_wait() at sleepq_wait+0x42
      _cv_wait() at _cv_wait+0x111
      usb_process() at usb_process+0x18b
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803a663d30, rbp = 0 ---
      
      Tracing command usb pid 15 tid 64057 td 0xffffff00027083e0
      sched_switch() at sched_switch+0x100
      mi_switch() at mi_switch+0x16f
      sleepq_wait() at sleepq_wait+0x42
      _cv_wait() at _cv_wait+0x111
      usb_process() at usb_process+0x18b
      fork_exit() at fork_exit+0x118
      fork_trampoline() at fork_trampoline+0xe
      --- trap 0, rip = 0, rsp = 0xffffff803a65ed30, rbp = 0 [/thread]
      
      1 Reply Last reply Reply Quote 0
      • Cry HavokC
        Cry Havok
        last edited by

        You had no problems before? It sounds like a hardware fault.

        1 Reply Last reply Reply Quote 0
        • jimpJ
          jimp Rebel Alliance Developer Netgate
          last edited by

          Crashes in the ffs code are usually from a failing disk/media.

          Or one that has been badly corrupted, often from it failing.

          Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

          Need help fast? Netgate Global Support!

          Do not Chat/PM for help!

          1 Reply Last reply Reply Quote 0
          • A
            ajm786
            last edited by

            @jimp:

            Crashes in the ffs code are usually from a failing disk/media.

            Or one that has been badly corrupted, often from it failing.

            This is on a computer that was perfectly stable.

            I assumed it was the onboard Atheros NIC, which I was using for the WAN. Looks like that was correct; I roughed a really old IBM EtherJet card with an Intel chipset and installed it, and it's been running for a week straight without crashing. I've got another Intel PCI 1000 Pro card on order, so that'll take care of it.

            However, I'm getting tons of errors in. Not much on the LAN interface, but tons on the WAN interface.

            WAN LAN
            Packets In 129453493.00 120254773.00
            Packets Out 129384527.00 119747247.00
            Bytes In 107.08 GB 81.70 GB
            Bytes Out 82.50 GB 106.57 GB
            Errors In 27585.00 6.00
            Errors Out 0.00 0.00
            Collisions 0.00 0.00

            Any idea what might be causing these? Again, these are on two Intel NIC cards.

            FYI, the LAN interface is an Intel EXPI9301CTBLK. Right now, the WAN is the EtherJet, but that's being replaced by an Intel PWLA8391GT PRO/1000 GT. Of course, I assumed that since these are straight Intel, I shouldn't have any of these issues.

            Thanks!

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              Loads of errors can be a bad cable or a poorly secured card that's moved in it's slot. I assume you would have tried changing the cables but I mention it in case.

              Steve

              1 Reply Last reply Reply Quote 0
              • A
                ajm786
                last edited by

                @stephenw10:

                Loads of errors can be a bad cable or a poorly secured card that's moved in it's slot. I assume you would have tried changing the cables but I mention it in case.

                Steve

                Thanks for the response. Can't be the cables, because prior to this build, I had pfSense running on a T40 laptop for 6 months straight with NO errors on the WAN or LAN interface, with the same cables.

                1 Reply Last reply Reply Quote 0
                • First post
                  Last post
                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.