PF Sense Crash with 100MB/s WAN throughput



  • I started Downloading GTA V at ~14MB/sec and PF Sense crashed. I don't think the system rebooted. Here's the core dump:

    Crash report begins.  Anonymous machine information:
    
    amd64
    10.1-RELEASE-p9
    FreeBSD 10.1-RELEASE-p9 #0 57b23e7(releng/10.1)-dirty: Mon Apr 13 20:30:25 CDT 2015     root@pfs22-amd64-builder:/usr/obj.amd64/usr/pfSensesrc/src/sys/pfSense_SMP.10
    
    Crash report details:
    
    Filename: /var/crash/bounds
    1
    
    Filename: /var/crash/info.0
    Dump header from device /dev/ada0s1b
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 105984B (0 MB)
      Blocksize: 512
      Dumptime: Sat May  9 09:20:10 2015
      Hostname: fuckmountain.no-ip.biz
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 10.1-RELEASE-p9 #0 57b23e7(releng/10.1)-dirty: Mon Apr 13 20:30:25 CDT 2015
        root@pfs22-amd64-builder:/usr/obj.amd64/usr/pfSensesrc/src/sys/pfSense_SMP.10
      Panic String: 
      Dump Parity: 3387775489
      Bounds: 0
      Dump Status: good
    
    Filename: /var/crash/info.last
    Dump header from device /dev/ada0s1b
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 105984B (0 MB)
      Blocksize: 512
      Dumptime: Sat May  9 09:20:10 2015
      Hostname: fuckmountain.no-ip.biz
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 10.1-RELEASE-p9 #0 57b23e7(releng/10.1)-dirty: Mon Apr 13 20:30:25 CDT 2015
        root@pfs22-amd64-builder:/usr/obj.amd64/usr/pfSensesrc/src/sys/pfSense_SMP.10
      Panic String: 
      Dump Parity: 3387775489
      Bounds: 0
      Dump Status: good
    
    Filename: /var/crash/minfree
    2048
    
    Filename: /var/crash/textdump.tar.0
    ddb.txt06000014000012523413632  7073 ustarrootwheeldb:0:kdb.enter.default>  run lockinfo
    db:1:lockinfo> show locks
    No such command
    db:1:locks>  show alllocks
    No such command
    db:1:alllocks>  show lockedvnods
    Locked vnodes
    db:0:kdb.enter.default>  show pcpu
    cpuid        = 1
    dynamic pcpu = 0xfffffe00998ac280
    curthread    = 0xfffff80003500000: pid 12 "irq27: bge0 bge1"
    curpcb       = 0xfffffe004d117cc0
    fpcurthread  = none
    idlethread   = 0xfffff800032d0920: tid 100004 "idle: cpu1"
    curpmap      = 0xffffffff8217c548
    tssp         = 0xffffffff821974f8
    commontssp   = 0xffffffff821974f8
    rsp0         = 0xfffffe004d117cc0
    gs32p        = 0xffffffff82198f50
    ldt          = 0xffffffff82198f90
    tss          = 0xffffffff82198f80
    db:0:kdb.enter.default>  bt
    Tracing pid 12 tid 100045 td 0xfffff80003500000
    bge_rxeof() at bge_rxeof+0x41e/frame 0xfffffe004d117ae0
    bge_intr() at bge_intr+0x179/frame 0xfffffe004d117b20
    intr_event_execute_handlers() at intr_event_execute_handlers+0xab/frame 0xfffffe004d117b60
    ithread_loop() at ithread_loop+0x96/frame 0xfffffe004d117bb0
    fork_exit() at fork_exit+0x9a/frame 0xfffffe004d117bf0
    fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe004d117bf0
    --- trap 0, rip = 0, rsp = 0xfffffe004d117cb0, rbp = 0 ---
    db:0:kdb.enter.default>  ps
      pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
    52025   250   250     0  S       nanslp   0xffffffff81f683f2 php-fpm
    40803 11207   250     0  S       nanslp   0xffffffff81f683f3 sleep
    47830     1 47830     0  Ss      (threaded)                  snort
    100410                   S       nanslp   0xffffffff81f683f0 snort
    100208                   S       bpf      0xfffff80133cbd400 snort
    32800     1 32800     0  Ss      (threaded)                  ntpd
    100102                   S       select   0xfffff800065c4b40 ntpd
    11207     1   250     0  S       wait     0xfffff8006b5984c0 sh
    30644     1 30338 65534  S       select   0xfffff8006802e2c0 dnsmasq
    26223     1 26223    65  Ss      select   0xfffff80133c419c0 dhclient
    21763     1 21763     0  Ss      select   0xfffff80006511c40 dhclient
    36135     1 36135  1002  Ss      select   0xfffff80133e04140 dhcpd
    34166     1 34166    65  Ss      select   0xfffff800d6cf0ac0 dhclient
    24607     1 24607     0  Ss      select   0xfffff80133a0ac40 dhclient
     2950  2912  2912     0  S       piperd   0xfffff8000653b5d0 rrdtool
     2912     1  2912     0  Ss      select   0xfffff8006b1386c0 apinger
    18688 18676 18688     0  S+      ttyin    0xfffff800050c70a8 sh
    18676 18048 18676     0  S+      wait     0xfffff800064d6980 sh
    18362 77823 18362     0  Ss      (threaded)                  sshlockout_pf
    100174                   S       nanslp   0xffffffff81f683f2 sshlockout_pf
    100075                   S       piperd   0xfffff800065e32e8 sshlockout_pf
    18048     1 18048     0  Ss+     wait     0xfffff800066184c0 login
    11941 11551 11551     0  S       nanslp   0xffffffff81f683f2 minicron
    11551     1 11551     0  Ss      wait     0xfffff80006443000 minicron
    11319 10984 10984     0  S       nanslp   0xffffffff81f683f1 minicron
    10984     1 10984     0  Ss      wait     0xfffff800d6fd0980 minicron
    10812 10357 10357     0  S       nanslp   0xffffffff81f683f3 minicron
    10357     1 10357     0  Ss      wait     0xfffff800684994c0 minicron
    95938 94723   250     0  S       bpf      0xfffff80006f5c800 bandwidthd
    95053 94723   250     0  S       bpf      0xfffff80006f5c200 bandwidthd
    95003 94723   250     0  S       bpf      0xfffff80068066e00 bandwidthd
    94723     1   250     0  S       bpf      0xfffff80006f5c000 bandwidthd
    94044 92206   250     0  S       bpf      0xfffff80006f5c400 bandwidthd
    93233 92206   250     0  S       bpf      0xfffff80006f5c600 bandwidthd
    92617 92206   250     0  S       bpf      0xfffff80133435400 bandwidthd
    92206     1   250     0  S       bpf      0xfffff80006f8ec00 bandwidthd
    78373     1 78373     0  Ss      nanslp   0xffffffff81f683f2 cron
    77823     1 77823     0  Ss      select   0xfffff800d6cf09c0 syslogd
    44882     1 44882     0  Ss      select   0xfffff80006431140 miniupnpd
    34325 34262 34325     0  Ss      (threaded)                  charon
    100128                   S       uwait    0xfffff80006f63700 charon
    100127                   S       uwait    0xfffff80006f63980 charon
    100126                   S       uwait    0xfffff80006434b00 charon
    100125                   S       uwait    0xfffff80006434d00 charon
    100124                   S       uwait    0xfffff80006434e00 charon
    100123                   S       uwait    0xfffff80006434a80 charon
    100122                   S       uwait    0xfffff80006f62100 charon
    100121                   S       uwait    0xfffff80006434980 charon
    100120                   S       uwait    0xfffff800065c4180 charon
    100119                   S       uwait    0xfffff800065c4080 charon
    100118                   S       uwait    0xfffff800065c3f00 charon
    100117                   S       uwait    0xfffff800065c3e00 charon
    100116                   S       select   0xfffff8006802ecc0 charon
    100115                   S       select   0xfffff80006f7a5c0 charon
    100114                   S       accept   0xfffff80068121066 charon
    100113                   S       uwait    0xfffff800065c3a00 charon
    100101                   S       sigwait  0xfffff80006f43000 charon
    34262     1 34262     0  Ss      select   0xfffff80006f63940 starter
    23228     1 23004     0  S       kqread   0xfffff800680b4a00 lighttpd
    16483     1 16483     0  Ss      select   0xfffff80006479a40 inetd
    15294     1 15294     0  Ss      bpf      0xfffff80006440a00 filterlog
      277     1   277     0  Ss      select   0xfffff80006513440 devd
      267   265   265     0  S       kqread   0xfffff800065db700 check_reload_status
      265     1   265     0  Ss      kqread   0xfffff80006658800 check_reload_status
      250     1   250     0  Ss      kqread   0xfffff800032f4000 php-fpm
       63     0     0     0  DL      mdwait   0xfffff800065fe800 [md0]
       20     0     0     0  DL      syncer   0xffffffff81fb9388 [syncer]
       19     0     0     0  DL      vlruwt   0xfffff80006443980 [vnlru]
       18     0     0     0  DL      psleep   0xffffffff81fb8584 [bufdaemon]
       17     0     0     0  DL      pgzero   0xffffffff8210b30c [pagezero]
        9     0     0     0  DL      pollid   0xffffffff81f66d70 [idlepoll]
        8     0     0     0  DL      psleep   0xffffffff8210aa40 [vmdaemon]
        7     0     0     0  DL      psleep   0xffffffff82196804 [pagedaemon]
        6     0     0     0  DL      waiting_ 0xffffffff82187270 [sctp_iterator]
        5     0     0     0  DL      pftm     0xffffffff80d065a0 [pf purge]
       16     0     0     0  DL      (threaded)                  [usb]
    100042                   D       -        0xfffffe00009df460 [usbus1]
    100041                   D       -        0xfffffe00009df408 [usbus1]
    100040                   D       -        0xfffffe00009df3b0 [usbus1]
    100039                   D       -        0xfffffe00009df358 [usbus1]
    100038                   D       -        0xfffffe00009c3460 [usbus0]
    100037                   D       -        0xfffffe00009c3408 [usbus0]
    100036                   D       -        0xfffffe00009c33b0 [usbus0]
    100035                   D       -        0xfffffe00009c3358 [usbus0]
        4     0     0     0  DL      (threaded)                  [cam]
    100056                   D       -        0xffffffff81ea0f40 [scanner]
    100024                   D       -        0xffffffff81ea1100 [doneq0]
        3     0     0     0  DL      crypto_r 0xffffffff82108f10 [crypto returns]
        2     0     0     0  DL      crypto_w 0xffffffff82108db8 [crypto]
       15     0     0     0  DL      -        0xffffffff81ebe600 [rand_harvestq]
       14     0     0     0  DL      (threaded)                  [geom]
    100019                   D       -        0xffffffff8217b9e0 [g_down]
    100018                   D       -        0xffffffff8217b9d8 [g_up]
    100017                   D       -        0xffffffff8217b9d0 [g_event]
       13     0     0     0  DL      (threaded)                  [ng_queue]
    100016                   D       sleep    0xffffffff81e5f3e8 [ng_queue3]
    100015                   D       sleep    0xffffffff81e5f3e8 [ng_queue2]
    100014                   D       sleep    0xffffffff81e5f3e8 [ng_queue1]
    100013                   D       sleep    0xffffffff81e5f3e8 [ng_queue0]
       12     0     0     0  RL      (threaded)                  [intr]
    100066                   I                                   [swi1: netisr 3]
    100065                   I                                   [swi1: netisr 2]
    100064                   I                                   [swi1: netisr 1]
    100054                   I                                   [swi1: pfsync]
    100052                   I                                   [swi1: pf send]
    100049                   I                                   [irq7: ppc0]
    100048                   I                                   [swi0: uart uart]
    100047                   I                                   [irq1: atkbd0]
    100045                   Run     CPU 1                       [irq27: bge0 bge1]
    100044                   I                                   [irq15: ata1]
    100043                   I                                   [irq14: ata0]
    100034                   I                                   [irq19: ohci0 ohci1]
    100029                   I                                   [swi5: fast taskq]
    100027                   I                                   [swi6: Giant taskq]
    100025                   I                                   [swi6: task queue]
    100012                   I                                   [swi3: vm]
    100011                   I                                   [swi4: clock]
    100010                   I                                   [swi4: clock]
    100009                   I                                   [swi4: clock]
    100008                   I                                   [swi4: clock]
    100007                   I                                   [swi1: netisr 0]
       11     0     0     0  RL      (threaded)                  [idle]
    100006                   Run     CPU 3                       [idle: cpu3]
    100005                   Run     CPU 2                       [idle: cpu2]
    100004                   CanRun                              [idle: cpu1]
    100003                   Run     CPU 0                       [idle: cpu0]
        1     0     1     0  SLs     wait     0xfffff800032ce4c0 [init]
       10     0     0     0  DL      audit_wo 0xffffffff8218ddf0 [audit]
        0     0     0     0  DLs     (threaded)                  [kernel]
    100104                   D       -        0xfffff8000648be00 [dummynet]
    100055                   D       -        0xfffff800032f1d00 [CAM taskq]
    100050                   D       -        0xfffff800050a4900 [mca taskq]
    100046                   D       -        0xfffff800034b5300 [em0 taskq]
    100033                   D       -        0xfffff800032c4900 [acpi_task_2]
    100032                   D       -        0xfffff800032c4900 [acpi_task_1]
    100031                   D       -        0xfffff800032c4900 [acpi_task_0]
    100030                   D       -        0xfffff800032c4b00 [kqueue taskq]
    100028                   D       -        0xfffff800032f1100 [thread]
    100026                   D       -        0xfffff800032f1600 [ffs_trim taskq]
    100020                   D       -        0xfffff800032f3500 [firmware taskq]
    100000                   D       swapin   0xffffffff8217bad8 [swapper]
    40896 95003   250     0  Z                                   bandwidthd
    41208 92617   250     0  Z                                   bandwidthd
    13796 94723   250     0  Z                                   bandwidthd
    13545 92206   250     0  Z                                   bandwidthd
    43322 93233   250     0  Z                                   bandwidthd
    43229 95053   250     0  Z                                   bandwidthd
    89047 95938   250     0  Z                                   bandwidthd
    88749 94044   250     0  Z                                   bandwidthd
    db:0:kdb.enter.default>  alltrace
    
    Tracing command php-fpm pid 52025 tid 100152 td 0xfffff8000688b920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d3448d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d344910
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d344990
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d3449c0
    _sleep() at _sleep+0x238/frame 0xfffffe004d344a40
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe004d344aa0
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe004d344ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d344bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d344bf0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x80191540a, rsp = 0x7fffffffab98, rbp = 0x7fffffffabf0 ---
    
    Tracing command sleep pid 40803 tid 100069 td 0xfffff800065c8920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d1ab8d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d1ab910
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d1ab990
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d1ab9c0
    _sleep() at _sleep+0x238/frame 0xfffffe004d1aba40
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe004d1abaa0
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe004d1abae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d1abbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d1abbf0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x80095940a, rsp = 0x7fffffffec88, rbp = 0x7fffffffecc0 ---
    
    Tracing command snort pid 47830 tid 100410 td 0xfffff8006b975000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d3498d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d349910
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d349990
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d3499c0
    _sleep() at _sleep+0x238/frame 0xfffffe004d349a40
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe004d349aa0
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe004d349ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d349bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d349bf0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x80221d40a, rsp = 0x7fffffbfdec8, rbp = 0x7fffffbfdf20 ---
    
    Tracing command snort pid 47830 tid 100208 td 0xfffff80133e4d920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d4367d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d436810
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d436890
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d4368c0
    _sleep() at _sleep+0x238/frame 0xfffffe004d436940
    bpfread() at bpfread+0x1c7/frame 0xfffffe004d436990
    devfs_read_f() at devfs_read_f+0xeb/frame 0xfffffe004d4369f0
    dofileread() at dofileread+0x95/frame 0xfffffe004d436a40
    kern_readv() at kern_readv+0x68/frame 0xfffffe004d436a90
    sys_read() at sys_read+0x63/frame 0xfffffe004d436ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d436bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d436bf0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80223e31a, rsp = 0x7fffffffe718, rbp = 0x7fffffffe750 ---
    
    Tracing command ntpd pid 32800 tid 100102 td 0xfffff800066c3490
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d254720
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d254760
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d2547e0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d254810
    _cv_wait_sig() at _cv_wait_sig+0x16a/frame 0xfffffe004d254860
    seltdwait() at seltdwait+0xae/frame 0xfffffe004d2548a0
    kern_select() at kern_select+0x8fa/frame 0xfffffe004d254aa0
    sys_select() at sys_select+0x54/frame 0xfffffe004d254ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d254bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d254bf0
    --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80146629a, rsp = 0x7fffffffdb78, rbp = 0x7fffffffdbb0 ---
    
    Tracing command sh pid 11207 tid 100258 td 0xfffff80006838490
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d44f6d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d44f710
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d44f790
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d44f7c0
    _sleep() at _sleep+0x27d/frame 0xfffffe004d44f840
    kern_wait6() at kern_wait6+0x5f4/frame 0xfffffe004d44f8f0
    sys_wait4() at sys_wait4+0x72/frame 0xfffffe004d44fae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d44fbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d44fbf0
    --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d66fba, rsp = 0x7fffffffe888, rbp = 0x7fffffffe930 ---
    
    Tracing command dnsmasq pid 30644 tid 100211 td 0xfffff80133e4c920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d445720
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d445760
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d4457e0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d445810
    _cv_wait_sig() at _cv_wait_sig+0x16a/frame 0xfffffe004d445860
    seltdwait() at seltdwait+0xae/frame 0xfffffe004d4458a0
    kern_select() at kern_select+0x8fa/frame 0xfffffe004d445aa0
    sys_select() at sys_select+0x54/frame 0xfffffe004d445ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d445bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d445bf0
    --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8014dc29a, rsp = 0x7fffffffe978, rbp = 0x7fffffffec00 ---
    
    Tracing command dhclient pid 26223 tid 100149 td 0xfffff8000644f920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d3357b0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d3357f0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d335870
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d3358a0
    _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x18b/frame 0xfffffe004d335900
    seltdwait() at seltdwait+0xa4/frame 0xfffffe004d335940
    sys_poll() at sys_poll+0x3a3/frame 0xfffffe004d335ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d335bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d335bf0
    --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x800b38a1a, rsp = 0x7fffffffebd8, rbp = 0x7fffffffec10 ---
    
    Tracing command dhclient pid 21763 tid 100082 td 0xfffff800065c7920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d1ec7c0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d1ec800
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d1ec880
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d1ec8b0
    _cv_wait_sig() at _cv_wait_sig+0x16a/frame 0xfffffe004d1ec900
    seltdwait() at seltdwait+0xae/frame 0xfffffe004d1ec940
    sys_poll() at sys_poll+0x3a3/frame 0xfffffe004d1ecae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d1ecbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d1ecbf0
    --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x800b38a1a, rsp = 0x7fffffffebe8, rbp = 0x7fffffffec10 ---
    
    Tracing command dhcpd pid 36135 tid 100207 td 0xfffff80133e4e000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d431710
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d431750
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d4317d0
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d431800
    _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x18b/frame 0xfffffe004d431860
    seltdwait() at seltdwait+0xa4/frame 0xfffffe004d4318a0
    kern_select() at kern_select+0x8fa/frame 0xfffffe004d431aa0
    sys_select() at sys_select+0x54/frame 0xfffffe004d431ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d431bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d431bf0
    --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800d5329a, rsp = 0x7fffffffea38, rbp = 0x7fffffffea60 ---
    
    Tracing command dhclient pid 34166 tid 100265 td 0xfffff80133e4c490
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d4727b0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d4727f0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d472870
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d4728a0
    _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x18b/frame 0xfffffe004d472900
    seltdwait() at seltdwait+0xa4/frame 0xfffffe004d472940
    sys_poll() at sys_poll+0x3a3/frame 0xfffffe004d472ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d472bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d472bf0
    --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x800b38a1a, rsp = 0x7fffffffebd8, rbp = 0x7fffffffec10 ---
    
    Tracing command dhclient pid 24607 tid 100070 td 0xfffff800065c8490
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d1b07c0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d1b0800
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d1b0880
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d1b08b0
    _cv_wait_sig() at _cv_wait_sig+0x16a/frame 0xfffffe004d1b0900
    seltdwait() at seltdwait+0xae/frame 0xfffffe004d1b0940
    sys_poll() at sys_poll+0x3a3/frame 0xfffffe004d1b0ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d1b0bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d1b0bf0
    --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x800b38a1a, rsp = 0x7fffffffebe8, rbp = 0x7fffffffec10 ---
    
    Tracing command rrdtool pid 2950 tid 100166 td 0xfffff800d6dba000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d380820
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d380860
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d3808e0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d380910
    _sleep() at _sleep+0x27d/frame 0xfffffe004d380990
    pipe_read() at pipe_read+0x2aa/frame 0xfffffe004d3809f0
    dofileread() at dofileread+0x95/frame 0xfffffe004d380a40
    kern_readv() at kern_readv+0x68/frame 0xfffffe004d380a90
    sys_read() at sys_read+0x63/frame 0xfffffe004d380ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d380bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d380bf0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8016f631a, rsp = 0x7fffffffeb28, rbp = 0x7fffffffeb40 ---
    
    Tracing command apinger pid 2912 tid 100261 td 0xfffff80068553490
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d45e7b0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d45e7f0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d45e870
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d45e8a0
    _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x18b/frame 0xfffffe004d45e900
    seltdwait() at seltdwait+0xa4/frame 0xfffffe004d45e940
    sys_poll() at sys_poll+0x3a3/frame 0xfffffe004d45eae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d45ebf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d45ebf0
    --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x800921a1a, rsp = 0x7fffffffcb88, rbp = 0x7fffffffec20 ---
    
    Tracing command sh pid 18688 tid 100144 td 0xfffff800684b6920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d31c7a0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d31c7e0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d31c860
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d31c890
    _cv_wait_sig() at _cv_wait_sig+0x16a/frame 0xfffffe004d31c8e0
    tty_wait() at tty_wait+0x1c/frame 0xfffffe004d31c900
    ttydisc_read() at ttydisc_read+0xf4/frame 0xfffffe004d31c960
    ttydev_read() at ttydev_read+0x86/frame 0xfffffe004d31c990
    devfs_read_f() at devfs_read_f+0xeb/frame 0xfffffe004d31c9f0
    dofileread() at dofileread+0x95/frame 0xfffffe004d31ca40
    kern_readv() at kern_readv+0x68/frame 0xfffffe004d31ca90
    sys_read() at sys_read+0x63/frame 0xfffffe004d31cae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d31cbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d31cbf0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800e1331a, rsp = 0x7fffffffe768, rbp = 0x7fffffffe860 ---
    
    Tracing command sh pid 18676 tid 100067 td 0xfffff800064e0920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d1a16d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d1a1710
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d1a1790
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d1a17c0
    _sleep() at _sleep+0x27d/frame 0xfffffe004d1a1840
    kern_wait6() at kern_wait6+0x5f4/frame 0xfffffe004d1a18f0
    sys_wait4() at sys_wait4+0x72/frame 0xfffffe004d1a1ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d1a1bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d1a1bf0
    --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d66fba, rsp = 0x7fffffffe818, rbp = 0x7fffffffe8c0 ---
    
    Tracing command sshlockout_pf pid 18362 tid 100174 td 0xfffff800d6d54490
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d38a8d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d38a910
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d38a990
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d38a9c0
    _sleep() at _sleep+0x238/frame 0xfffffe004d38aa40
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe004d38aaa0
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe004d38aae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d38abf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d38abf0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b8040a, rsp = 0x7fffffbfdf48, rbp = 0x7fffffbfdf70 ---
    
    Tracing command sshlockout_pf pid 18362 tid 100075 td 0xfffff800064df000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d1c9820
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d1c9860
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d1c98e0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d1c9910
    _sleep() at _sleep+0x27d/frame 0xfffffe004d1c9990
    pipe_read() at pipe_read+0x2aa/frame 0xfffffe004d1c99f0
    dofileread() at dofileread+0x95/frame 0xfffffe004d1c9a40
    kern_readv() at kern_readv+0x68/frame 0xfffffe004d1c9a90
    sys_read() at sys_read+0x63/frame 0xfffffe004d1c9ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d1c9bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d1c9bf0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800ba131a, rsp = 0x7fffffffe7d8, rbp = 0x7fffffffe7f0 ---
    
    Tracing command login pid 18048 tid 100094 td 0xfffff800065e9490
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d22c6d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d22c710
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d22c790
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d22c7c0
    _sleep() at _sleep+0x27d/frame 0xfffffe004d22c840
    kern_wait6() at kern_wait6+0x5f4/frame 0xfffffe004d22c8f0
    sys_wait4() at sys_wait4+0x72/frame 0xfffffe004d22cae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d22cbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d22cbf0
    --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800cf0fba, rsp = 0x7fffffffebc8, rbp = 0x7fffffffed40 ---
    
    Tracing command minicron pid 11941 tid 100080 td 0xfffff8000665b920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d1e28d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d1e2910
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d1e2990
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d1e29c0
    _sleep() at _sleep+0x238/frame 0xfffffe004d1e2a40
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe004d1e2aa0
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe004d1e2ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d1e2bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d1e2bf0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x80095a40a, rsp = 0x7fffffffeb88, rbp = 0x7fffffffebe0 ---
    
    Tracing command minicron pid 11551 tid 100077 td 0xfffff80006450920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d1d36d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d1d3710
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d1d3790
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d1d37c0
    _sleep() at _sleep+0x27d/frame 0xfffffe004d1d3840
    kern_wait6() at kern_wait6+0x5f4/frame 0xfffffe004d1d38f0
    sys_wait4() at sys_wait4+0x72/frame 0xfffffe004d1d3ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d1d3bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d1d3bf0
    --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8008cefba, rsp = 0x7fffffffebe8, rbp = 0x7fffffffec30 ---
    
    Tracing command minicron pid 11319 tid 100157 td 0xfffff800683cc000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d3538d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d353910
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d353990
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d3539c0
    _sleep() at _sleep+0x238/frame 0xfffffe004d353a40
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe004d353aa0
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe004d353ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d353bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d353bf0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x80095a40a, rsp = 0x7fffffffeb98, rbp = 0x7fffffffebf0 ---
    
    Tracing command minicron pid 10984 tid 100159 td 0xfffff800687f3920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d35d6d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d35d710
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d35d790
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d35d7c0
    _sleep() at _sleep+0x27d/frame 0xfffffe004d35d840
    kern_wait6() at kern_wait6+0x5f4/frame 0xfffffe004d35d8f0
    sys_wait4() at sys_wait4+0x72/frame 0xfffffe004d35dae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d35dbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d35dbf0
    --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8008cefba, rsp = 0x7fffffffebf8, rbp = 0x7fffffffec40 ---
    
    Tracing command minicron pid 10812 tid 100163 td 0xfffff800d6dbb000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d3718d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d371910
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d371990
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d3719c0
    _sleep() at _sleep+0x238/frame 0xfffffe004d371a40
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe004d371aa0
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe004d371ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d371bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d371bf0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x80095a40a, rsp = 0x7fffffffebb8, rbp = 0x7fffffffec10 ---
    
    Tracing command minicron pid 10357 tid 100146 td 0xfffff800684b6000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d3266d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d326710
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d326790
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d3267c0
    _sleep() at _sleep+0x27d/frame 0xfffffe004d326840
    kern_wait6() at kern_wait6+0x5f4/frame 0xfffffe004d3268f0
    sys_wait4() at sys_wait4+0x72/frame 0xfffffe004d326ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d326bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d326bf0
    --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8008cefba, rsp = 0x7fffffffec18, rbp = 0x7fffffffec60 ---
    
    Tracing command bandwidthd pid 95938 tid 100136 td 0xfffff800683be920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d2f47d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d2f4810
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d2f4890
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d2f48c0
    _sleep() at _sleep+0x238/frame 0xfffffe004d2f4940
    bpfread() at bpfread+0x1c7/frame 0xfffffe004d2f4990
    devfs_read_f() at devfs_read_f+0xeb/frame 0xfffffe004d2f49f0
    dofileread() at dofileread+0x95/frame 0xfffffe004d2f4a40
    kern_readv() at kern_readv+0x68/frame 0xfffffe004d2f4a90
    sys_read() at sys_read+0x63/frame 0xfffffe004d2f4ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d2f4bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d2f4bf0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8014b131a, rsp = 0x7fffffffe8d8, rbp = 0x7fffffffea10 ---
    
    Tracing command bandwidthd pid 95053 tid 100090 td 0xfffff8000665a000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d2187d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d218810
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d218890
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d2188c0
    _sleep() at _sleep+0x238/frame 0xfffffe004d218940
    bpfread() at bpfread+0x1c7/frame 0xfffffe004d218990
    devfs_read_f() at devfs_read_f+0xeb/frame 0xfffffe004d2189f0
    dofileread() at dofileread+0x95/frame 0xfffffe004d218a40
    kern_readv() at kern_readv+0x68/frame 0xfffffe004d218a90
    sys_read() at sys_read+0x63/frame 0xfffffe004d218ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d218bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d218bf0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8014b131a, rsp = 0x7fffffffe8d8, rbp = 0x7fffffffea10 ---
    
    Tracing command bandwidthd pid 95003 tid 100160 td 0xfffff800684b5000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d3627d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d362810
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d362890
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d3628c0
    _sleep() at _sleep+0x238/frame 0xfffffe004d362940
    bpfread() at bpfread+0x1c7/frame 0xfffffe004d362990
    devfs_read_f() at devfs_read_f+0xeb/frame 0xfffffe004d3629f0
    dofileread() at dofileread+0x95/frame 0xfffffe004d362a40
    kern_readv() at kern_readv+0x68/frame 0xfffffe004d362a90
    sys_read() at sys_read+0x63/frame 0xfffffe004d362ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d362bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d362bf0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8014b131a, rsp = 0x7fffffffe8d8, rbp = 0x7fffffffea10 ---
    
    Tracing command bandwidthd pid 94723 tid 100088 td 0xfffff800065bb490
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d20e7d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d20e810
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d20e890
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d20e8c0
    _sleep() at _sleep+0x238/frame 0xfffffe004d20e940
    bpfread() at bpfread+0x1c7/frame 0xfffffe004d20e990
    devfs_read_f() at devfs_read_f+0xeb/frame 0xfffffe004d20e9f0
    dofileread() at dofileread+0x95/frame 0xfffffe004d20ea40
    kern_readv() at kern_readv+0x68/frame 0xfffffe004d20ea90
    sys_read() at sys_read+0x63/frame 0xfffffe004d20eae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d20ebf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d20ebf0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8014b131a, rsp = 0x7fffffffe8d8, rbp = 0x7fffffffea10 ---
    
    Tracing command bandwidthd pid 94044 tid 100093 td 0xfffff800066ad920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d2277d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d227810
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d227890
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d2278c0
    _sleep() at _sleep+0x238/frame 0xfffffe004d227940
    bpfread() at bpfread+0x1c7/frame 0xfffffe004d227990
    devfs_read_f() at devfs_read_f+0xeb/frame 0xfffffe004d2279f0
    dofileread() at dofileread+0x95/frame 0xfffffe004d227a40
    kern_readv() at kern_readv+0x68/frame 0xfffffe004d227a90
    sys_read() at sys_read+0x63/frame 0xfffffe004d227ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d227bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d227bf0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8014b131a, rsp = 0x7fffffffe8d8, rbp = 0x7fffffffea10 ---
    
    Tracing command bandwidthd pid 93233 tid 100110 td 0xfffff80068156000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d27c7d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d27c810
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d27c890
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d27c8c0
    _sleep() at _sleep+0x238/frame 0xfffffe004d27c940
    bpfread() at bpfread+0x1c7/frame 0xfffffe004d27c990
    devfs_read_f() at devfs_read_f+0xeb/frame 0xfffffe004d27c9f0
    dofileread() at dofileread+0x95/frame 0xfffffe004d27ca40
    kern_readv() at kern_readv+0x68/frame 0xfffffe004d27ca90
    sys_read() at sys_read+0x63/frame 0xfffffe004d27cae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d27cbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d27cbf0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8014b131a, rsp = 0x7fffffffe8d8, rbp = 0x7fffffffea10 ---
    
    Tracing command bandwidthd pid 92617 tid 100147 td 0xfffff800684b5920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d32b7d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d32b810
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d32b890
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d32b8c0
    _sleep() at _sleep+0x238/frame 0xfffffe004d32b940
    bpfread() at bpfread+0x1c7/frame 0xfffffe004d32b990
    devfs_read_f() at devfs_read_f+0xeb/frame 0xfffffe004d32b9f0
    dofileread() at dofileread+0x95/frame 0xfffffe004d32ba40
    kern_readv() at kern_readv+0x68/frame 0xfffffe004d32ba90
    sys_read() at sys_read+0x63/frame 0xfffffe004d32bae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d32bbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d32bbf0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8014b131a, rsp = 0x7fffffffe8d8, rbp = 0x7fffffffea10 ---
    
    Tracing command bandwidthd pid 92206 tid 100142 td 0xfffff80006450490
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d3127d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d312810
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d312890
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d3128c0
    _sleep() at _sleep+0x238/frame 0xfffffe004d312940
    bpfread() at bpfread+0x1c7/frame 0xfffffe004d312990
    devfs_read_f() at devfs_read_f+0xeb/frame 0xfffffe004d3129f0
    dofileread() at dofileread+0x95/frame 0xfffffe004d312a40
    kern_readv() at kern_readv+0x68/frame 0xfffffe004d312a90
    sys_read() at sys_read+0x63/frame 0xfffffe004d312ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d312bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d312bf0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8014b131a, rsp = 0x7fffffffe8d8, rbp = 0x7fffffffea10 ---
    
    Tracing command cron pid 78373 tid 100151 td 0xfffff8000644f000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d33f8d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d33f910
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d33f990
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d33f9c0
    _sleep() at _sleep+0x238/frame 0xfffffe004d33fa40
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe004d33faa0
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe004d33fae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d33fbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d33fbf0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800d8040a, rsp = 0x7fffffffea88, rbp = 0x7fffffffecb0 ---
    
    Tracing command syslogd pid 77823 tid 100167 td 0xfffff800d6cb9920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d385720
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d385760
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d3857e0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d385810
    _cv_wait_sig() at _cv_wait_sig+0x16a/frame 0xfffffe004d385860
    seltdwait() at seltdwait+0xae/frame 0xfffffe004d3858a0
    kern_select() at kern_select+0x8fa/frame 0xfffffe004d385aa0
    sys_select() at sys_select+0x54/frame 0xfffffe004d385ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d385bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d385bf0
    --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800b9529a, rsp = 0x7fffffffdbf8, rbp = 0x7fffffffec10 ---
    
    Tracing command miniupnpd pid 44882 tid 100141 td 0xfffff800683cc920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d30d710
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d30d750
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d30d7d0
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe004d30d800
    _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x18b/frame 0xfffffe004d30d860
    seltdwait() at seltdwait+0xa4/frame 0xfffffe004d30d8a0
    kern_select() at kern_select+0x8fa/frame 0xfffffe004d30daa0
    sys_select() at sys_select+0x54/frame 0xfffffe004d30dae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d30dbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d30dbf0
    --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80120029a, rsp = 0x7fffffffe028, rbp = 0x7fffffffec50 ---
    
    Tracing command charon pid 34325 tid 100128 td 0xfffff800683a2490
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d2d6850
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d2d6890
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d2d6910
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d2d6940
    _sleep() at _sleep+0x27d/frame 0xfffffe004d2d69c0
    umtxq_sleep() at umtxq_sleep+0x125/frame 0xfffffe004d2d6a20
    do_wait() at do_wait+0x329/frame 0xfffffe004d2d6aa0
    __umtx_op_wait_uint_private() at __umtx_op_wait_uint_private+0x83/frame 0xfffffe004d2d6ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d2d6bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d2d6bf0
    --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x8011638cc, rsp = 0x7ffffddeee58, rbp = 0x7ffffddeee80 ---
    
    Tracing command charon pid 34325 tid 100127 td 0xfffff800683a2920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d2d1850
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d2d1890
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d2d1910
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d2d1940
    _sleep() at _sleep+0x27d/frame 0xfffffe004d2d19c0
    umtxq_sleep() at umtxq_sleep+0x125/frame 0xfffffe004d2d1a20
    do_wait() at do_wait+0x329/frame 0xfffffe004d2d1aa0
    __umtx_op_wait_uint_private() at __umtx_op_wait_uint_private+0x83/frame 0xfffffe004d2d1ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d2d1bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d2d1bf0
    --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x8011638cc, rsp = 0x7ffffdfefe58, rbp = 0x7ffffdfefe80 ---
    
    Tracing command charon pid 34325 tid 100126 td 0xfffff800683a3000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d2cc850
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d2cc890
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d2cc910
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d2cc940
    _sleep() at _sleep+0x27d/frame 0xfffffe004d2cc9c0
    umtxq_sleep() at umtxq_sleep+0x125/frame 0xfffffe004d2cca20
    do_wait() at do_wait+0x329/frame 0xfffffe004d2ccaa0
    __umtx_op_wait_uint_private() at __umtx_op_wait_uint_private+0x83/frame 0xfffffe004d2ccae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d2ccbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d2ccbf0
    --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x8011638cc, rsp = 0x7ffffe1f0e58, rbp = 0x7ffffe1f0e80 ---
    
    Tracing command charon pid 34325 tid 100125 td 0xfffff800683a3490
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d2c7850
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d2c7890
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d2c7910
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d2c7940
    _sleep() at _sleep+0x27d/frame 0xfffffe004d2c79c0
    umtxq_sleep() at umtxq_sleep+0x125/frame 0xfffffe004d2c7a20
    do_wait() at do_wait+0x329/frame 0xfffffe004d2c7aa0
    __umtx_op_wait_uint_private() at __umtx_op_wait_uint_private+0x83/frame 0xfffffe004d2c7ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d2c7bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d2c7bf0
    --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x8011638cc, rsp = 0x7ffffe3f1e58, rbp = 0x7ffffe3f1e80 ---
    
    Tracing command charon pid 34325 tid 100124 td 0xfffff800683a3920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d2c2850
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d2c2890
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d2c2910
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d2c2940
    _sleep() at _sleep+0x27d/frame 0xfffffe004d2c29c0
    umtxq_sleep() at umtxq_sleep+0x125/frame 0xfffffe004d2c2a20
    do_wait() at do_wait+0x329/frame 0xfffffe004d2c2aa0
    __umtx_op_wait_uint_private() at __umtx_op_wait_uint_private+0x83/frame 0xfffffe004d2c2ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d2c2bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d2c2bf0
    --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x8011638cc, rsp = 0x7ffffe5f2e58, rbp = 0x7ffffe5f2e80 ---
    
    Tracing command charon pid 34325 tid 100123 td 0xfffff800683a4000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d2bd850
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d2bd890
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d2bd910
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d2bd940
    _sleep() at _sleep+0x27d/frame 0xfffffe004d2bd9c0
    umtxq_sleep() at umtxq_sleep+0x125/frame 0xfffffe004d2bda20
    do_wait() at do_wait+0x329/frame 0xfffffe004d2bdaa0
    __umtx_op_wait_uint_private() at __umtx_op_wait_uint_private+0x83/frame 0xfffffe004d2bdae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d2bdbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d2bdbf0
    --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x8011638cc, rsp = 0x7ffffe7f3e58, rbp = 0x7ffffe7f3e80 ---
    
    Tracing command charon pid 34325 tid 100122 td 0xfffff800683a4490
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d2b8850
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d2b8890
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d2b8910
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d2b8940
    _sleep() at _sleep+0x27d/frame 0xfffffe004d2b89c0
    umtxq_sleep() at umtxq_sleep+0x125/frame 0xfffffe004d2b8a20
    do_wait() at do_wait+0x329/frame 0xfffffe004d2b8aa0
    __umtx_op_wait_uint_private() at __umtx_op_wait_uint_private+0x83/frame 0xfffffe004d2b8ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d2b8bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d2b8bf0
    --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x8011638cc, rsp = 0x7ffffe9f4e58, rbp = 0x7ffffe9f4e80 ---
    
    Tracing command charon pid 34325 tid 100121 td 0xfffff800683a4920
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d2b3850
    mi_switch() at mi_switch+0xe1/frame 0xfffffe004d2b3890
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe004d2b3910
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe004d2b3940
    _sleep() at _sleep+0x27d/frame 0xfffffe004d2b39c0
    umtxq_sleep() at umtxq_sleep+0x125/frame 0xfffffe004d2b3a20
    do_wait() at do_wait+0x329/frame 0xfffffe004d2b3aa0
    __umtx_op_wait_uint_private() at __umtx_op_wait_uint_private+0x83/frame 0xfffffe004d2b3ae0
    amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe004d2b3bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe004d2b3bf0
    --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x8011638cc, rsp = 0x7ffffebf5e58, rbp = 0x7ffffebf5e80 ---
    
    Tracing command charon pid 34325 tid 100120 td 0xfffff80006f3a000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe004d2ae850
    mi_switch() at mi_switch+0xe1/fr[/thread]
    


  • The forum truncated my crash report (and the rest of my post) even though it was in code tags and it worked fine in the forum post preview. Here's my system information:

    Name	fuckmountain.no-ip.biz
    Version	2.2.2-RELEASE (amd64) 
    built on Mon Apr 13 20:10:22 CDT 2015 
    FreeBSD 10.1-RELEASE-p9
    
    You are on the latest version.
    Platform	pfSense
    CPU Type	Dual Core AMD Opteron(tm) Processor 285
    4 CPUs: 2 package(s) x 2 core(s)
    Uptime	00 Hour 13 Minutes 04 Seconds
    Current date/time	
    Sat May 9 9:34:09 CDT 2015
    DNS server(s)	127.0.0.1
    208.67.222.222
    208.67.220.220
    208.67.222.220
    208.67.220.222
    Last config change	Sat May 9 6:30:10 CDT 2015
    State table size	
    0% (1731/508000) 
    Show states
    MBUF Usage	
    10% (2540/26584)
    Load average	
    0.82, 0.86, 0.56
    CPU usage	
    0%
    Memory usage	
    10% of 5084 MB
    SWAP usage	
    0% of 16384 MB
    Disk usage	
    / (ufs): 2% of 129G	
    /var/run (ufs in RAM): 4% of 3.4M	
    

  • Rebel Alliance Developer Netgate

    That looks like a problem with the Broadcom (bge) card or driver in that system. Is that an on-board or add-on NIC?
    What, if any, changes have you put in place in /boot/loader.conf.local ?