pfSense has detected a crash report or programming bug. Click here for more information
-
Hi,
One of my pfsense boxes started to crash, the only thing I changed lately was the AES-NI CPU Crypto set to active.
Any help is very welcome.
Here is the dump, the xxx you see is due to data removed by me (I tried to insert the entire dump, but it is not allowing me to post it due to the number of characters):
Crash report begins. Anonymous machine information: amd64 11.2-RELEASE-p6 FreeBSD 11.2-RELEASE-p6 #3 518496b29ae(RELENG_2_4_4): Wed Dec 12 07:41:44 EST 2018 root@buildbot2.nyi.netgate.com:/build/ce-crossbuild-244/obj/amd64/ZfGpH5cd/build/ce-crossbuild-244/pfSense/tmp/FreeBSD-src/sys/pfSense Crash report details: No PHP errors found. Filename: /var/crash/info.0 Dump header from device: /dev/label/swap0 Architecture: amd64 Architecture Version: 1 Dump Length: 70656 Blocksize: 512 Dumptime: Tue May 7 23:23:10 2019 Hostname: teletalk-firewall.localdomain Magic: FreeBSD Text Dump Version String: FreeBSD 11.2-RELEASE-p6 #3 518496b29ae(RELENG_2_4_4): Wed Dec 12 07:41:44 EST 2018 root@buildbot2.nyi.netgate.com:/build/ce-crossbuild-244/obj/amd64/ZfGpH5cd/build/ce-crossbuild-244/pfSe Panic String: ffs_blkfree_cg: freeing free frag Dump Parity: 2872411486 Bounds: 0 Dump Status: good Filename: /var/crash/textdump.tar.0 ddb.txt06000014000013464436616 7107 ustarrootwheeldb:0:kdb.enter.default> run lockinfo db:1:lockinfo> show locks No such command; use "help" to list available commands db:1:lockinfo> show alllocks No such command; use "help" to list available commands db:1:lockinfo> show lockedvnods Locked vnodes db:0:kdb.enter.default> show pcpu cpuid = 6 dynamic pcpu = 0xfffffe00f845d480 curthread = 0xfffff8001803d000: pid 29 "/ worker" curpcb = 0xfffffe009b36fb80 fpcurthread = none idlethread = 0xfffff80003538000: tid 100009 "idle: cpu6" curpmap = 0xffffffff82b83898 tssp = 0xffffffff82bb4980 commontssp = 0xffffffff82bb4980 rsp0 = 0xfffffe009b36fb80 gs32p = 0xffffffff82bbb1d8 ldt = 0xffffffff82bbb218 tss = 0xffffffff82bbb208 db:0:kdb.enter.default> bt Tracing pid 29 tid 100140 td 0xfffff8001803d000 kdb_enter() at kdb_enter+0x3b/frame 0xfffffe009b36f6f0 vpanic() at vpanic+0x194/frame 0xfffffe009b36f750 panic() at panic+0x43/frame 0xfffffe009b36f7b0 ffs_blkfree_cg() at ffs_blkfree_cg+0x5ac/frame 0xfffffe009b36f850 ffs_blkfree() at ffs_blkfree+0x99/frame 0xfffffe009b36f8b0 freework_freeblock() at freework_freeblock+0x257/frame 0xfffffe009b36f910 handle_workitem_freeblocks() at handle_workitem_freeblocks+0x119/frame 0xfffffe009b36f970 process_worklist_item() at process_worklist_item+0x24a/frame 0xfffffe009b36f9e0 softdep_process_worklist() at softdep_process_worklist+0xc2/frame 0xfffffe009b36fa30 softdep_flush() at softdep_flush+0x11f/frame 0xfffffe009b36fa70 fork_exit() at fork_exit+0x83/frame 0xfffffe009b36fab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe009b36fab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db:0:kdb.enter.default> ps pid ppid pgrp uid state wmesg wchan cmd 97756 19406 32 0 S nanslp 0xffffffff829cf607 sleep 30584 1 30584 0 Ss (threaded) filterdns 100254 S uwait 0xfffff80063e53c80 filterdns 100255 S uwait 0xfffff80063e53e00 filterdns 100256 S uwait 0xfffff8006c196200 filterdns 100257 S uwait 0xfffff8006c196100 filterdns 100258 S uwait 0xfffff800759e0a00 filterdns 100259 S uwait 0xfffff8006db56480 filterdns 100260 S uwait 0xfffff8006db57880 filterdns 100261 S uwait 0xfffff8006db57980 filterdns 100262 S uwait 0xfffff8006db57a00 filterdns 100263 S uwait 0xfffff8006db57b00 filterdns 100264 S uwait 0xfffff8006db55f00 filterdns 100265 S uwait 0xfffff8006db56080 filterdns 100266 S uwait 0xfffff8006db56180 filterdns 100267 S uwait 0xfffff8006db56280 filterdns 100268 S uwait 0xfffff8006db56380 filterdns 100269 S uwait 0xfffff8006db55a00 filterdns 100270 S uwait 0xfffff8006db55b00 filterdns 100271 S uwait 0xfffff8006db55c00 filterdns 100272 S uwait 0xfffff8006db55d00 filterdns 100273 S uwait 0xfffff8006db55e00 filterdns 100274 S uwait 0xfffff8006db55580 filterdns 100275 S uwait 0xfffff8006db57780 filterdns 100276 S uwait 0xfffff8006db55700 filterdns 100277 S uwait 0xfffff8006db55800 filterdns 100278 S uwait 0xfffff8006db55900 filterdns 100279 S uwait 0xfffff8006db55080 filterdns 100280 S uwait 0xfffff8006db55180 filterdns 100281 S uwait 0xfffff8006db55280 filterdns 100282 S uwait 0xfffff8006db55380 filterdns 100283 S uwait 0xfffff8006db55480 filterdns 100284 S uwait 0xfffff8006db54b00 filterdns 100285 S uwait 0xfffff8006db54c00 filterdns 100286 S uwait 0xfffff8006db54d00 filterdns 100287 S uwait 0xfffff8006db54e00 filterdns 100288 S uwait 0xfffff8006db54f00 filterdns 100289 S uwait 0xfffff80018c8ca80 filterdns 100290 S uwait 0xfffff800189e5800 filterdns 100291 S uwait 0xfffff80018c8d900 filterdns 100292 S uwait 0xfffff8006db54980 filterdns 100293 S uwait 0xfffff80063e53180 filterdns 100294 S uwait 0xfffff800180f9d80 filterdns 100295 S uwait 0xfffff80063e51e00 filterdns 100296 S uwait 0xfffff800035b1900 filterdns 100297 S uwait 0xfffff800180f9f00 filterdns 100298 S uwait 0xfffff800180f9400 filterdns 100299 S uwait 0xfffff800180f9b80 filterdns 100300 S uwait 0xfffff800180f9a80 filterdns 100301 S uwait 0xfffff800180f9d00 filterdns 100302 S uwait 0xfffff800189e2580 filterdns 100303 S uwait 0xfffff80018a29500 filterdns 100304 S uwait 0xfffff80018023500 filterdns 100305 S uwait 0xfffff80063e50a80 filterdns 100306 S uwait 0xfffff8001842d580 filterdns 100307 S uwait 0xfffff80003a6fb80 filterdns 100308 S uwait 0xfffff80018c86580 xxxx.net 100309 S uwait 0xfffff800189e5d00 xxxxx 100310 S uwait 0xfffff80018203180 xxxx.net 100311 S uwait 0xfffff8001842c680 xxxxx 100312 S uwait 0xfffff80063e50f00 xxxxxxx.net 100313 S uwait 0xfffff800713b4c00 xxxxxx 100314 S uwait 0xfffff800759dd280 xxxxxx 100315 S uwait 0xfffff800759dd380 mail2.uptimerobot.c 100316 S uwait 0xfffff800759dd480 engine5.uptimerobot 100317 S uwait 0xfffff800759dd580 engine6.uptimerobot 100318 S uwait 0xfffff800759dd680 engine7.uptimerobot 100319 S uwait 0xfffff800759dcd80 engine8.uptimerobot 100320 S uwait 0xfffff800759dce80 engine9.uptimerobot 100321 S uwait 0xfffff800759dd000 engine10.uptimerobo 100322 S uwait 0xfffff800759dd100 engine11.uptimerobo 100323 S uwait 0xfffff800759dd180 engine12.uptimerobo 100324 S uwait 0xfffff800759dc880 engine13.uptimerobo 100325 S uwait 0xfffff800759dc980 engine14.uptimerobo 100326 S uwait 0xfffff800759dca80 engine15.uptimerobo 100327 S uwait 0xfffff800759dcb80 engine16.uptimerobo 100328 S uwait 0xfffff800759dcc80 engine17.uptimerobo 100329 S uwait 0xfffff800759dc380 engine18.uptimerobo 100330 S uwait 0xfffff800759dc480 engine19.uptimerobo 100331 S uwait 0xfffff800759dc580 engine20.uptimerobo 100332 S uwait 0xfffff800759dc680 engine21.uptimerobo 100333 S uwait 0xfffff800759dc780 engine22.uptimerobo 100334 S uwait 0xfffff800759dbe80 engine23.uptimerobo 100335 S uwait 0xfffff8006db54580 engine24.uptimerobo 100336 S uwait 0xfffff800759dc080 engine25.uptimerobo 100337 S uwait 0xfffff800759dc180 engine26.uptimerobo 100338 S uwait 0xfffff800759dc280 engine27.uptimerobo 100339 S uwait 0xfffff800759db980 engine28.uptimerobo 100340 S uwait 0xfffff800759dba80 remote1.uptimerobot 100341 S uwait 0xfffff800759dbb80 remote2.uptimerobot 100342 S uwait 0xfffff800759dbc80 remote3.uptimerobot 100343 S uwait 0xfffff800759dbd80 remote4.uptimerobot 100344 S uwait 0xfffff800759dd900 remote5.uptimerobot 100345 S uwait 0xfffff800759ddc80 remote6.uptimerobot 100346 S uwait 0xfffff800759dd800 remote7.uptimerobot 100347 S uwait 0xfffff800759db780 remote8.uptimerobot 100348 S uwait 0xfffff800759db880 remote9.uptimerobot 100349 S uwait 0xfffff80018c8cc00 remote10.uptimerobo 100350 S uwait 0xfffff800759db480 remote11.uptimerobo 100351 S uwait 0xfffff800759db580 signal-thread 29803 1 29803 0 Ss+ ttyin 0xfffff800180138b0 getty 29496 1 29496 0 Ss+ ttyin 0xfffff80018013cb0 getty 29309 1 29309 0 Ss+ ttyin 0xfffff800180140b0 getty 29096 1 29096 0 Ss+ ttyin 0xfffff8001005b4b0 getty 28998 1 28998 0 Ss+ ttyin 0xfffff800100524b0 getty 28861 1 28861 0 Ss+ ttyin 0xfffff800100528b0 getty 28762 1 28762 0 Ss+ ttyin 0xfffff80010052cb0 getty 28715 1 28715 0 Ss+ ttyin 0xfffff8001005b0b0 getty 8931 8064 7803 122 S nanslp 0xffffffff829cf604 zabbix_agentd 8750 8064 7803 122 S accept 0xfffff8001878ea8c zabbix_agentd 8548 8064 7803 122 S select 0xfffff8006c1987c0 zabbix_agentd 8496 8064 7803 122 S select 0xfffff80018c8bdc0 zabbix_agentd 8374 8064 7803 122 S nanslp 0xffffffff829cf601 zabbix_agentd 8064 1 7803 122 S wait 0xfffff80018091590 zabbix_agentd 95391 344 344 0 S accept 0xfffff8001853b3cc php-fpm 64409 63003 344 0 S bpf 0xfffff800189a3000 bandwidthd 64113 63003 344 0 S bpf 0xfffff80018044600 bandwidthd 63992 63003 344 0 S bpf 0xfffff80063dfbe00 bandwidthd 63659 62542 344 0 S bpf 0xfffff80063dfb000 bandwidthd 63617 62542 344 0 S bpf 0xfffff80018047e00 bandwidthd 63282 62542 344 0 S bpf 0xfffff80063e0fa00 bandwidthd 63003 1 344 0 S bpf 0xfffff80063e15800 bandwidthd 62542 1 344 0 S bpf 0xfffff800189a3600 bandwidthd 60610 60320 59768 0 S piperd 0xfffff800180dfbe0 sh 60320 59768 59768 0 S wait 0xfffff80018221590 sh 60181 59768 59768 0 S (threaded) sshg-blocker 100224 S piperd 0xfffff800182162f8 sshg-blocker 100226 S nanslp 0xffffffff829cf600 sshg-blocker 60114 59768 59768 0 S piperd 0xfffff800181f35f0 sshg-parser 59880 59768 59768 0 S piperd 0xfffff800181f38e8 cat 59768 42076 59768 0 Ss wait 0xfffff80075689000 sh 44850 44334 44334 0 S nanslp 0xffffffff829cf603 minicron 44334 1 44334 0 Ss wait 0xfffff80018268590 minicron 44126 43901 43901 0 S nanslp 0xffffffff829cf607 minicron 43901 1 43901 0 Ss wait 0xfffff80018090590 minicron 43613 43204 43204 0 S nanslp 0xffffffff829cf603 minicron 43204 1 43204 0 Ss wait 0xfffff8001825f590 minicron 42076 1 42076 0 Ss select 0xfffff800035cedc0 syslogd 19406 1 32 0 S+ wait 0xfffff800182ea000 sh 80259 1 80259 0 Ss (threaded) ntpd 100199 S select 0xfffff80018a294c0 ntpd 100550 S usem 0xfffff8006130af00 ntpd 79634 1 79634 0 Ss nanslp 0xffffffff829cf601 cron 78672 78138 78138 0 S kqread 0xfffff800181adb00 nginx 78408 78138 78138 0 S kqread 0xfffff80018879e00 nginx 78138 1 78138 0 Ss pause 0xfffff80003a5d0a8 nginx 63686 1 63686 59 Ss (threaded) unbound 100147 S kqread 0xfffff80010134500 unbound 100190 S kqread 0xfffff80003a76800 unbound 100191 S kqread 0xfffff800101b2d00 unbound 100192 S kqread 0xfffff800181ab200 unbound 100193 S kqread 0xfffff80018a06600 unbound 100194 S kqread 0xfffff800189d0400 unbound 100195 S kqread 0xfffff80003a76b00 unbound 100196 S kqread 0xfffff800101b3c00 unbound 52144 1 52144 0 Ss (threaded) dpinger 100177 S uwait 0xfffff80018206300 dpinger 100185 S sbwait 0xfffff8001847f144 dpinger 100186 S nanslp 0xffffffff829cf600 dpinger 100187 S nanslp 0xffffffff829cf600 dpinger 100188 S accept 0xfffff80018738a8c dpinger 48162 1 48162 0 Ss select 0xfffff800189e43c0 sshd 25369 1 25369 0 Ss bpf 0xfffff80018045c00 filterlog 23904 1 23904 0 Ss select 0xfffff800035cf040 openvpn 445 1 445 0 Ss select 0xfffff80018205b40 devd 426 1 426 0 Ss select 0xfffff80018003640 hv_vss_daemon 424 1 424 0 Ss select 0xfffff8001012fe40 hv_kvp_daemon 385 383 383 0 S kqread 0xfffff80010086e00 check_reload_status 383 1 383 0 Ss kqread 0xfffff80003a77800 check_reload_status 346 344 344 0 S accept 0xfffff8001853b3cc php-fpm 345 344 344 0 S accept 0xfffff8001853b3cc php-fpm 344 1 344 0 Ss kqread 0xfffff800181ac500 php-fpm 64 0 0 0 DL mdwait 0xfffff800100e4000 [md0] 31 0 0 0 DL syncer 0xffffffff82ac8480 [syncer] 30 0 0 0 DL vlruwt 0xfffff80003a63590 [vnlru] 29 0 0 0 RL (threaded) [bufdaemon] 100133 D psleep 0xffffffff82ac7728 [bufdaemon] 100140 Run CPU 6 [/ worker] 28 0 0 0 DL - 0xffffffff82ac78fc [bufspacedaemon] 27 0 0 0 DL pgzero 0xffffffff82b09468 [pagezero] 26 0 0 0 DL psleep 0xffffffff82b09458 [vmdaemon] 25 0 0 0 DL (threaded) [pagedaemon] 100127 D psleep 0xffffffff82bb3c25 [dom0] 100129 D launds 0xffffffff82b058c4 [laundry: dom0] 100131 D umarcl 0xffffffff82b05238 [uma] 24 0 0 0 DL - 0xffffffff828a21d8 [rand_harvestq] 23 0 0 0 DL pftm 0xffffffff80f3f040 [pf purge] 22 0 0 0 DL waiting_ 0xffffffff82baaae0 [sctp_iterator] 21 0 0 0 DL - 0xffffffff82ac7194 [soaiod4] 20 0 0 0 DL - 0xffffffff82ac7194 [soaiod3] 19 0 0 0 DL - 0xffffffff82ac7194 [soaiod2] 18 0 0 0 DL - 0xffffffff82ac7194 [soaiod1] 17 0 0 0 DL - 0xfffff8000383e448 [fdc0] 16 0 0 0 DL (threaded) [cam] 100080 D - 0xffffffff82754f00 [doneq0] 100081 D - 0xffffffff82755000 [doneq1] 100102 D - 0xffffffff82754d48 [scanner] 15 0 0 0 DL crypto_r 0xfffff800035d86d0 [crypto returns 7] 9 0 0 0 DL crypto_r 0xfffff800035d8670 [crypto returns 6] 8 0 0 0 DL crypto_r 0xfffff800035d8610 [crypto returns 5] 7 0 0 0 DL crypto_r 0xfffff800035d85b0 [crypto returns 4] 6 0 0 0 DL crypto_r 0xfffff800035d8550 [crypto returns 3] 5 0 0 0 DL crypto_r 0xfffff800035d84f0 [crypto returns 2] 4 0 0 0 DL crypto_r 0xfffff800035d8490 [crypto returns 1] 3 0 0 0 DL crypto_r 0xfffff800035d8430 [crypto returns 0] 2 0 0 0 DL crypto_w 0xffffffff82b03d60 [crypto] 14 0 0 0 DL (threaded) [geom] 100059 D - 0xffffffff82b82b40 [g_event] 100060 D - 0xffffffff82b82b48 [g_up] 100061 D - 0xffffffff82b82b50 [g_down] 13 0 0 0 DL (threaded) [ng_queue] 100051 D sleep 0xffffffff8270d580 [ng_queue0] 100052 D sleep 0xffffffff8270d580 [ng_queue1] 100053 D sleep 0xffffffff8270d580 [ng_queue2] 100054 D sleep 0xffffffff8270d580 [ng_queue3] 100055 D sleep 0xffffffff8270d580 [ng_queue4] 100056 D sleep 0xffffffff8270d580 [ng_queue5] 100057 D sleep 0xffffffff8270d580 [ng_queue6] 100058 D sleep 0xffffffff8270d580 [ng_queue7] 12 0 0 0 WL (threaded) [intr] 100011 I [swi6: task queue] 100012 I [swi6: Giant taskq] 100015 I [swi5: fast taskq] 100034 I [swi1: netisr 0] 100035 I [swi1: netisr 1] 100036 I [swi1: netisr 2] 100037 I [swi1: netisr 3] 100038 I [swi1: netisr 4] 100039 I [swi1: netisr 5] 100040 I [swi1: netisr 6] 100041 I [swi1: netisr 7] 100042 I [swi4: clock (0)] 100043 I [swi4: clock (1)] 100044 I [swi4: clock (2)] 100045 I [swi4: clock (3)] 100046 I [swi4: clock (4)] 100047 I [swi4: clock (5)] 100048 I [swi4: clock (6)] 100049 I [swi4: clock (7)] 100050 I [swi3: vm] 100082 I [irq14: ata0] 100083 I [irq15: ata1] 100084 I [irq1: atkbd0] 100085 I [irq12: psm0] 100086 I [swi0: uart uart] 100095 I [swi1: pf send] 100096 I [swi1: pfsync] 11 0 0 0 RL (threaded) [idle] 100003 Run CPU 0 [idle: cpu0] 100004 Run CPU 1 [idle: cpu1] 100005 Run CPU 2 [idle: cpu2] 100006 Run CPU 3 [idle: cpu3] 100007 Run CPU 4 [idle: cpu4] 100008 Run CPU 5 [idle: cpu5] 100009 CanRun [idle: cpu6] 100010 Run CPU 7 [idle: cpu7] 1 0 1 0 SLs wait 0xfffff80003527590 [init] 10 0 0 0 DL audit_wo 0xffffffff82bb1cc0 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0xffffffff82b82b88 [swapper] 100013 D - 0xfffff80003512200 [thread taskq] 100014 D - 0xfffff80003512000 [aiod_kick taskq] 100016 D - 0xfffff80003511a00 [kqueue_ctx taskq] 100017 D - 0xfffff80003511800 [if_io_tqg_0] 100018 D - 0xfffff80003511600 [if_io_tqg_1] 100019 D - 0xfffff80003511400 [if_io_tqg_2] 100020 D - 0xfffff80003511200 [if_io_tqg_3] 100021 D - 0xfffff80003511000 [if_io_tqg_4] 100022 D - 0xfffff80003553000 [if_io_tqg_5] 100023 D - 0xfffff80003552d00 [if_io_tqg_6] 100024 D - 0xfffff80003552b00 [if_io_tqg_7] 100025 D - 0xfffff80003552900 [softirq_0] 100026 D - 0xfffff80003552700 [softirq_1] 100027 D - 0xfffff80003552500 [softirq_2] 100028 D - 0xfffff80003552300 [softirq_3] 100029 D - 0xfffff80003552100 [softirq_4] 100030 D - 0xfffff80003551e00 [softirq_5] 100031 D - 0xfffff80003551c00 [softirq_6] 100032 D - 0xfffff80003551a00 [softirq_7] 100033 D - 0xfffff80003551800 [if_config_tqg_0] 100062 D - 0xfffff800035b5d00 [firmware taskq] 100063 D - 0xfffff800035b5300 [crypto_0] 100064 D - 0xfffff800035b5300 [crypto_1] 100065 D - 0xfffff800035b5300 [crypto_2] 100066 D - 0xfffff800035b5300 [crypto_3] 100067 D - 0xfffff800035b5300 [crypto_4] 100068 D - 0xfffff800035b5300 [crypto_5] 100069 D - 0xfffff800035b5300 [crypto_6] 100070 D - 0xfffff800035b5300 [crypto_7] 100088 D - 0xfffff80003837100 [mca taskq] 100098 D - 0xfffff80003aa8100 [acpi_task_0] 100099 D - 0xfffff80003aa8100 [acpi_task_1] 100100 D - 0xfffff80003aa8100 [acpi_task_2] 100101 D - 0xfffff80003553800 [CAM taskq] 100103 D - 0xfffff80003b3e600 [hvevent0] 100104 D - 0xfffff80003b3e400 [hvmsg0] 100105 D - 0xfffff80003b3e200 [hvevent1] 100106 D - 0xfffff80003b3e000 [hvmsg1] 100107 D - 0xfffff80003b3dd00 [hvevent2] 100108 D - 0xfffff80003b3db00 [hvmsg2] 100109 D - 0xfffff80003b3d900 [hvevent3] 100110 D - 0xfffff80003b3d700 [hvmsg3] 100111 D - 0xfffff80003b3d500 [hvevent4] 100112 D - 0xfffff80003b3d300 [hvmsg4] 100113 D - 0xfffff80003b3d100 [hvevent5] 100114 D - 0xfffff80003b3ce00 [hvmsg5] 100115 D - 0xfffff80003b3cc00 [hvevent6] 100116 D - 0xfffff80003b3ca00 [hvmsg6] 100117 D - 0xfffff80003b3c800 [hvevent7] 100118 D - 0xfffff80003b3c600 [hvmsg7] 100119 D - 0xfffff80003b3c300 [vmbusdev] 100120 D - 0xfffff80003b3c100 [vmbussch] 100121 D - 0xfffff8000ffcee00 [hn0 tx0] 100122 D - 0xfffff8000ffcec00 [hn0 mgmt] 100123 D - 0xfffff80010147600 [hn1 tx0] 100124 D - 0xfffff80010147400 [hn1 mgmt] 100125 D - 0xfffff80010145a00 [hn2 tx0] 100126 D - 0xfffff80010145800 [hn2 mgmt] 100183 D - 0xfffff800181abb00 [dummynet] 721 64113 344 0 Z bandwidthd 504 63617 344 0 Z bandwidthd 99861 63659 344 0 Z bandwidthd 99607 64409 344 0 Z bandwidthd 28440 63003 344 0 Z bandwidthd 28360 62542 344 0 Z bandwidthd 89250 63282 344 0 Z bandwidthd 89000 63992 344 0 Z bandwidthd db:0:kdb.enter.default> alltrace Tracing command sleep pid 97756 tid 100234 td 0xfffff8007562b000 sched_switch() at sched_switch+0x8ad/frame 0xfffffe009cc4a760 mi_switch() at mi_switch+0xe6/frame 0xfffffe009cc4a790 sleepq_catch_signals() at sleepq_catch_signals+0x40c/frame 0xfffffe009cc4a7f0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe009cc4a830 _sleep() at _sleep+0x20c/frame 0xfffffe009cc4a8b0 kern_clock_nanosleep() at kern_clock_nanosleep+0x1a7/frame 0xfffffe009cc4a940 sys_nanosleep() at sys_nanosleep+0x5f/frame 0xfffffe009cc4a980 amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe009cc4aab0 fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe009cc4aab0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8009055da, rsp = 0x7fffffffec38, rbp = 0x7fffffffec80 ---