Kernel lockup
-
Hi Jimp
Did you have any thoughts about the one I had and posted which was after the vpn fix.
Thanks
-
jimp where do I look for the backtrace? I will check /var/crash if its same as default FreeBSD behaviour.
right I read this
https://doc.pfsense.org/index.php/Unexpected_Reboot_Troubleshooting
yeah, I did disable the swap partition (it does exist tho), so I will enable it again just in case I get another panic.
-
The firewall should present a box on top of the dashboard with a link to see the crash report if it was able to collect one.
-
edited my post, sorry I edited as you replied.
-
Well it looks like you are the second person having this issue.
I posted a thread earlier and got NO reply.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213856
That is the bug and I have sent multiple crash dumps already.
-
Just because you have a "Fatal Trap 12" doesn't mean it's the same bug. The stack trace doesn't look anywhere near the same.
-
Well it looks like you are the second person having this issue.
I posted a thread earlier and got NO reply.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213856
That is the bug and I have sent multiple crash dumps already.
panic can be caused by many things, this is why I didnt jump in claiming its a bug, as could be hardware or software caused.
My fault I had disabled the swap partition (I had configured swap file instead for SSD trim reasons). So the lack of backtrace is because of that.FreeBSD historically has needed issues resolving in a .0 release, this could be one of those issues, but it could be hardware as well.
Also you posted directly to FreeBSD developers not on pfsense bug report.
-
no more panic's but given the number of people who reported panic's on FreeBSD 11 reporting it been fixed with the igb queue count set to 1 or 2, I adjusted it to 2 for safety. I have enabled MSIX again.
-
no more panic's but given the number of people who reported panic's on FreeBSD 11 reporting it been fixed with the igb queue count set to 1 or 2, I adjusted it to 2 for safety. I have enabled MSIX again.
What "it" been fixed? Is there some ticket on freebsd community regarding this problem?
-
yeah there is a ticket but I dont have the url to hand sorry, everyone who was on the ticket reported panics been fixed with setting it to 1.
Only 1 guy said it was also stable on 2 tho and he posted that on here somewhere. The default is to match core count so in my case default is 4.
hw.igb.num_queues=2
-
I reinstalled pfSense 2.4 2 days ago using zfs, I made sure I have the loader.conf.local in place to reduce igb queues to 1 but I have had a panic late last night, again no dump was made so no backtrace, this time tho a swap partition was enabled.
Will scan my ram.
Will disable adaptive mode for cpu so it stays at one speed only as cpu clock fluctuations can cause instabilities.
Will check bios to see if anything in there that could cause compatibility issues.I do notice on bootup that the boot up log says its not using any dump device, so I will try manually specifying it in rc.conf.local to see if it reports in the boot log one is set.
I wonder if it may be a good idea to track 11-STABLE instead of 11.0 as .0 releases of FreeBSD can be problematic.
-
I wonder if it may be a good idea to track 11-STABLE instead of 11.0 as .0 releases of FreeBSD can be problematic.
The stable branch is a much more moving target than the releng branch, it would be hell a lot more work to keep up with the changes.
-
Have done the 2 ram tests I planned.
Also have the dump device correctly showing during the boot process now so if I have another panic I should get a dump to backtrace.
I have disabled turbo mode and EIST on the cpu and dynamic voltage on the ram.
-
chrcoluk what hardware are you running pfSense
on? Your sig says "Braswell N3150" but that isn't specific. Sorry if I missed it in another post. -
chrcoluk what hardware are you running pfSense
on?That is unacceptable usage. You must say "What hardware are you running pfSense
software on?";D ;D ::) ::)
SCNR. Won't further comment on this new round of legal bleep. I hoped the unfortunate episode ~2+ years ago was enough of a lesson. Apparently not.
-
chrcoluk
Did you add kern.ipc.nmbclusters="1000000"?
What is your WAN bandwidth? -
I dont know the exact hardware packaged as I purchased a compact unit from amazon which is simply labeled as a BRASWELL N3150 unit, which tells you I have a celeron N3150 on a braswell chipset.
I think I am going to adjust the bios again to enable EIST and turbo mode but also configure powerd to never let the clocks drop below the stock max speed that means I get turbo mode back again for extra performance whilst not letting it drop to idle voltages, I will keep the dynamic voltages disabled on the ram.
Also nmbclusters has been increased in the loader.conf from almost day one of using pfsense.
WAN bandwidth at the time of the crash was just idle, the absolute max bandwidth usage possible is about 70mbit down and 20mbit up.
-
Ok, try to bump voltage on memory, +0.1V
-
I dont have that control in the bios.
But I have disabled now dynamic voltage mode (DVFS) so it should always be at full voltage now instead of power saving lower voltages.
-
Am getting issues again, the good news is I am getting proper kernel dumps now, the last two I hit the submit button, but to let more eyes look at the problem I will dump the data here as well, the bit of research I have done seems to indicate its filesystem related, but I am most definitely not sure on this so appreciate an opinion from anyone.
In the meantime since my laptop uses the same spec ram I am going to swap the ram with my laptop memory, even tho this existing ram has passed stability tests.
Crash report begins. Anonymous machine information: amd64 11.0-RELEASE-p8 FreeBSD 11.0-RELEASE-p8 #78 3f54ce3814d(RELENG_2_4): Wed Mar 15 18:23:59 CDT 2017 root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense Crash report details: No PHP errors found. Filename: /var/crash/bounds 1 Filename: /var/crash/info.0 Dump header from device: /dev/ada0p3 Architecture: amd64 Architecture Version: 1 Dump Length: 123392 Blocksize: 512 Dumptime: Wed Jun 7 00:40:31 2017 Hostname: PFSENSE.home Magic: FreeBSD Text Dump Version String: FreeBSD 11.0-RELEASE-p8 #78 3f54ce3814d(RELENG_2_4): Wed Mar 15 18:23:59 CDT 2017 root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense Panic String: Dump Parity: 318326786 Bounds: 0 Dump Status: good Filename: /var/crash/info.last Dump header from device: /dev/ada0p3 Architecture: amd64 Architecture Version: 1 Dump Length: 123392 Blocksize: 512 Dumptime: Wed Jun 7 00:40:31 2017 Hostname: PFSENSE.home Magic: FreeBSD Text Dump Version String: FreeBSD 11.0-RELEASE-p8 #78 3f54ce3814d(RELENG_2_4): Wed Mar 15 18:23:59 CDT 2017 root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense Panic String: Dump Parity: 318326786 Bounds: 0 Dump Status: good Filename: /var/crash/textdump.tar.0 ddb.txt06000014000013115636557 7106 ustarrootwheeldb:0:kdb.enter.default> run lockinfo db:1:lockinfo> show locks No such command db:1:locks> show alllocks No such command db:1:alllocks> show lockedvnods Locked vnodes db:0:kdb.enter.default> show pcpu cpuid = 1 dynamic pcpu = 0xfffffe0167de2000 curthread = 0xfffff8015e162a00: pid 54120 "sh" curpcb = 0xfffffe0113c6dcc0 fpcurthread = 0xfffff8015e162a00: pid 54120 "sh" idlethread = 0xfffff8000490aa00: tid 100004 "idle: cpu1" curpmap = 0xfffff801621b3138 tssp = 0xffffffff82a1dd78 commontssp = 0xffffffff82a1dd78 rsp0 = 0xfffffe0113c6dcc0 gs32p = 0xffffffff82a245d0 ldt = 0xffffffff82a24610 tss = 0xffffffff82a24600 db:0:kdb.enter.default> bt Tracing pid 54120 tid 100774 td 0xfffff8015e162a00 turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0113c6d480 __rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0113c6d4b0 vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0113c6d530 vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0113c6d560 exec_new_vmspace() at exec_new_vmspace+0x22f/frame 0xfffffe0113c6d5e0 exec_elf64_imgact() at exec_elf64_imgact+0xa58/frame 0xfffffe0113c6d6f0 kern_execve() at kern_execve+0x74d/frame 0xfffffe0113c6da50 sys_execve() at sys_execve+0x4a/frame 0xfffffe0113c6dad0 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c6dbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c6dbf0 --- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x800b40d8a, rsp = 0x7fffffffe188, rbp = 0x7fffffffe2d0 --- db:0:kdb.enter.default> ps pid ppid pgrp uid state wmesg wchan cmd 54226 53957 334 0 R CPU 0 sh 54120 53957 334 0 R CPU 1 sh 54035 53957 334 0 R CPU 2 cat 53957 33060 334 0 R CPU 3 sh 39824 334 334 0 S accept 0xfffff8003b01006c php-fpm 43353 32987 43353 0 S+ select 0xfffff801629be7c0 top 32987 2745 32987 0 S+ wait 0xfffff8003b194528 bash 2745 2354 2745 0 Ss+ wait 0xfffff8003b03d528 sh 2354 8180 2354 0 Ss select 0xfffff80102b6a5c0 sshd 21591 1 334 0 S nanslp 0xffffffff82865d21 php 21318 1 21318 0 Ss select 0xfffff80102d507c0 upsd 21063 1 21063 66 Ss select 0xfffff8003b13e6c0 apcupsd-ups 19999 19580 19580 66 S nanslp 0xffffffff82865d21 upsmon 19580 1 19580 0 Ss piperd 0xfffff80162a03be0 upsmon 33060 1 334 0 S piperd 0xfffff8015e2bf8e8 sh 15682 1 15682 136 Ss select 0xfffff8001d948240 dhcpd 88257 1 88257 59 Ss (threaded) unbound 100711 S kqread 0xfffff8015eee6c00 unbound 100760 S kqread 0xfffff80011585800 unbound 100761 S kqread 0xfffff80011585a00 unbound 100762 S kqread 0xfffff800101aea00 unbound 55992 1 55992 0 Ss (threaded) dpinger 100681 S uwait 0xfffff80162823e00 dpinger 100729 S sbwait 0xfffff8016285c4a4 dpinger 100730 S nanslp 0xffffffff82865d20 dpinger 100731 S nanslp 0xffffffff82865d21 dpinger 100732 S accept 0xfffff80162d3d72c dpinger 55571 1 55571 0 Ss (threaded) dpinger 100701 S uwait 0xfffff8001d4c3b80 dpinger 100725 S sbwait 0xfffff80162a2f144 dpinger 100726 S nanslp 0xffffffff82865d22 dpinger 100727 S nanslp 0xffffffff82865d20 dpinger 100728 S accept 0xfffff801620e2a8c dpinger 89259 1 89259 0 Ss select 0xfffff8001d4c1e40 openvpn 87846 1 87846 0 Ss select 0xfffff8016220e7c0 openvpn 68550 1 68550 0 Ss select 0xfffff80102d520c0 powerd 67165 67155 67165 0 S+ ttyin 0xfffff8001d3dfca8 sh 67155 65512 67155 0 S+ wait 0xfffff8016212f528 sh 66898 98897 66898 0 Ss (threaded) sshlockout_pf 100605 S piperd 0xfffff8001d9db000 sshlockout_pf 100683 S nanslp 0xffffffff82865d20 sshlockout_pf 66742 1 66742 0 Ss+ ttyin 0xfffff8001d46b0a8 getty 66646 1 66646 0 Ss+ ttyin 0xfffff8001d46b4a8 getty 66575 1 66575 0 Ss+ ttyin 0xfffff8001d46b8a8 getty 66538 1 66538 0 Ss+ ttyin 0xfffff8001d46bca8 getty 66211 1 66211 0 Ss+ ttyin 0xfffff8001d46c0a8 getty 66158 1 66158 0 Ss+ ttyin 0xfffff80005179ca8 getty 65836 1 65836 0 Ss+ ttyin 0xfffff8001d3df0a8 getty 65512 1 65512 0 Ss+ wait 0xfffff8001d976a50 login 50681 50482 50482 0 S nanslp 0xffffffff82865d20 minicron 50482 1 50482 0 Ss wait 0xfffff8001df18a50 minicron 50272 49938 49938 0 S nanslp 0xffffffff82865d23 minicron 49938 1 49938 0 Ss wait 0xfffff8001d976528 minicron 49476 49314 49314 0 S nanslp 0xffffffff82865d23 minicron 49314 1 49314 0 Ss wait 0xfffff8016212aa50 minicron 44216 1 44216 0 Ss (threaded) apcupsd 100658 S nanslp 0xffffffff82865d21 apcupsd 100670 S sigwait 0xfffff80162a01000 apcupsd 100671 S accept 0xfffff8016212206c apcupsd 3948 1 3848 0 S kqread 0xfffff8001dc25200 lighttpd_pfb 98897 1 98897 0 Ss select 0xfffff80102d509c0 syslogd 53013 1 53013 0 Ss kqread 0xfffff8001dbe1600 dnscrypt-proxy 52232 1 52232 0 Ss kqread 0xfffff8001db85000 dnscrypt-proxy 50361 1 50361 0 Ss kqread 0xfffff8001df2d900 dnscrypt-proxy 47847 1 47764 0 S select 0xfffff8003b3464c0 radvd 40799 1 40799 0 Ss (threaded) ntpd 100646 S select 0xfffff8003b1408c0 ntpd 40296 0 0 0 DL mdwait 0xfffff80102bf3000 [md3] 38103 1 38103 0 Ss nanslp 0xffffffff82865d22 cron 37771 37268 37268 0 S kqread 0xfffff8001dc40600 nginx 37735 37268 37268 0 S kqread 0xfffff8001d5b0700 nginx 37705 37268 37268 0 S kqread 0xfffff8001dc1c900 nginx 37554 37268 37268 0 S kqread 0xfffff8001dbbaa00 nginx 37494 37268 37268 0 S kqread 0xfffff8001db43a00 nginx 37268 1 37268 0 Ss pause 0xfffff80102eecaf8 nginx 20138 1 20138 0 Ss bpf 0xfffff8003bd67000 filterlog 15349 1 15349 0 Ss select 0xfffff8003b13e1c0 dhcp6c 14626 1 14626 65 Ss select 0xfffff8001dc38ec0 dhclient 9309 1 9309 0 Ss select 0xfffff8003b3460c0 dhclient 8519 1 8519 0 Ss (threaded) sshlockout_pf 100597 S uwait 0xfffff8001dc37600 sshlockout_pf 100601 S nanslp 0xffffffff82865d20 sshlockout_pf 8180 1 8180 0 Ss select 0xfffff8001d4c2640 sshd 393 0 0 0 DL mdwait 0xfffff8001daf1800 [md2] 383 0 0 0 DL mdwait 0xfffff8003b132000 [md1] 368 1 368 0 Ss select 0xfffff8003b1411c0 devd 350 348 348 0 S kqread 0xfffff8001dcabd00 check_reload_status 348 1 348 0 Ss kqread 0xfffff8001db94800 check_reload_status 334 1 334 0 Ss kqread 0xfffff8001ddd2100 php-fpm 103 0 0 0 DL mdwait 0xfffff8001d565000 [md0] 27 0 0 0 DL syncer 0xffffffff8295f880 [syncer] 26 0 0 0 DL vlruwt 0xfffff8001d406528 [vnlru] 25 0 0 0 DL psleep 0xffffffff8295e104 [bufdaemon] 24 0 0 0 DL - 0xffffffff8295edb4 [bufspacedaemon] 23 0 0 0 DL pgzero 0xffffffff829734e4 [pagezero] 22 0 0 0 DL psleep 0xffffffff8296f98c [vmdaemon] 21 0 0 0 DL (threaded) [pagedaemon] 100072 D psleep 0xffffffff82a1d105 [pagedaemon] 100081 D umarcl 0xffffffff8296f2b8 [uma] 20 0 0 0 DL - 0xffffffff8295d914 [soaiod4] 19 0 0 0 DL - 0xffffffff8295d914 [soaiod3] 18 0 0 0 DL - 0xffffffff8295d914 [soaiod2] 9 0 0 0 DL - 0xffffffff8295d914 [soaiod1] 17 0 0 0 DL cooling 0xfffff80005139f58 [acpi_cooling0] 16 0 0 0 DL tzpoll 0xffffffff82621138 [acpi_thermal] 8 0 0 0 DL - 0xffffffff8273a4f0 [rand_harvestq] 7 0 0 0 DL pftm 0xffffffff80f4b6a0 [pf purge] 6 0 0 0 DL waiting_ 0xffffffff82a0c740 [sctp_iterator] 5 0 0 0 DL (threaded) [zfskern] 100055 D arc_recl 0xffffffff82f3f4d0 [arc_reclaim_thread] 100056 D arc_user 0xffffffff82f3f4f0 [arc_user_evicts_thr] 100057 D arc_dnlc 0xffffffff82f3f480 [arc_dnlc_evicts_thr] 100058 D l2arc_fe 0xffffffff82f3f888 [l2arc_feed_thread] 100550 D spa->spa 0xfffff8001d55fbd0 [trim zroot] 100564 D tx->tx_q 0xfffff8001d45fa30 [txg_thread_enter] 100565 D tx->tx_s 0xfffff8001d45fa10 [txg_thread_enter] 15 0 0 0 DL (threaded) [usb] 100035 D - 0xfffffe0001121460 [usbus0] 100036 D - 0xfffffe00011214b8 [usbus0] 100037 D - 0xfffffe0001121510 [usbus0] 100038 D - 0xfffffe0001121568 [usbus0] 100039 D - 0xfffffe00011215c0 [usbus0] 4 0 0 0 DL (threaded) [cam] 100032 D - 0xffffffff8260d380 [doneq0] 100067 D - 0xffffffff8260d1c8 [scanner] 3 0 0 0 DL crypto_r 0xffffffff8296de70 [crypto returns] 2 0 0 0 DL crypto_w 0xffffffff8296dd18 [crypto] 14 0 0 0 DL (threaded) [geom] 100023 D - 0xffffffff829e4940 [g_event] 100024 D - 0xffffffff829e4948 [g_up] 100025 D - 0xffffffff829e4950 [g_down] 13 0 0 0 DL (threaded) [ng_queue] 100019 D sleep 0xffffffff825cb230 [ng_queue0] 100020 D sleep 0xffffffff825cb230 [ng_queue1] 100021 D sleep 0xffffffff825cb230 [ng_queue2] 100022 D sleep 0xffffffff825cb230 [ng_queue3] 12 0 0 0 WL (threaded) [intr] 100007 I [swi3: vm] 100008 I [swi4: clock (0)] 100009 I [swi4: clock (1)] 100010 I [swi4: clock (2)] 100011 I [swi4: clock (3)] 100012 I [swi1: netisr 0] 100013 I [swi5: fast taskq] 100015 I [swi6: task queue] 100016 I [swi6: Giant taskq] 100033 I [irq256: ahci0] 100034 I [irq257: xhci0] 100040 I [irq258: re0] 100041 I [irq259: pcib2] 100042 I [irq260: re1] 100043 I [irq261: igb0:que 0] 100045 I [irq262: igb0:link] 100046 I [irq263: igb1:que 0] 100048 I [irq264: igb1:link] 100061 I [swi1: pf send] 100062 I [swi1: pfsync] 100079 I [swi1: netisr 1] 100080 I [swi1: netisr 2] 100082 I [swi1: netisr 3] 11 0 0 0 RL (threaded) [idle] 100003 CanRun [idle: cpu0] 100004 CanRun [idle: cpu1] 100005 CanRun [idle: cpu2] 100006 CanRun [idle: cpu3] 1 0 1 0 SLs wait 0xfffff80004908528 [init] 10 0 0 0 DL audit_wo 0xffffffff82a138c0 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0xffffffff829e4978 [swapper] 100014 D - 0xfffff8000491ee00 [kqueue_ctx taskq] 100017 D - 0xfffff8000491e600 [aiod_kick taskq] 100018 D - 0xfffff8000491e400 [thread] 100026 D - 0xfffff8000491d600 [firmware taskq] 100029 D - 0xfffff80004ae4d00 [acpi_task_0] 100030 D - 0xfffff80004ae4d00 [acpi_task_1] 100031 D - 0xfffff80004ae4d00 [acpi_task_2] 100044 D - 0xfffff80005106b00 [igb0 que] 100047 D - 0xfffff80004ba4d00 [igb1 que] 100049 D - 0xfffff80005162800 [system_taskq_0] 100050 D - 0xfffff80005162800 [system_taskq_1] 100051 D - 0xfffff80005162800 [system_taskq_2] 100052 D - 0xfffff80005162800 [system_taskq_3] 100053 D - 0xfffff80004ae5000 [mca taskq] 100054 D - 0xfffff80005161b00 [dbu_evict] 100066 D - 0xfffff80004ae4900 [CAM taskq] 100083 D - 0xfffff8001d467300 [if_io_tqg_0] 100084 D - 0xfffff8001d467100 [if_io_tqg_1] 100085 D - 0xfffff8001d4a4b00 [if_io_tqg_2] 100086 D - 0xfffff8001d4a4900 [if_io_tqg_3] 100087 D - 0xfffff8001d4a4700 [if_config_tqg_0] 100322 D - 0xfffff8001d5fed00 [zio_null_issue] 100323 D - 0xfffff8001d5b2600 [zio_null_intr] 100324 D - 0xfffff8001d5b2300 [zio_read_issue_0] 100325 D - 0xfffff8001d5b2300 [zio_read_issue_1] 100326 D - 0xfffff8001d5b2300 [zio_read_issue_2] 100327 D - 0xfffff8001d5b2300 [zio_read_issue_3] 100328 D - 0xfffff8001d5b2300 [zio_read_issue_4] 100329 D - 0xfffff8001d5b2300 [zio_read_issue_5] 100330 D - 0xfffff8001d5b2300 [zio_read_issue_6] 100331 D - 0xfffff8001d5b2300 [zio_read_issue_7] 100332 D - 0xfffff8001d5ff700 [zio_read_intr_0_0] 100333 D - 0xfffff8001d5ff700 [zio_read_intr_0_1] 100334 D - 0xfffff8001d5ff700 [zio_read_intr_0_2] 100335 D - 0xfffff8001d5ff700 [zio_read_intr_0_3] 100336 D - 0xfffff8001d5ff700 [zio_read_intr_0_4] 100337 D - 0xfffff8001d5ff700 [zio_read_intr_0_5] 100338 D - 0xfffff8001d5ff700 [zio_read_intr_0_6] 100339 D - 0xfffff8001d5ff700 [zio_read_intr_0_7] 100340 D - 0xfffff8001d5ff700 [zio_read_intr_0_8] 100341 D - 0xfffff8001d5ff700 [zio_read_intr_0_9] 100342 D - 0xfffff8001d5ff700 [zio_read_intr_0_10] 100343 D - 0xfffff8001d5ff700 [zio_read_intr_0_11] 100344 D - 0xfffff8001d5b2200 [zio_read_intr_1_0] 100345 D - 0xfffff8001d5b2200 [zio_read_intr_1_1] 100346 D - 0xfffff8001d5b2200 [zio_read_intr_1_2] 100347 D - 0xfffff8001d5b2200 [zio_read_intr_1_3] 100348 D - 0xfffff8001d5b2200 [zio_read_intr_1_4] 100349 D - 0xfffff8001d5b2200 [zio_read_intr_1_5] 100350 D - 0xfffff8001d5b2200 [zio_read_intr_1_6] 100351 D - 0xfffff8001d5b2200 [zio_read_intr_1_7] 100352 D - 0xfffff8001d5b2200 [zio_read_intr_1_8] 100353 D - 0xfffff8001d5b2200 [zio_read_intr_1_9] 100354 D - 0xfffff8001d5b2200 [zio_read_intr_1_10] 100355 D - 0xfffff8001d5b2200 [zio_read_intr_1_11] 100356 D - 0xfffff8001d5fda00 [zio_read_intr_2_0] 100357 D - 0xfffff8001d5fda00 [zio_read_intr_2_1] 100358 D - 0xfffff8001d5fda00 [zio_read_intr_2_2] 100359 D - 0xfffff8001d5fda00 [zio_read_intr_2_3] 100360 D - 0xfffff8001d5fda00 [zio_read_intr_2_4] 100361 D - 0xfffff8001d5fda00 [zio_read_intr_2_5] 100362 D - 0xfffff8001d5fda00 [zio_read_intr_2_6] 100363 D - 0xfffff8001d5fda00 [zio_read_intr_2_7] 100364 D - 0xfffff8001d5fda00 [zio_read_intr_2_8] 100365 D - 0xfffff8001d5fda00 [zio_read_intr_2_9] 100366 D - 0xfffff8001d5fda00 [zio_read_intr_2_10] 100367 D - 0xfffff8001d5fda00 [zio_read_intr_2_11] 100368 D - 0xfffff8001d5ddc00 [zio_read_intr_3_0] 100369 D - 0xfffff8001d5ddc00 [zio_read_intr_3_1] 100370 D - 0xfffff8001d5ddc00 [zio_read_intr_3_2] 100371 D - 0xfffff8001d5ddc00 [zio_read_intr_3_3] 100372 D - 0xfffff8001d5ddc00 [zio_read_intr_3_4] 100373 D - 0xfffff8001d5ddc00 [zio_read_intr_3_5] 100374 D - 0xfffff8001d5ddc00 [zio_read_intr_3_6] 100375 D - 0xfffff8001d5ddc00 [zio_read_intr_3_7] 100376 D - 0xfffff8001d5ddc00 [zio_read_intr_3_8] 100377 D - 0xfffff8001d5ddc00 [zio_read_intr_3_9] 100378 D - 0xfffff8001d5ddc00 [zio_read_intr_3_10] 100379 D - 0xfffff8001d5ddc00 [zio_read_intr_3_11] 100380 D - 0xfffff8001d552500 [zio_read_intr_4_0] 100381 D - 0xfffff8001d552500 [zio_read_intr_4_1] 100382 D - 0xfffff8001d552500 [zio_read_intr_4_2] 100383 D - 0xfffff8001d552500 [zio_read_intr_4_3] 100384 D - 0xfffff8001d552500 [zio_read_intr_4_4] 100385 D - 0xfffff8001d552500 [zio_read_intr_4_5] 100386 D - 0xfffff8001d552500 [zio_read_intr_4_6] 100387 D - 0xfffff8001d552500 [zio_read_intr_4_7] 100388 D - 0xfffff8001d552500 [zio_read_intr_4_8] 100389 D - 0xfffff8001d552500 [zio_read_intr_4_9] 100390 D - 0xfffff8001d552500 [zio_read_intr_4_10] 100391 D - 0xfffff8001d552500 [zio_read_intr_4_11] 100392 D - 0xfffff8001d4a2800 [zio_read_intr_5_0] 100393 D - 0xfffff8001d4a2800 [zio_read_intr_5_1] 100394 D - 0xfffff8001d4a2800 [zio_read_intr_5_2] 100395 D - 0xfffff8001d4a2800 [zio_read_intr_5_3] 100396 D - 0xfffff8001d4a2800 [zio_read_intr_5_4] 100397 D - 0xfffff8001d4a2800 [zio_read_intr_5_5] 100398 D - 0xfffff8001d4a2800 [zio_read_intr_5_6] 100399 D - 0xfffff8001d4a2800 [zio_read_intr_5_7] 100400 D - 0xfffff8001d4a2800 [zio_read_intr_5_8] 100401 D - 0xfffff8001d4a2800 [zio_read_intr_5_9] 100402 D - 0xfffff8001d4a2800 [zio_read_intr_5_10] 100403 D - 0xfffff8001d4a2800 [zio_read_intr_5_11] 100404 D - 0xfffff8001d4a2300 [zio_read_intr_6_0] 100405 D - 0xfffff8001d4a2300 [zio_read_intr_6_1] 100406 D - 0xfffff8001d4a2300 [zio_read_intr_6_2] 100407 D - 0xfffff8001d4a2300 [zio_read_intr_6_3] 100408 D - 0xfffff8001d4a2300 [zio_read_intr_6_4] 100409 D - 0xfffff8001d4a2300 [zio_read_intr_6_5] 100410 D - 0xfffff8001d4a2300 [zio_read_intr_6_6] 100411 D - 0xfffff8001d4a2300 [zio_read_intr_6_7] 100412 D - 0xfffff8001d4a2300 [zio_read_intr_6_8] 100413 D - 0xfffff8001d4a2300 [zio_read_intr_6_9] 100414 D - 0xfffff8001d4a2300 [zio_read_intr_6_10] 100415 D - 0xfffff8001d4a2300 [zio_read_intr_6_11] 100416 D - 0xfffff8001d4a1000 [zio_read_intr_7_0] 100417 D - 0xfffff8001d4a1000 [zio_read_intr_7_1] 100418 D - 0xfffff8001d4a1000 [zio_read_intr_7_2] 100419 D - 0xfffff8001d4a1000 [zio_read_intr_7_3] 100420 D - 0xfffff8001d4a1000 [zio_read_intr_7_4] 100421 D - 0xfffff8001d4a1000 [zio_read_intr_7_5] 100422 D - 0xfffff8001d4a1000 [zio_read_intr_7_6] 100423 D - 0xfffff8001d4a1000 [zio_read_intr_7_7] 100424 D - 0xfffff8001d4a1000 [zio_read_intr_7_8] 100425 D - 0xfffff8001d4a1000 [zio_read_intr_7_9] 100426 D - 0xfffff8001d4a1000 [zio_read_intr_7_10] 100427 D - 0xfffff8001d4a1000 [zio_read_intr_7_11] 100428 D - 0xfffff8001d589300 [zio_write_issue_0] 100429 D - 0xfffff8001d589300 [zio_write_issue_1] 100430 D - 0xfffff8001d589300 [zio_write_issue_2] 100431 D - 0xfffff8001d589200 [zio_write_issue_hig] 100432 D - 0xfffff8001d589200 [zio_write_issue_hig] 100433 D - 0xfffff8001d589200 [zio_write_issue_hig] 100434 D - 0xfffff8001d589200 [zio_write_issue_hig] 100435 D - 0xfffff8001d589200 [zio_write_issue_hig] 100436 D - 0xfffff8001d589000 [zio_write_intr_0] 100437 D - 0xfffff8001d589000 [zio_write_intr_1] 100438 D - 0xfffff8001d589000 [zio_write_intr_2] 100439 D - 0xfffff8001d589000 [zio_write_intr_3] 100440 D - 0xfffff8001d589000 [zio_write_intr_4] 100441 D - 0xfffff8001d589000 [zio_write_intr_5] 100442 D - 0xfffff8001d589000 [zio_write_intr_6] 100443 D - 0xfffff8001d589000 [zio_write_intr_7] 100444 D - 0xfffff8001d588100 [zio_write_intr_high] 100445 D - 0xfffff8001d588100 [zio_write_intr_high] 100446 D - 0xfffff8001d588100 [zio_write_intr_high] 100447 D - 0xfffff8001d588100 [zio_write_intr_high] 100448 D - 0xfffff8001d588100 [zio_write_intr_high] 100449 D - 0xfffff8001d587800 [zio_free_issue_0_0] 100450 D - 0xfffff8001d587800 [zio_free_issue_0_1] 100451 D - 0xfffff8001d587800 [zio_free_issue_0_2] 100452 D - 0xfffff8001d587800 [zio_free_issue_0_3] 100453 D - 0xfffff8001d587800 [zio_free_issue_0_4] 100454 D - 0xfffff8001d587800 [zio_free_issue_0_5] 100455 D - 0xfffff8001d587800 [zio_free_issue_0_6] 100456 D - 0xfffff8001d587800 [zio_free_issue_0_7] 100457 D - 0xfffff8001d587800 [zio_free_issue_0_8] 100458 D - 0xfffff8001d587800 [zio_free_issue_0_9] 100459 D - 0xfffff8001d587800 [zio_free_issue_0_10] 100460 D - 0xfffff8001d587800 [zio_free_issue_0_11] 100461 D - 0xfffff8001d587100 [zio_free_issue_1_0] 100462 D - 0xfffff8001d587100 [zio_free_issue_1_1] 100463 D - 0xfffff8001d587100 [zio_free_issue_1_2] 100464 D - 0xfffff8001d587100 [zio_free_issue_1_3] 100465 D - 0xfffff8001d587100 [zio_free_issue_1_4] 100466 D - 0xfffff8001d587100 [zio_free_issue_1_5] 100467 D - 0xfffff8001d587100 [zio_free_issue_1_6] 100468 D - 0xfffff8001d587100 [zio_free_issue_1_7] 100469 D - 0xfffff8001d587100 [zio_free_issue_1_8] 100470 D - 0xfffff8001d587100 [zio_free_issue_1_9] 100471 D - 0xfffff8001d587100 [zio_free_issue_1_10] 100472 D - 0xfffff8001d587100 [zio_free_issue_1_11] 100473 D - 0xfffff8001d5b1d00 [zio_free_issue_2_0] 100474 D - 0xfffff8001d5b1d00 [zio_free_issue_2_1] 100475 D - 0xfffff8001d5b1d00 [zio_free_issue_2_2] 100476 D - 0xfffff8001d5b1d00 [zio_free_issue_2_3] 100477 D - 0xfffff8001d5b1d00 [zio_free_issue_2_4] 100478 D - 0xfffff8001d5b1d00 [zio_free_issue_2_5] 100479 D - 0xfffff8001d5b1d00 [zio_free_issue_2_6] 100480 D - 0xfffff8001d5b1d00 [zio_free_issue_2_7] 100481 D - 0xfffff8001d5b1d00 [zio_free_issue_2_8] 100482 D - 0xfffff8001d5b1d00 [zio_free_issue_2_9] 100483 D - 0xfffff8001d5b1d00 [zio_free_issue_2_10] 100484 D - 0xfffff8001d5b1d00 [zio_free_issue_2_11] 100485 D - 0xfffff8001d5b1000 [zio_free_issue_3_0] 100486 D - 0xfffff8001d5b1000 [zio_free_issue_3_1] 100487 D - 0xfffff8001d5b1000 [zio_free_issue_3_2] 100488 D - 0xfffff8001d5b1000 [zio_free_issue_3_3] 100489 D - 0xfffff8001d5b1000 [zio_free_issue_3_4] 100490 D - 0xfffff8001d5b1000 [zio_free_issue_3_5] 100491 D - 0xfffff8001d5b1000 [zio_free_issue_3_6] 100492 D - 0xfffff8001d5b1000 [zio_free_issue_3_7] 100493 D - 0xfffff8001d5b1000 [zio_free_issue_3_8] 100494 D - 0xfffff8001d5b1000 [zio_free_issue_3_9] 100495 D - 0xfffff8001d5b1000 [zio_free_issue_3_10] 100496 D - 0xfffff8001d5b1000 [zio_free_issue_3_11] 100497 D - 0xfffff8001d5afb00 [zio_free_issue_4_0] 100498 D - 0xfffff8001d5afb00 [zio_free_issue_4_1] 100499 D - 0xfffff8001d5afb00 [zio_free_issue_4_2] 100500 D - 0xfffff8001d5afb00 [zio_free_issue_4_3] 100501 D - 0xfffff8001d5afb00 [zio_free_issue_4_4] 100502 D - 0xfffff8001d5afb00 [zio_free_issue_4_5] 100503 D - 0xfffff8001d5afb00 [zio_free_issue_4_6] 100504 D - 0xfffff8001d5afb00 [zio_free_issue_4_7] 100505 D - 0xfffff8001d5afb00 [zio_free_issue_4_8] 100506 D - 0xfffff8001d5afb00 [zio_free_issue_4_9] 100507 D - 0xfffff8001d5afb00 [zio_free_issue_4_10] 100508 D - 0xfffff8001d5afb00 [zio_free_issue_4_11] 100509 D - 0xfffff8001d58ab00 [zio_free_issue_5_0] 100510 D - 0xfffff8001d58ab00 [zio_free_issue_5_1] 100511 D - 0xfffff8001d58ab00 [zio_free_issue_5_2] 100512 D - 0xfffff8001d58ab00 [zio_free_issue_5_3] 100513 D - 0xfffff8001d58ab00 [zio_free_issue_5_4] 100514 D - 0xfffff8001d58ab00 [zio_free_issue_5_5] 100515 D - 0xfffff8001d58ab00 [zio_free_issue_5_6] [/thread]
