Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    Kernel lockup

    2.4 Development Snapshots
    8
    25
    3815
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      chrcoluk last edited by

      Had a kernel lockup this morning, extract from logs below (note log is set to have newest line on top)

      Jan 4 10:46:17	kernel		Fatal trap 12: page fault while in kernel mode
      Jan 4 10:46:17	kernel		kernel trap 12 with interrupts disabled
      Jan 4 10:46:17	kernel		current process	= 97781 (sh)
      Jan 4 10:46:17	kernel		processor eflags	= resume, IOPL = 0
      Jan 4 10:46:17	kernel		= DPL 0, pres 1, long 1, def32 0, gran 1
      Jan 4 10:46:17	kernel		code segment	= base 0x0, limit 0xfffff, type 0x1b
      Jan 4 10:46:17	kernel		frame pointer	= 0x28:0xfffffe01181b2480
      Jan 4 10:46:17	kernel		stack pointer	= 0x28:0xfffffe01181b2450
      Jan 4 10:46:17	kernel		instruction pointer	= 0x20:0xffffffff80cb908c
      Jan 4 10:46:17	kernel		fault code	= supervisor read data, page not present
      Jan 4 10:46:17	kernel		fault virtual address	= 0x30
      Jan 4 10:46:17	kernel		cpuid = 0; apic id = 00
      

      Had no lockups on 2.2 but I also havent added load to the box until the past week.

      I undid some tunables I set for the nic in loader.conf.custom, and disabled MSIX.  Will see if it stabilises, if I get more issues, I will test the ram.

      If needed I can swap the ram as its the same exact sodimm that I had put in my laptop last year, so I can remove from laptop to put in the pfsense unit if I find the ram faulty.

      pfSense 2.6.0 - ISP AAISP UK

      1 Reply Last reply Reply Quote 0
      • jimp
        jimp Rebel Alliance Developer Netgate last edited by

        Was there no other data from the crash? No crash report or backtrace?

        There is not enough information in those log messages to make any guesses about a cause.

        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

        Need help fast? Netgate Global Support!

        Do not Chat/PM for help!

        1 Reply Last reply Reply Quote 0
        • S
          skyeci last edited by

          Hi Jimp

          Did you have any thoughts about the one I had and posted which was after the vpn fix.

          Thanks

          1 Reply Last reply Reply Quote 0
          • C
            chrcoluk last edited by

            jimp where do I look for the backtrace? I will check /var/crash if its same as default FreeBSD behaviour.

            right I read this

            https://doc.pfsense.org/index.php/Unexpected_Reboot_Troubleshooting

            yeah, I did disable the swap partition (it does exist tho), so I will enable it again just in case I get another panic.

            pfSense 2.6.0 - ISP AAISP UK

            1 Reply Last reply Reply Quote 0
            • jimp
              jimp Rebel Alliance Developer Netgate last edited by

              The firewall should present a box on top of the dashboard with a link to see the crash report if it was able to collect one.

              Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

              Need help fast? Netgate Global Support!

              Do not Chat/PM for help!

              1 Reply Last reply Reply Quote 0
              • C
                chrcoluk last edited by

                edited my post, sorry I edited as you replied.

                pfSense 2.6.0 - ISP AAISP UK

                1 Reply Last reply Reply Quote 0
                • D
                  Dazog last edited by

                  Well it looks like you are the second person having this issue.

                  I posted a thread earlier and got NO reply.

                  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213856

                  That is the bug and I have sent multiple crash dumps already.

                  1 Reply Last reply Reply Quote 0
                  • jimp
                    jimp Rebel Alliance Developer Netgate last edited by

                    Just because you have a "Fatal Trap 12" doesn't mean it's the same bug. The stack trace doesn't look anywhere near the same.

                    Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                    Need help fast? Netgate Global Support!

                    Do not Chat/PM for help!

                    1 Reply Last reply Reply Quote 0
                    • C
                      chrcoluk last edited by

                      @Dazog:

                      Well it looks like you are the second person having this issue.

                      I posted a thread earlier and got NO reply.

                      https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213856

                      That is the bug and I have sent multiple crash dumps already.

                      panic can be caused by many things, this is why I didnt jump in claiming its a bug, as could be hardware or software caused.
                      My fault I had disabled the swap partition (I had configured swap file instead for SSD trim reasons).  So the lack of backtrace is because of that.

                      FreeBSD historically has needed issues resolving in a .0 release, this could be one of those issues, but it could be hardware as well.

                      Also you posted directly to FreeBSD developers not on pfsense bug report.

                      pfSense 2.6.0 - ISP AAISP UK

                      1 Reply Last reply Reply Quote 0
                      • C
                        chrcoluk last edited by

                        no more panic's but given the number of people who reported panic's on FreeBSD 11 reporting it been fixed with the igb queue count set to 1 or 2, I adjusted it to 2 for safety.  I have enabled MSIX again.

                        pfSense 2.6.0 - ISP AAISP UK

                        1 Reply Last reply Reply Quote 0
                        • w0w
                          w0w last edited by

                          @chrcoluk:

                          no more panic's but given the number of people who reported panic's on FreeBSD 11 reporting it been fixed with the igb queue count set to 1 or 2, I adjusted it to 2 for safety.  I have enabled MSIX again.

                          What "it" been fixed? Is there some ticket on freebsd community regarding this problem?

                          1 Reply Last reply Reply Quote 0
                          • C
                            chrcoluk last edited by

                            yeah there is a ticket but I dont have the url to hand sorry, everyone who was on the ticket reported panics been fixed with setting it to 1.

                            Only 1 guy said it was also stable on 2 tho and he posted that on here somewhere.  The default is to match core count so in my case default is 4.

                            hw.igb.num_queues=2

                            pfSense 2.6.0 - ISP AAISP UK

                            1 Reply Last reply Reply Quote 0
                            • C
                              chrcoluk last edited by

                              I reinstalled pfSense 2.4 2 days ago using zfs, I made sure I have the loader.conf.local in place to reduce igb queues to 1 but I have had a panic late last night, again no dump was made so no backtrace, this time tho a swap partition was enabled.

                              Will scan my ram.
                              Will disable adaptive mode for cpu so it stays at one speed only as cpu clock fluctuations can cause instabilities.
                              Will check bios to see if anything in there that could cause compatibility issues.

                              I do notice on bootup that the boot up log says its not using any dump device, so I will try manually specifying it in rc.conf.local to see if it reports in the boot log one is set.

                              I wonder if it may be a good idea to track 11-STABLE instead of 11.0 as .0 releases of FreeBSD can be problematic.

                              pfSense 2.6.0 - ISP AAISP UK

                              1 Reply Last reply Reply Quote 0
                              • K
                                kpa last edited by

                                @chrcoluk:

                                I wonder if it may be a good idea to track 11-STABLE instead of 11.0 as .0 releases of FreeBSD can be problematic.

                                The stable branch is a much more moving target than the releng branch, it would be hell a lot more work to keep up with the changes.

                                1 Reply Last reply Reply Quote 0
                                • C
                                  chrcoluk last edited by

                                  Have done the 2 ram tests I planned.

                                  Also have the dump device correctly showing during the boot process now so if I have another panic I should get a dump to backtrace.

                                  I have disabled turbo mode and EIST on the cpu and dynamic voltage on the ram.

                                  pfSense 2.6.0 - ISP AAISP UK

                                  1 Reply Last reply Reply Quote 0
                                  • luckman212
                                    luckman212 LAYER 8 last edited by

                                    chrcoluk what hardware are you running pfSense® on?  Your sig says "Braswell N3150" but that isn't specific.  Sorry if I missed it in another post.

                                    1 Reply Last reply Reply Quote 0
                                    • D
                                      doktornotor Banned last edited by

                                      @luckman212:

                                      chrcoluk what hardware are you running pfSense® on?

                                      That is unacceptable usage. You must say "What hardware are you running pfSense® software on?"

                                      ;D ;D ::) ::)

                                      SCNR. Won't further comment on this new round of legal bleep. I hoped the unfortunate episode ~2+ years ago was enough of a lesson. Apparently not.

                                      1 Reply Last reply Reply Quote 0
                                      • w0w
                                        w0w last edited by

                                        chrcoluk
                                        Did you add kern.ipc.nmbclusters="1000000"?
                                        What is your WAN bandwidth?

                                        1 Reply Last reply Reply Quote 0
                                        • C
                                          chrcoluk last edited by

                                          I dont know the exact hardware packaged as I purchased a compact unit from amazon which is simply labeled as a BRASWELL N3150 unit, which tells you I have a celeron N3150 on a braswell chipset.

                                          I think I am going to adjust the bios again to enable EIST and turbo mode but also configure powerd to never let the clocks drop below the stock max speed that means I get turbo mode back again for extra performance whilst not letting it drop to idle voltages, I will keep the dynamic voltages disabled on the ram.

                                          Also nmbclusters has been increased in the loader.conf from almost day one of using pfsense.

                                          WAN bandwidth at the time of the crash was just idle, the absolute max bandwidth usage possible is about 70mbit down and 20mbit up.

                                          pfSense 2.6.0 - ISP AAISP UK

                                          1 Reply Last reply Reply Quote 0
                                          • w0w
                                            w0w last edited by

                                            Ok, try to bump voltage on memory, +0.1V

                                            1 Reply Last reply Reply Quote 0
                                            • C
                                              chrcoluk last edited by

                                              I dont have that control in the bios.

                                              But I have disabled now dynamic voltage mode (DVFS) so it should always be at full voltage now instead of power saving lower voltages.

                                              pfSense 2.6.0 - ISP AAISP UK

                                              1 Reply Last reply Reply Quote 0
                                              • C
                                                chrcoluk last edited by

                                                Am getting issues again, the good news is I am getting proper kernel dumps now, the last two I hit the submit button, but to let more eyes look at the problem I will dump the data here as well, the bit of research I have done seems to indicate its filesystem related, but I am most definitely not sure on this so appreciate an opinion from anyone.

                                                In the meantime since my laptop uses the same spec ram I am going to swap the ram with my laptop memory, even tho this existing ram has passed stability tests.

                                                					Crash report begins.  Anonymous machine information:
                                                
                                                amd64
                                                11.0-RELEASE-p8
                                                FreeBSD 11.0-RELEASE-p8 #78 3f54ce3814d(RELENG_2_4): Wed Mar 15 18:23:59 CDT 2017     root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense
                                                
                                                Crash report details:
                                                
                                                No PHP errors found.
                                                
                                                Filename: /var/crash/bounds
                                                1
                                                
                                                Filename: /var/crash/info.0
                                                Dump header from device: /dev/ada0p3
                                                  Architecture: amd64
                                                  Architecture Version: 1
                                                  Dump Length: 123392
                                                  Blocksize: 512
                                                  Dumptime: Wed Jun  7 00:40:31 2017
                                                  Hostname: PFSENSE.home
                                                  Magic: FreeBSD Text Dump
                                                  Version String: FreeBSD 11.0-RELEASE-p8 #78 3f54ce3814d(RELENG_2_4): Wed Mar 15 18:23:59 CDT 2017
                                                    root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense
                                                  Panic String: 
                                                  Dump Parity: 318326786
                                                  Bounds: 0
                                                  Dump Status: good
                                                
                                                Filename: /var/crash/info.last
                                                Dump header from device: /dev/ada0p3
                                                  Architecture: amd64
                                                  Architecture Version: 1
                                                  Dump Length: 123392
                                                  Blocksize: 512
                                                  Dumptime: Wed Jun  7 00:40:31 2017
                                                  Hostname: PFSENSE.home
                                                  Magic: FreeBSD Text Dump
                                                  Version String: FreeBSD 11.0-RELEASE-p8 #78 3f54ce3814d(RELENG_2_4): Wed Mar 15 18:23:59 CDT 2017
                                                    root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense
                                                  Panic String: 
                                                  Dump Parity: 318326786
                                                  Bounds: 0
                                                  Dump Status: good
                                                
                                                Filename: /var/crash/textdump.tar.0
                                                ddb.txt06000014000013115636557  7106 ustarrootwheeldb:0:kdb.enter.default>  run lockinfo
                                                db:1:lockinfo> show locks
                                                No such command
                                                db:1:locks>  show alllocks
                                                No such command
                                                db:1:alllocks>  show lockedvnods
                                                Locked vnodes
                                                db:0:kdb.enter.default>  show pcpu
                                                cpuid        = 1
                                                dynamic pcpu = 0xfffffe0167de2000
                                                curthread    = 0xfffff8015e162a00: pid 54120 "sh"
                                                curpcb       = 0xfffffe0113c6dcc0
                                                fpcurthread  = 0xfffff8015e162a00: pid 54120 "sh"
                                                idlethread   = 0xfffff8000490aa00: tid 100004 "idle: cpu1"
                                                curpmap      = 0xfffff801621b3138
                                                tssp         = 0xffffffff82a1dd78
                                                commontssp   = 0xffffffff82a1dd78
                                                rsp0         = 0xfffffe0113c6dcc0
                                                gs32p        = 0xffffffff82a245d0
                                                ldt          = 0xffffffff82a24610
                                                tss          = 0xffffffff82a24600
                                                db:0:kdb.enter.default>  bt
                                                Tracing pid 54120 tid 100774 td 0xfffff8015e162a00
                                                turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0113c6d480
                                                __rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0113c6d4b0
                                                vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0113c6d530
                                                vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0113c6d560
                                                exec_new_vmspace() at exec_new_vmspace+0x22f/frame 0xfffffe0113c6d5e0
                                                exec_elf64_imgact() at exec_elf64_imgact+0xa58/frame 0xfffffe0113c6d6f0
                                                kern_execve() at kern_execve+0x74d/frame 0xfffffe0113c6da50
                                                sys_execve() at sys_execve+0x4a/frame 0xfffffe0113c6dad0
                                                amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c6dbf0
                                                Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c6dbf0
                                                --- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x800b40d8a, rsp = 0x7fffffffe188, rbp = 0x7fffffffe2d0 ---
                                                db:0:kdb.enter.default>  ps
                                                  pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
                                                54226 53957   334     0  R       CPU 0                       sh
                                                54120 53957   334     0  R       CPU 1                       sh
                                                54035 53957   334     0  R       CPU 2                       cat
                                                53957 33060   334     0  R       CPU 3                       sh
                                                39824   334   334     0  S       accept   0xfffff8003b01006c php-fpm
                                                43353 32987 43353     0  S+      select   0xfffff801629be7c0 top
                                                32987  2745 32987     0  S+      wait     0xfffff8003b194528 bash
                                                 2745  2354  2745     0  Ss+     wait     0xfffff8003b03d528 sh
                                                 2354  8180  2354     0  Ss      select   0xfffff80102b6a5c0 sshd
                                                21591     1   334     0  S       nanslp   0xffffffff82865d21 php
                                                21318     1 21318     0  Ss      select   0xfffff80102d507c0 upsd
                                                21063     1 21063    66  Ss      select   0xfffff8003b13e6c0 apcupsd-ups
                                                19999 19580 19580    66  S       nanslp   0xffffffff82865d21 upsmon
                                                19580     1 19580     0  Ss      piperd   0xfffff80162a03be0 upsmon
                                                33060     1   334     0  S       piperd   0xfffff8015e2bf8e8 sh
                                                15682     1 15682   136  Ss      select   0xfffff8001d948240 dhcpd
                                                88257     1 88257    59  Ss      (threaded)                  unbound
                                                100711                   S       kqread   0xfffff8015eee6c00 unbound
                                                100760                   S       kqread   0xfffff80011585800 unbound
                                                100761                   S       kqread   0xfffff80011585a00 unbound
                                                100762                   S       kqread   0xfffff800101aea00 unbound
                                                55992     1 55992     0  Ss      (threaded)                  dpinger
                                                100681                   S       uwait    0xfffff80162823e00 dpinger
                                                100729                   S       sbwait   0xfffff8016285c4a4 dpinger
                                                100730                   S       nanslp   0xffffffff82865d20 dpinger
                                                100731                   S       nanslp   0xffffffff82865d21 dpinger
                                                100732                   S       accept   0xfffff80162d3d72c dpinger
                                                55571     1 55571     0  Ss      (threaded)                  dpinger
                                                100701                   S       uwait    0xfffff8001d4c3b80 dpinger
                                                100725                   S       sbwait   0xfffff80162a2f144 dpinger
                                                100726                   S       nanslp   0xffffffff82865d22 dpinger
                                                100727                   S       nanslp   0xffffffff82865d20 dpinger
                                                100728                   S       accept   0xfffff801620e2a8c dpinger
                                                89259     1 89259     0  Ss      select   0xfffff8001d4c1e40 openvpn
                                                87846     1 87846     0  Ss      select   0xfffff8016220e7c0 openvpn
                                                68550     1 68550     0  Ss      select   0xfffff80102d520c0 powerd
                                                67165 67155 67165     0  S+      ttyin    0xfffff8001d3dfca8 sh
                                                67155 65512 67155     0  S+      wait     0xfffff8016212f528 sh
                                                66898 98897 66898     0  Ss      (threaded)                  sshlockout_pf
                                                100605                   S       piperd   0xfffff8001d9db000 sshlockout_pf
                                                100683                   S       nanslp   0xffffffff82865d20 sshlockout_pf
                                                66742     1 66742     0  Ss+     ttyin    0xfffff8001d46b0a8 getty
                                                66646     1 66646     0  Ss+     ttyin    0xfffff8001d46b4a8 getty
                                                66575     1 66575     0  Ss+     ttyin    0xfffff8001d46b8a8 getty
                                                66538     1 66538     0  Ss+     ttyin    0xfffff8001d46bca8 getty
                                                66211     1 66211     0  Ss+     ttyin    0xfffff8001d46c0a8 getty
                                                66158     1 66158     0  Ss+     ttyin    0xfffff80005179ca8 getty
                                                65836     1 65836     0  Ss+     ttyin    0xfffff8001d3df0a8 getty
                                                65512     1 65512     0  Ss+     wait     0xfffff8001d976a50 login
                                                50681 50482 50482     0  S       nanslp   0xffffffff82865d20 minicron
                                                50482     1 50482     0  Ss      wait     0xfffff8001df18a50 minicron
                                                50272 49938 49938     0  S       nanslp   0xffffffff82865d23 minicron
                                                49938     1 49938     0  Ss      wait     0xfffff8001d976528 minicron
                                                49476 49314 49314     0  S       nanslp   0xffffffff82865d23 minicron
                                                49314     1 49314     0  Ss      wait     0xfffff8016212aa50 minicron
                                                44216     1 44216     0  Ss      (threaded)                  apcupsd
                                                100658                   S       nanslp   0xffffffff82865d21 apcupsd
                                                100670                   S       sigwait  0xfffff80162a01000 apcupsd
                                                100671                   S       accept   0xfffff8016212206c apcupsd
                                                 3948     1  3848     0  S       kqread   0xfffff8001dc25200 lighttpd_pfb
                                                98897     1 98897     0  Ss      select   0xfffff80102d509c0 syslogd
                                                53013     1 53013     0  Ss      kqread   0xfffff8001dbe1600 dnscrypt-proxy
                                                52232     1 52232     0  Ss      kqread   0xfffff8001db85000 dnscrypt-proxy
                                                50361     1 50361     0  Ss      kqread   0xfffff8001df2d900 dnscrypt-proxy
                                                47847     1 47764     0  S       select   0xfffff8003b3464c0 radvd
                                                40799     1 40799     0  Ss      (threaded)                  ntpd
                                                100646                   S       select   0xfffff8003b1408c0 ntpd
                                                40296     0     0     0  DL      mdwait   0xfffff80102bf3000 [md3]
                                                38103     1 38103     0  Ss      nanslp   0xffffffff82865d22 cron
                                                37771 37268 37268     0  S       kqread   0xfffff8001dc40600 nginx
                                                37735 37268 37268     0  S       kqread   0xfffff8001d5b0700 nginx
                                                37705 37268 37268     0  S       kqread   0xfffff8001dc1c900 nginx
                                                37554 37268 37268     0  S       kqread   0xfffff8001dbbaa00 nginx
                                                37494 37268 37268     0  S       kqread   0xfffff8001db43a00 nginx
                                                37268     1 37268     0  Ss      pause    0xfffff80102eecaf8 nginx
                                                20138     1 20138     0  Ss      bpf      0xfffff8003bd67000 filterlog
                                                15349     1 15349     0  Ss      select   0xfffff8003b13e1c0 dhcp6c
                                                14626     1 14626    65  Ss      select   0xfffff8001dc38ec0 dhclient
                                                 9309     1  9309     0  Ss      select   0xfffff8003b3460c0 dhclient
                                                 8519     1  8519     0  Ss      (threaded)                  sshlockout_pf
                                                100597                   S       uwait    0xfffff8001dc37600 sshlockout_pf
                                                100601                   S       nanslp   0xffffffff82865d20 sshlockout_pf
                                                 8180     1  8180     0  Ss      select   0xfffff8001d4c2640 sshd
                                                  393     0     0     0  DL      mdwait   0xfffff8001daf1800 [md2]
                                                  383     0     0     0  DL      mdwait   0xfffff8003b132000 [md1]
                                                  368     1   368     0  Ss      select   0xfffff8003b1411c0 devd
                                                  350   348   348     0  S       kqread   0xfffff8001dcabd00 check_reload_status
                                                  348     1   348     0  Ss      kqread   0xfffff8001db94800 check_reload_status
                                                  334     1   334     0  Ss      kqread   0xfffff8001ddd2100 php-fpm
                                                  103     0     0     0  DL      mdwait   0xfffff8001d565000 [md0]
                                                   27     0     0     0  DL      syncer   0xffffffff8295f880 [syncer]
                                                   26     0     0     0  DL      vlruwt   0xfffff8001d406528 [vnlru]
                                                   25     0     0     0  DL      psleep   0xffffffff8295e104 [bufdaemon]
                                                   24     0     0     0  DL      -        0xffffffff8295edb4 [bufspacedaemon]
                                                   23     0     0     0  DL      pgzero   0xffffffff829734e4 [pagezero]
                                                   22     0     0     0  DL      psleep   0xffffffff8296f98c [vmdaemon]
                                                   21     0     0     0  DL      (threaded)                  [pagedaemon]
                                                100072                   D       psleep   0xffffffff82a1d105 [pagedaemon]
                                                100081                   D       umarcl   0xffffffff8296f2b8 [uma]
                                                   20     0     0     0  DL      -        0xffffffff8295d914 [soaiod4]
                                                   19     0     0     0  DL      -        0xffffffff8295d914 [soaiod3]
                                                   18     0     0     0  DL      -        0xffffffff8295d914 [soaiod2]
                                                    9     0     0     0  DL      -        0xffffffff8295d914 [soaiod1]
                                                   17     0     0     0  DL      cooling  0xfffff80005139f58 [acpi_cooling0]
                                                   16     0     0     0  DL      tzpoll   0xffffffff82621138 [acpi_thermal]
                                                    8     0     0     0  DL      -        0xffffffff8273a4f0 [rand_harvestq]
                                                    7     0     0     0  DL      pftm     0xffffffff80f4b6a0 [pf purge]
                                                    6     0     0     0  DL      waiting_ 0xffffffff82a0c740 [sctp_iterator]
                                                    5     0     0     0  DL      (threaded)                  [zfskern]
                                                100055                   D       arc_recl 0xffffffff82f3f4d0 [arc_reclaim_thread]
                                                100056                   D       arc_user 0xffffffff82f3f4f0 [arc_user_evicts_thr]
                                                100057                   D       arc_dnlc 0xffffffff82f3f480 [arc_dnlc_evicts_thr]
                                                100058                   D       l2arc_fe 0xffffffff82f3f888 [l2arc_feed_thread]
                                                100550                   D       spa->spa 0xfffff8001d55fbd0 [trim zroot]
                                                100564                   D       tx->tx_q 0xfffff8001d45fa30 [txg_thread_enter]
                                                100565                   D       tx->tx_s 0xfffff8001d45fa10 [txg_thread_enter]
                                                   15     0     0     0  DL      (threaded)                  [usb]
                                                100035                   D       -        0xfffffe0001121460 [usbus0]
                                                100036                   D       -        0xfffffe00011214b8 [usbus0]
                                                100037                   D       -        0xfffffe0001121510 [usbus0]
                                                100038                   D       -        0xfffffe0001121568 [usbus0]
                                                100039                   D       -        0xfffffe00011215c0 [usbus0]
                                                    4     0     0     0  DL      (threaded)                  [cam]
                                                100032                   D       -        0xffffffff8260d380 [doneq0]
                                                100067                   D       -        0xffffffff8260d1c8 [scanner]
                                                    3     0     0     0  DL      crypto_r 0xffffffff8296de70 [crypto returns]
                                                    2     0     0     0  DL      crypto_w 0xffffffff8296dd18 [crypto]
                                                   14     0     0     0  DL      (threaded)                  [geom]
                                                100023                   D       -        0xffffffff829e4940 [g_event]
                                                100024                   D       -        0xffffffff829e4948 [g_up]
                                                100025                   D       -        0xffffffff829e4950 [g_down]
                                                   13     0     0     0  DL      (threaded)                  [ng_queue]
                                                100019                   D       sleep    0xffffffff825cb230 [ng_queue0]
                                                100020                   D       sleep    0xffffffff825cb230 [ng_queue1]
                                                100021                   D       sleep    0xffffffff825cb230 [ng_queue2]
                                                100022                   D       sleep    0xffffffff825cb230 [ng_queue3]
                                                   12     0     0     0  WL      (threaded)                  [intr]
                                                100007                   I                                   [swi3: vm]
                                                100008                   I                                   [swi4: clock (0)]
                                                100009                   I                                   [swi4: clock (1)]
                                                100010                   I                                   [swi4: clock (2)]
                                                100011                   I                                   [swi4: clock (3)]
                                                100012                   I                                   [swi1: netisr 0]
                                                100013                   I                                   [swi5: fast taskq]
                                                100015                   I                                   [swi6: task queue]
                                                100016                   I                                   [swi6: Giant taskq]
                                                100033                   I                                   [irq256: ahci0]
                                                100034                   I                                   [irq257: xhci0]
                                                100040                   I                                   [irq258: re0]
                                                100041                   I                                   [irq259: pcib2]
                                                100042                   I                                   [irq260: re1]
                                                100043                   I                                   [irq261: igb0:que 0]
                                                100045                   I                                   [irq262: igb0:link]
                                                100046                   I                                   [irq263: igb1:que 0]
                                                100048                   I                                   [irq264: igb1:link]
                                                100061                   I                                   [swi1: pf send]
                                                100062                   I                                   [swi1: pfsync]
                                                100079                   I                                   [swi1: netisr 1]
                                                100080                   I                                   [swi1: netisr 2]
                                                100082                   I                                   [swi1: netisr 3]
                                                   11     0     0     0  RL      (threaded)                  [idle]
                                                100003                   CanRun                              [idle: cpu0]
                                                100004                   CanRun                              [idle: cpu1]
                                                100005                   CanRun                              [idle: cpu2]
                                                100006                   CanRun                              [idle: cpu3]
                                                    1     0     1     0  SLs     wait     0xfffff80004908528 [init]
                                                   10     0     0     0  DL      audit_wo 0xffffffff82a138c0 [audit]
                                                    0     0     0     0  DLs     (threaded)                  [kernel]
                                                100000                   D       swapin   0xffffffff829e4978 [swapper]
                                                100014                   D       -        0xfffff8000491ee00 [kqueue_ctx taskq]
                                                100017                   D       -        0xfffff8000491e600 [aiod_kick taskq]
                                                100018                   D       -        0xfffff8000491e400 [thread]
                                                100026                   D       -        0xfffff8000491d600 [firmware taskq]
                                                100029                   D       -        0xfffff80004ae4d00 [acpi_task_0]
                                                100030                   D       -        0xfffff80004ae4d00 [acpi_task_1]
                                                100031                   D       -        0xfffff80004ae4d00 [acpi_task_2]
                                                100044                   D       -        0xfffff80005106b00 [igb0 que]
                                                100047                   D       -        0xfffff80004ba4d00 [igb1 que]
                                                100049                   D       -        0xfffff80005162800 [system_taskq_0]
                                                100050                   D       -        0xfffff80005162800 [system_taskq_1]
                                                100051                   D       -        0xfffff80005162800 [system_taskq_2]
                                                100052                   D       -        0xfffff80005162800 [system_taskq_3]
                                                100053                   D       -        0xfffff80004ae5000 [mca taskq]
                                                100054                   D       -        0xfffff80005161b00 [dbu_evict]
                                                100066                   D       -        0xfffff80004ae4900 [CAM taskq]
                                                100083                   D       -        0xfffff8001d467300 [if_io_tqg_0]
                                                100084                   D       -        0xfffff8001d467100 [if_io_tqg_1]
                                                100085                   D       -        0xfffff8001d4a4b00 [if_io_tqg_2]
                                                100086                   D       -        0xfffff8001d4a4900 [if_io_tqg_3]
                                                100087                   D       -        0xfffff8001d4a4700 [if_config_tqg_0]
                                                100322                   D       -        0xfffff8001d5fed00 [zio_null_issue]
                                                100323                   D       -        0xfffff8001d5b2600 [zio_null_intr]
                                                100324                   D       -        0xfffff8001d5b2300 [zio_read_issue_0]
                                                100325                   D       -        0xfffff8001d5b2300 [zio_read_issue_1]
                                                100326                   D       -        0xfffff8001d5b2300 [zio_read_issue_2]
                                                100327                   D       -        0xfffff8001d5b2300 [zio_read_issue_3]
                                                100328                   D       -        0xfffff8001d5b2300 [zio_read_issue_4]
                                                100329                   D       -        0xfffff8001d5b2300 [zio_read_issue_5]
                                                100330                   D       -        0xfffff8001d5b2300 [zio_read_issue_6]
                                                100331                   D       -        0xfffff8001d5b2300 [zio_read_issue_7]
                                                100332                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_0]
                                                100333                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_1]
                                                100334                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_2]
                                                100335                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_3]
                                                100336                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_4]
                                                100337                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_5]
                                                100338                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_6]
                                                100339                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_7]
                                                100340                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_8]
                                                100341                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_9]
                                                100342                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_10]
                                                100343                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_11]
                                                100344                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_0]
                                                100345                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_1]
                                                100346                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_2]
                                                100347                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_3]
                                                100348                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_4]
                                                100349                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_5]
                                                100350                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_6]
                                                100351                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_7]
                                                100352                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_8]
                                                100353                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_9]
                                                100354                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_10]
                                                100355                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_11]
                                                100356                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_0]
                                                100357                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_1]
                                                100358                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_2]
                                                100359                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_3]
                                                100360                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_4]
                                                100361                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_5]
                                                100362                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_6]
                                                100363                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_7]
                                                100364                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_8]
                                                100365                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_9]
                                                100366                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_10]
                                                100367                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_11]
                                                100368                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_0]
                                                100369                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_1]
                                                100370                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_2]
                                                100371                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_3]
                                                100372                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_4]
                                                100373                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_5]
                                                100374                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_6]
                                                100375                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_7]
                                                100376                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_8]
                                                100377                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_9]
                                                100378                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_10]
                                                100379                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_11]
                                                100380                   D       -        0xfffff8001d552500 [zio_read_intr_4_0]
                                                100381                   D       -        0xfffff8001d552500 [zio_read_intr_4_1]
                                                100382                   D       -        0xfffff8001d552500 [zio_read_intr_4_2]
                                                100383                   D       -        0xfffff8001d552500 [zio_read_intr_4_3]
                                                100384                   D       -        0xfffff8001d552500 [zio_read_intr_4_4]
                                                100385                   D       -        0xfffff8001d552500 [zio_read_intr_4_5]
                                                100386                   D       -        0xfffff8001d552500 [zio_read_intr_4_6]
                                                100387                   D       -        0xfffff8001d552500 [zio_read_intr_4_7]
                                                100388                   D       -        0xfffff8001d552500 [zio_read_intr_4_8]
                                                100389                   D       -        0xfffff8001d552500 [zio_read_intr_4_9]
                                                100390                   D       -        0xfffff8001d552500 [zio_read_intr_4_10]
                                                100391                   D       -        0xfffff8001d552500 [zio_read_intr_4_11]
                                                100392                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_0]
                                                100393                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_1]
                                                100394                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_2]
                                                100395                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_3]
                                                100396                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_4]
                                                100397                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_5]
                                                100398                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_6]
                                                100399                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_7]
                                                100400                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_8]
                                                100401                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_9]
                                                100402                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_10]
                                                100403                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_11]
                                                100404                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_0]
                                                100405                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_1]
                                                100406                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_2]
                                                100407                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_3]
                                                100408                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_4]
                                                100409                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_5]
                                                100410                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_6]
                                                100411                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_7]
                                                100412                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_8]
                                                100413                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_9]
                                                100414                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_10]
                                                100415                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_11]
                                                100416                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_0]
                                                100417                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_1]
                                                100418                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_2]
                                                100419                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_3]
                                                100420                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_4]
                                                100421                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_5]
                                                100422                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_6]
                                                100423                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_7]
                                                100424                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_8]
                                                100425                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_9]
                                                100426                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_10]
                                                100427                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_11]
                                                100428                   D       -        0xfffff8001d589300 [zio_write_issue_0]
                                                100429                   D       -        0xfffff8001d589300 [zio_write_issue_1]
                                                100430                   D       -        0xfffff8001d589300 [zio_write_issue_2]
                                                100431                   D       -        0xfffff8001d589200 [zio_write_issue_hig]
                                                100432                   D       -        0xfffff8001d589200 [zio_write_issue_hig]
                                                100433                   D       -        0xfffff8001d589200 [zio_write_issue_hig]
                                                100434                   D       -        0xfffff8001d589200 [zio_write_issue_hig]
                                                100435                   D       -        0xfffff8001d589200 [zio_write_issue_hig]
                                                100436                   D       -        0xfffff8001d589000 [zio_write_intr_0]
                                                100437                   D       -        0xfffff8001d589000 [zio_write_intr_1]
                                                100438                   D       -        0xfffff8001d589000 [zio_write_intr_2]
                                                100439                   D       -        0xfffff8001d589000 [zio_write_intr_3]
                                                100440                   D       -        0xfffff8001d589000 [zio_write_intr_4]
                                                100441                   D       -        0xfffff8001d589000 [zio_write_intr_5]
                                                100442                   D       -        0xfffff8001d589000 [zio_write_intr_6]
                                                100443                   D       -        0xfffff8001d589000 [zio_write_intr_7]
                                                100444                   D       -        0xfffff8001d588100 [zio_write_intr_high]
                                                100445                   D       -        0xfffff8001d588100 [zio_write_intr_high]
                                                100446                   D       -        0xfffff8001d588100 [zio_write_intr_high]
                                                100447                   D       -        0xfffff8001d588100 [zio_write_intr_high]
                                                100448                   D       -        0xfffff8001d588100 [zio_write_intr_high]
                                                100449                   D       -        0xfffff8001d587800 [zio_free_issue_0_0]
                                                100450                   D       -        0xfffff8001d587800 [zio_free_issue_0_1]
                                                100451                   D       -        0xfffff8001d587800 [zio_free_issue_0_2]
                                                100452                   D       -        0xfffff8001d587800 [zio_free_issue_0_3]
                                                100453                   D       -        0xfffff8001d587800 [zio_free_issue_0_4]
                                                100454                   D       -        0xfffff8001d587800 [zio_free_issue_0_5]
                                                100455                   D       -        0xfffff8001d587800 [zio_free_issue_0_6]
                                                100456                   D       -        0xfffff8001d587800 [zio_free_issue_0_7]
                                                100457                   D       -        0xfffff8001d587800 [zio_free_issue_0_8]
                                                100458                   D       -        0xfffff8001d587800 [zio_free_issue_0_9]
                                                100459                   D       -        0xfffff8001d587800 [zio_free_issue_0_10]
                                                100460                   D       -        0xfffff8001d587800 [zio_free_issue_0_11]
                                                100461                   D       -        0xfffff8001d587100 [zio_free_issue_1_0]
                                                100462                   D       -        0xfffff8001d587100 [zio_free_issue_1_1]
                                                100463                   D       -        0xfffff8001d587100 [zio_free_issue_1_2]
                                                100464                   D       -        0xfffff8001d587100 [zio_free_issue_1_3]
                                                100465                   D       -        0xfffff8001d587100 [zio_free_issue_1_4]
                                                100466                   D       -        0xfffff8001d587100 [zio_free_issue_1_5]
                                                100467                   D       -        0xfffff8001d587100 [zio_free_issue_1_6]
                                                100468                   D       -        0xfffff8001d587100 [zio_free_issue_1_7]
                                                100469                   D       -        0xfffff8001d587100 [zio_free_issue_1_8]
                                                100470                   D       -        0xfffff8001d587100 [zio_free_issue_1_9]
                                                100471                   D       -        0xfffff8001d587100 [zio_free_issue_1_10]
                                                100472                   D       -        0xfffff8001d587100 [zio_free_issue_1_11]
                                                100473                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_0]
                                                100474                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_1]
                                                100475                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_2]
                                                100476                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_3]
                                                100477                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_4]
                                                100478                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_5]
                                                100479                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_6]
                                                100480                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_7]
                                                100481                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_8]
                                                100482                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_9]
                                                100483                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_10]
                                                100484                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_11]
                                                100485                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_0]
                                                100486                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_1]
                                                100487                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_2]
                                                100488                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_3]
                                                100489                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_4]
                                                100490                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_5]
                                                100491                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_6]
                                                100492                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_7]
                                                100493                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_8]
                                                100494                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_9]
                                                100495                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_10]
                                                100496                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_11]
                                                100497                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_0]
                                                100498                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_1]
                                                100499                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_2]
                                                100500                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_3]
                                                100501                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_4]
                                                100502                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_5]
                                                100503                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_6]
                                                100504                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_7]
                                                100505                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_8]
                                                100506                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_9]
                                                100507                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_10]
                                                100508                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_11]
                                                100509                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_0]
                                                100510                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_1]
                                                100511                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_2]
                                                100512                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_3]
                                                100513                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_4]
                                                100514                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_5]
                                                100515                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_6]
                                                [/thread]
                                                

                                                pfSense 2.6.0 - ISP AAISP UK

                                                1 Reply Last reply Reply Quote 0
                                                • C
                                                  chrcoluk last edited by

                                                  continued here - hit post limit

                                                  100516                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_7]
                                                  100517                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_8]
                                                  100518                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_9]
                                                  100519                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_10]
                                                  100520                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_11]
                                                  100521                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_0]
                                                  100522                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_1]
                                                  100523                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_2]
                                                  100524                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_3]
                                                  100525                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_4]
                                                  100526                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_5]
                                                  100527                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_6]
                                                  100528                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_7]
                                                  100529                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_8]
                                                  100530                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_9]
                                                  100531                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_10]
                                                  100532                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_11]
                                                  100533                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_0]
                                                  100534                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_1]
                                                  100535                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_2]
                                                  100536                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_3]
                                                  100537                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_4]
                                                  100538                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_5]
                                                  100539                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_6]
                                                  100540                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_7]
                                                  100541                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_8]
                                                  100542                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_9]
                                                  100543                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_10]
                                                  100544                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_11]
                                                  100545                   D       -        0xfffff8001d5db700 [zio_free_intr]
                                                  100546                   D       -        0xfffff8001d5db300 [zio_claim_issue]
                                                  100547                   D       -        0xfffff8001d5db600 [zio_claim_intr]
                                                  100548                   D       -        0xfffff8001d5dae00 [zio_ioctl_issue]
                                                  100549                   D       -        0xfffff8001d5dab00 [zio_ioctl_intr]
                                                  100551                   D       -        0xfffff8001d5dac00 [metaslab_group_task]
                                                  100552                   D       -        0xfffff8001d5dac00 [metaslab_group_task]
                                                  100553                   D       -        0xfffff8001d5da000 [zfs_vn_rele_taskq]
                                                  100566                   D       -        0xfffff8001d8dd200 [zil_clean]
                                                  100572                   D       -        0xfffff8001daa6600 [zil_clean]
                                                  100573                   D       -        0xfffff8001d9d8a00 [zil_clean]
                                                  100574                   D       -        0xfffff8001d6fd100 [zil_clean]
                                                  oops, ran out of processes early!
                                                  db:0:kdb.enter.default>  alltrace
                                                  
                                                  Tracing command sh pid 54226 tid 100593 td 0xfffff8001dece000
                                                  cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff82977730
                                                  ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xffffffff82977750
                                                  trap() at trap+0x3a/frame 0xffffffff82977960
                                                  nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff82977960
                                                  --- trap 0x13, rip = 0xffffffff811a4a14, rsp = 0xfffffe0113a4f490, rbp = 0xfffffe0113a4f560 ---
                                                  pmap_remove_pages() at pmap_remove_pages+0x384/frame 0xfffffe0113a4f560
                                                  exec_new_vmspace() at exec_new_vmspace+0x21d/frame 0xfffffe0113a4f5e0
                                                  exec_elf64_imgact() at exec_elf64_imgact+0xa58/frame 0xfffffe0113a4f6f0
                                                  kern_execve() at kern_execve+0x74d/frame 0xfffffe0113a4fa50
                                                  sys_execve() at sys_execve+0x4a/frame 0xfffffe0113a4fad0
                                                  amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113a4fbf0
                                                  Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113a4fbf0
                                                  --- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x800b40d8a, rsp = 0x7fffffffe188, rbp = 0x7fffffffe2d0 ---
                                                  
                                                  Tracing command sh pid 54120 tid 100774 td 0xfffff8015e162a00
                                                  turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0113c6d480
                                                  __rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0113c6d4b0
                                                  vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0113c6d530
                                                  vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0113c6d560
                                                  exec_new_vmspace() at exec_new_vmspace+0x22f/frame 0xfffffe0113c6d5e0
                                                  exec_elf64_imgact() at exec_elf64_imgact+0xa58/frame 0xfffffe0113c6d6f0
                                                  kern_execve() at kern_execve+0x74d/frame 0xfffffe0113c6da50
                                                  sys_execve() at sys_execve+0x4a/frame 0xfffffe0113c6dad0
                                                  amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c6dbf0
                                                  Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c6dbf0
                                                  --- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x800b40d8a, rsp = 0x7fffffffe188, rbp = 0x7fffffffe2d0 ---
                                                  
                                                  Tracing command cat pid 54035 tid 100570 td 0xfffff8001d601000
                                                  cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00ea3f4cf0
                                                  ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe00ea3f4d10
                                                  trap() at trap+0x3a/frame 0xfffffe00ea3f4f20
                                                  nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00ea3f4f20
                                                  --- trap 0x13, rip = 0xffffffff82e91de0, rsp = 0xfffffe01137de588, rbp = 0xfffffe01137de5b0 ---
                                                  zrl_add_impl() at zrl_add_impl/frame 0xfffffe01137de5b0
                                                  zfs_freebsd_getattr() at zfs_freebsd_getattr+0xa2e/frame 0xfffffe01137de820
                                                  VOP_GETATTR_APV() at VOP_GETATTR_APV+0x87/frame 0xfffffe01137de850
                                                  vm_mmap_vnode() at vm_mmap_vnode+0x164/frame 0xfffffe01137de990
                                                  vn_mmap() at vn_mmap+0xe5/frame 0xfffffe01137dea20
                                                  sys_mmap() at sys_mmap+0x40c/frame 0xfffffe01137dead0
                                                  amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe01137debf0
                                                  Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe01137debf0
                                                  --- syscall (477, FreeBSD ELF64, sys_mmap), rip = 0x800613ffa, rsp = 0x7fffffffde68, rbp = 0x7fffffffdfc0 ---
                                                  
                                                  Tracing command sh pid 53957 tid 101116 td 0xfffff80162b5a500
                                                  cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00ea3fccf0
                                                  ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe00ea3fcd10
                                                  trap() at trap+0x3a/frame 0xfffffe00ea3fcf20
                                                  nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00ea3fcf20
                                                  --- trap 0x13, rip = 0xffffffff811a3b4b, rsp = 0xfffffe0113c9a850, rbp = 0xfffffe0113c9a890 ---
                                                  pmap_try_insert_pv_entry() at pmap_try_insert_pv_entry+0x13b/frame 0xfffffe0113c9a890
                                                  pmap_copy() at pmap_copy+0x65c/frame 0xfffffe0113c9a930
                                                  vmspace_fork() at vmspace_fork+0x8d9/frame 0xfffffe0113c9a9b0
                                                  fork1() at fork1+0x5ec/frame 0xfffffe0113c9aa70
                                                  sys_fork() at sys_fork+0x39/frame 0xfffffe0113c9aad0
                                                  amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c9abf0
                                                  Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c9abf0
                                                  --- syscall (2, FreeBSD ELF64, sys_fork), rip = 0x800b5d77a, rsp = 0x7fffffffe508, rbp = 0x7fffffffe540 ---
                                                  
                                                  Tracing command php-fpm pid 39824 tid 100686 td 0xfffff8003b242000
                                                  sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113bb2850
                                                  mi_switch() at mi_switch+0xd2/frame 0xfffffe0113bb2880
                                                  sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113bb2900
                                                  sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113bb2930
                                                  _sleep() at _sleep+0x294/frame 0xfffffe0113bb29c0
                                                  kern_accept4() at kern_accept4+0x1f2/frame 0xfffffe0113bb2a70
                                                  accept1() at accept1+0x60/frame 0xfffffe0113bb2ad0
                                                  amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113bb2bf0
                                                  Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113bb2bf0
                                                  --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x801d947da, rsp = 0x7ffffffec848, rbp = 0x7ffffffec880 ---
                                                  
                                                  Tracing command top pid 43353 tid 100695 td 0xfffff80162d08a00
                                                  sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113b35700
                                                  mi_switch() at mi_switch+0xd2/frame 0xfffffe0113b35730
                                                  sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113b357b0
                                                  sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113b357e0
                                                  _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe0113b35850
                                                  seltdwait() at seltdwait+0xc7/frame 0xfffffe0113b358a0
                                                  kern_select() at kern_select+0x979/frame 0xfffffe0113b35a90
                                                  sys_select() at sys_select+0x54/frame 0xfffffe0113b35ad0
                                                  amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113b35bf0
                                                  Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113b35bf0
                                                  --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80121000a, rsp = 0x7fffffffe888, rbp = 0x7fffffffea60 ---
                                                  
                                                  Tracing command bash pid 32987 tid 100680 td 0xfffff801628d1000
                                                  sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113b946a0
                                                  mi_switch() at mi_switch+0xd2/frame 0xfffffe0113b946d0
                                                  sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113b94750
                                                  sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113b94780
                                                  _sleep() at _sleep+0x294/frame 0xfffffe0113b94810
                                                  kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe0113b948e0
                                                  sys_wait4() at sys_wait4+0x73/frame 0xfffffe0113b94ad0
                                                  amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113b94bf0
                                                  Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113b94bf0
                                                  --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800e5d57a, rsp = 0x7fffffffe7a8, rbp = 0x7fffffffe7f0 ---
                                                  
                                                  Tracing command sh pid 2745 tid 100748 td 0xfffff8015e029a00
                                                  sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113c366a0
                                                  mi_switch() at mi_switch+0xd2/frame 0xfffffe0113c366d0
                                                  sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113c36750
                                                  sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113c36780
                                                  _sleep() at _sleep+0x294/frame 0xfffffe0113c36810
                                                  kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe0113c368e0
                                                  sys_wait4() at sys_wait4+0x73/frame 0xfffffe0113c36ad0
                                                  amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c36bf0
                                                  Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c36bf0
                                                  --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800b5d57a, rsp = 0x7fffffffe768, rbp = 0x7fffffffe820 ---
                                                  
                                                  Tracing command sshd pid 2354 tid 100699 td 0xfffff8001dd13000
                                                  sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113bbc700
                                                  mi_switch() at mi_switch+0xd2/frame 0xfffffe0113bbc730
                                                  sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113bbc7b0
                                                  sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113bbc7e0
                                                  _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe0113bbc850
                                                  seltdwait() at seltdwait+0xc7/frame 0xfffffe0113bbc8a0
                                                  kern_select() at kern_select+0x979/frame 0xfffffe0113bbca90
                                                  sys_select() at sys_select+0x54/frame 0xfffffe0113bbcad0
                                                  amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113bbcbf0
                                                  Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113bbcbf0
                                                  --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x801db400a, rsp = 0x7fffffffe5a8, rbp = 0x7fffffffe5e0 ---
                                                  
                                                  Tracing command php pid 21591 tid 100704 td 0xfffff8001d4b5500
                                                  sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113bd58c0
                                                  mi_switch() at mi_switch+0xd2/frame 0xfffffe0113bd58f0
                                                  sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113bd5970
                                                  sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113bd59a0
                                                  _sleep() at _sleep+0x26f/frame 0xfffffe0113bd5a30
                                                  kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe0113bd5a90
                                                  sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe0113bd5ad0
                                                  amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113bd5bf0
                                                  Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113bd5bf0
                                                  --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x801d8571a, rsp = 0x7fffffffc7e8, rbp = 0x7fffffffc810 ---
                                                  
                                                  Tracing command upsd pid 21318 tid 100739 td 0xfffff80162cd4500
                                                  sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113c09770
                                                  mi_switch() at mi_switch+0xd2/frame 0xfffffe0113c097a0
                                                  sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113c09820
                                                  sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113c09850
                                                  _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe0113c098c0
                                                  seltdwait() at seltdwait+0xc7/frame 0xfffffe0113c09910
                                                  kern_poll() at kern_poll+0x296/frame 0xfffffe0113c09ab0
                                                  sys_poll() at sys_poll+0x61/frame 0xfffffe0113c09ad0
                                                  amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c09bf0
                                                  Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c09bf0
                                                  --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x800d4fd9a, rsp = 0x7fffffffe668, rbp = 0x7fffffffe6a0 ---
                                                  
                                                  Tracing command apcupsd-ups pid 21063 tid 100611 td 0xfffff8003be18500
                                                  sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113aa9700
                                                  mi_switch() at mi_switch+0xd2/frame 0xfffffe0113aa9730
                                                  sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113aa97b0
                                                  sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113aa97e0
                                                  _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe0113aa9850
                                                  seltdwait() at seltdwait+0xc7/frame 0xfffffe0113aa98a0
                                                  kern_select() at kern_select+0x979/frame 0xfffffe0113aa9a90
                                                  sys_select() at sys_select+0x54/frame 0xfffffe0113aa9ad0
                                                  amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113aa9bf0
                                                  Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113aa9bf0
                                                  --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800b9b00a, rsp = 0x7fffffffe678, rbp = 0x7fffffffe6b0 ---
                                                  
                                                  Tracing command upsmon pid 19999 tid 100703 td 0xfffff8001d4b5a00
                                                  sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113bd08c0
                                                  mi_switch() at mi_switch+0xd2/frame 0xfffffe0113bd08f0
                                                  sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113bd0970
                                                  sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113bd09a0
                                                  _sleep() at _sleep+0x26f/frame 0xfffffe0113bd0a30
                                                  kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe0113bd0a90
                                                  sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe0113bd0ad0
                                                  amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113bd0bf0
                                                  Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113bd0bf0
                                                  --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800d3c71a, rsp = 0x7fffffffe5c8, rbp = 0x7fffffffe5f0 ---
                                                  
                                                  Tracing command upsmon pid 19580 tid 100685 td 0xfffff80162d0a500
                                                  sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113bad810
                                                  mi_switch() at mi_switch+0xd2/frame 0xfffffe0113bad840
                                                  sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113bad8c0
                                                  sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113bad8f0
                                                  _sleep() at _sleep+0x294/frame 0xfffffe0113bad980
                                                  pipe_read() at pipe_read+0x36d/frame 0xfffffe0113bad9e0
                                                  dofileread() at dofileread+0x98/frame 0xfffffe0113bada30
                                                  kern_readv() at kern_readv+0x68/frame 0xfffffe0113bada80
                                                  sys_read() at sys_read+0x84/frame 0xfffffe0113badad0
                                                  amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113badbf0
                                                  Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113badbf0
                                                  --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800da502a, rsp = 0x7fffffffe628, rbp = 0x7fffffffe660 ---
                                                  
                                                  Tracing command sh pid 33060 tid 100578 td 0xfffff8001d4e5a00
                                                  sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113a04810
                                                  mi_switch() at mi_switch+0xd2/frame 0xfffffe0113a04840
                                                  sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113a048c0
                                                  sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113a048f0
                                                  _sleep() at _sleep+0x294/frame 0xfffffe0113a04980
                                                  pipe_read() at pipe_read+0x36d/frame 0xfffffe0113a049e0
                                                  dofileread() at dofileread+0x98/frame 0xfffffe0113a04a30
                                                  kern_readv() at kern_readv+0x68/frame 0xfffffe0113a04a80
                                                  sys_read() at sys_read+0x84/frame 0xfffffe0113a04ad0
                                                  amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113a04bf0
                                                  Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113a04bf0
                                                  --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800bc602a, rsp = 0x7fffffffe6a8, rbp = 0x7fffffffe880 ---
                                                  
                                                  Tracing command dhcpd pid 15682 tid 100713 td 0xfffff80162d08000
                                                  sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113c04700
                                                  mi_switch() at mi_switch+0xd2/frame 0xfffffe0113c04730
                                                  sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113c047b0
                                                  sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113c047e0
                                                  _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe0113c04850
                                                  seltdwait() at seltdwait+0xc7/frame 0xfffffe0113c048a0
                                                  kern_select() at kern_select+0x979/frame 0xfffffe0113c04a90
                                                  sys_select() at sys_select+0x54/frame 0xfffffe0113c04ad0
                                                  amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c04bf0
                                                  Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c04bf0
                                                  --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800d6400a, rsp = 0x7fffffffea38, rbp = 0x7fffffffea60 ---
                                                  
                                                  Tracing command unbound pid 88257 tid 100711 td 0xfffff80162cc2a00
                                                  sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113bf8660
                                                  mi_switch() at mi_switch+0xd2/frame 0xfffffe0113bf8690
                                                  sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113bf8710
                                                  sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113bf8740
                                                  _sleep() at _sleep+0x294/frame 0xfffffe0113bf87d0
                                                  kqueue_kevent() at kqueue_kevent+0x336/frame 0xfffffe0113bf89b0
                                                  kern_kevent() at kern_kevent+0xb5/frame 0xfffffe0113bf8a20
                                                  sys_kevent() at sys_kevent+0x11c/frame 0xfffffe0113bf8ad0
                                                  amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113bf8bf0
                                                  Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113bf8bf0
                                                  --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x80177275a, rsp = 0x7fffffffe9f8, rbp = 0x7fffffffea40 ---
                                                  
                                                  Tracing command unbound pid 88257 tid 100760 td 0xfffff80162da8a00
                                                  sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113c40660
                                                  mi_switch() at mi_switch+0xd2/frame 0xfffffe0113c40690
                                                  sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113c40710
                                                  sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113c40740
                                                  _sleep() at _sleep+0x294/frame 0xfffffe0113c407d0
                                                  kqueue_kevent() at kqueue_kevent+0x336/frame 0xfffffe0113c409b0
                                                  kern_kevent() at kern_kevent+0xb5/frame 0xfffffe0113c40a20
                                                  sys_kevent() at sys_kevent+0x11c/frame 0xfffffe0113c40ad0
                                                  amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c40bf0
                                                  Xfast_syscall() at Xfast_syscall+0config.txt0600001341413115636557  7552 ustarrootwheeloptions	CONFIG_AUTOGENERATED
                                                  ident	pfSense
                                                  machine	amd64
                                                  cpu	HAMMER
                                                  makeoptions	WITH_CTF=1
                                                  makeoptions	DEBUG=-g
                                                  options	CONSPEED=115200
                                                  options	MROUTING
                                                  options	HIFN_RNDTEST
                                                  options	HIFN_DEBUG
                                                  options	MSGTQL=2048
                                                  options	MSGSSZ=32
                                                  options	MSGSEG=512
                                                  options	MSGMNI=40
                                                  options	MSGMNB=8192
                                                  options	ALTQ_CODEL
                                                  options	ALTQ_NOPCC
                                                  options	ALTQ_FAIRQ
                                                  options	ALTQ_PRIQ
                                                  options	ALTQ_HFSC
                                                  options	ALTQ_RIO
                                                  options	ALTQ_RED
                                                  options	ALTQ_CBQ
                                                  options	ALTQ
                                                  options	TCP_SIGNATURE
                                                  options	IPSEC_NAT_T
                                                  options	NETGRAPH_PRED1
                                                  options	NETGRAPH_DEFLATE
                                                  options	NETGRAPH_CAR
                                                  options	NETGRAPH_PIPE
                                                  options	NETGRAPH_TCPMSS
                                                  options	NETGRAPH_TEE
                                                  options	NETGRAPH_HOLE
                                                  options	NETGRAPH_FRAME_RELAY
                                                  options	NETGRAPH_ASYNC
                                                  options	NETGRAPH_ECHO
                                                  options	NETGRAPH_CISCO
                                                  options	NETGRAPH_BRIDGE
                                                  options	NETGRAPH_ONE2MANY
                                                  options	NETGRAPH_LMI
                                                  options	NETGRAPH_KSOCKET
                                                  options	NETGRAPH_VJC
                                                  options	NETGRAPH_UI
                                                  options	NETGRAPH_MPPC_ENCRYPTION
                                                  options	NETGRAPH_TTY
                                                  options	NETGRAPH_SOCKET
                                                  options	NETGRAPH_RFC1490
                                                  options	NETGRAPH_PPTPGRE
                                                  options	NETGRAPH_PPPOE
                                                  options	NETGRAPH_PPP
                                                  options	NETGRAPH_EIFACE
                                                  options	NETGRAPH_IFACE
                                                  options	NETGRAPH_ETHER
                                                  options	NETGRAPH_BPF
                                                  options	NETGRAPH_L2TP
                                                  options	NETGRAPH_VLAN
                                                  options	NETGRAPH
                                                  options	IPSTEALTH
                                                  options	IPFIREWALL_VERBOSE
                                                  options	IPFIREWALL_DEFAULT_TO_ACCEPT
                                                  options	PPS_SYNC
                                                  options	NULLFS
                                                  options	UNIONFS
                                                  options	TMPFS
                                                  options	GEOM_BDE
                                                  options	GEOM_ELI
                                                  options	GEOM_UZIP
                                                  options	GEOM_MIRROR
                                                  options	DDB
                                                  options	XENHVM
                                                  options	ATH_ENABLE_11N
                                                  options	AH_AR5416_INTERRUPT_MITIGATION
                                                  options	AH_SUPPORT_AR5416
                                                  options	IEEE80211_SUPPORT_MESH
                                                  options	IEEE80211_AMPDU_AGE
                                                  options	IEEE80211_DEBUG
                                                  options	SC_PIXEL_MODE
                                                  options	VESA
                                                  options	AHD_REG_PRETTY_PRINT
                                                  options	AHC_REG_PRETTY_PRINT
                                                  options	PCI_IOV
                                                  options	PCI_HP
                                                  options	ACPI_DMAR
                                                  options	DEVICE_NUMA
                                                  options	SMP
                                                  options	KDB
                                                  options	RCTL
                                                  options	RACCT_DEFAULT_TO_DISABLED
                                                  options	RACCT
                                                  options	INCLUDE_CONFIG_FILE
                                                  options	DDB_CTF
                                                  options	KDTRACE_HOOKS
                                                  options	KDTRACE_FRAME
                                                  options	CAPABILITIES
                                                  options	CAPABILITY_MODE
                                                  options	AUDIT
                                                  options	HWPMC_HOOKS
                                                  options	KBD_INSTALL_CDEV
                                                  options	PRINTF_BUFR_SIZE=128
                                                  options	_KPOSIX_PRIORITY_SCHEDULING
                                                  options	SYSVSEM
                                                  options	SYSVMSG
                                                  options	SYSVSHM
                                                  options	STACK
                                                  options	KTRACE
                                                  options	SCSI_DELAY=5000
                                                  options	COMPAT_FREEBSD10
                                                  options	COMPAT_FREEBSD9
                                                  options	COMPAT_FREEBSD32
                                                  options	GEOM_LABEL
                                                  options	GEOM_RAID
                                                  options	GEOM_PART_GPT
                                                  options	PSEUDOFS
                                                  options	PROCFS
                                                  options	CD9660
                                                  options	MSDOSFS
                                                  options	NFS_ROOT
                                                  options	NFSLOCKD
                                                  options	NFSD
                                                  options	NFSCL
                                                  options	MD_ROOT
                                                  options	QUOTA
                                                  options	UFS_GJOURNAL
                                                  options	UFS_DIRHASH
                                                  options	UFS_ACL
                                                  options	SOFTUPDATES
                                                  options	FFS
                                                  options	SCTP
                                                  options	TCP_OFFLOAD
                                                  options	IPSEC
                                                  options	INET6
                                                  options	INET
                                                  options	PREEMPTION
                                                  options	SCHED_ULE
                                                  options	NEW_PCIB
                                                  options	GEOM_PART_MBR
                                                  options	GEOM_PART_EBR_COMPAT
                                                  options	GEOM_PART_EBR
                                                  options	GEOM_PART_BSD
                                                  device	isa
                                                  device	mem
                                                  device	io
                                                  device	uart_ns8250
                                                  device	cpufreq
                                                  device	acpi
                                                  device	pci
                                                  device	fdc
                                                  device	ahci
                                                  device	ata
                                                  device	mvs
                                                  device	siis
                                                  device	ahc
                                                  device	ahd
                                                  device	esp
                                                  device	hptiop
                                                  device	isp
                                                  device	mpt
                                                  device	mps
                                                  device	mpr
                                                  device	sym
                                                  device	trm
                                                  device	adv
                                                  device	adw
                                                  device	aic
                                                  device	bt
                                                  device	isci
                                                  device	scbus
                                                  device	ch
                                                  device	da
                                                  device	sa
                                                  device	cd
                                                  device	pass
                                                  device	ses
                                                  device	amr
                                                  device	arcmsr
                                                  device	ciss
                                                  device	dpt
                                                  device	hptmv
                                                  device	hptnr
                                                  device	hptrr
                                                  device	hpt27xx
                                                  device	iir
                                                  device	ips
                                                  device	mly
                                                  device	twa
                                                  device	tws
                                                  device	aac
                                                  device	aacp
                                                  device	aacraid
                                                  device	ida
                                                  device	mfi
                                                  device	mlx
                                                  device	mrsas
                                                  device	pmspcv
                                                  device	twe
                                                  device	nvme
                                                  device	nvd
                                                  device	atkbdc
                                                  device	atkbd
                                                  device	psm
                                                  device	kbdmux
                                                  device	vga
                                                  device	splash
                                                  device	sc
                                                  device	vt
                                                  device	vt_vga
                                                  device	vt_efifb
                                                  device	agp
                                                  device	cbb
                                                  device	pccard
                                                  device	cardbus
                                                  device	uart
                                                  device	ppc
                                                  device	ppbus
                                                  device	lpt
                                                  device	ppi
                                                  device	puc
                                                  device	bxe
                                                  device	de
                                                  device	em
                                                  device	igb
                                                  device	ix
                                                  device	ixv
                                                  device	ixl
                                                  device	ixlv
                                                  device	le
                                                  device	ti
                                                  device	txp
                                                  device	vx
                                                  device	miibus
                                                  device	ae
                                                  device	age
                                                  device	alc
                                                  device	ale
                                                  device	bce
                                                  device	bfe
                                                  device	bge
                                                  device	cas
                                                  device	dc
                                                  device	et
                                                  device	fxp
                                                  device	gem
                                                  device	hme
                                                  device	jme
                                                  device	lge
                                                  device	msk
                                                  device	nfe
                                                  device	nge
                                                  device	pcn
                                                  device	re
                                                  device	rl
                                                  device	sf
                                                  device	sge
                                                  device	sis
                                                  device	sk
                                                  device	ste
                                                  device	stge
                                                  device	tl
                                                  device	tx
                                                  device	vge
                                                  device	vr
                                                  device	wb
                                                  device	xl
                                                  device	wlan
                                                  device	wlan_wep
                                                  device	wlan_ccmp
                                                  device	wlan_tkip
                                                  device	wlan_amrr
                                                  device	an
                                                  device	ath
                                                  device	ath_pci
                                                  device	ath_hal
                                                  device	ath_rate_sample
                                                  device	ipw
                                                  device	iwi
                                                  device	iwn
                                                  device	malo
                                                  device	mwl
                                                  device	ral
                                                  device	wi
                                                  device	wpi
                                                  device	loop
                                                  device	random
                                                  device	padlock_rng
                                                  device	rdrand_rng
                                                  device	ether
                                                  device	vlan
                                                  device	tun
                                                  device	md
                                                  device	gif
                                                  device	firmware
                                                  device	bpf
                                                  device	uhci
                                                  device	ohci
                                                  device	ehci
                                                  device	xhci
                                                  device	usb
                                                  device	ukbd
                                                  device	umass
                                                  device	sound
                                                  device	snd_cmi
                                                  device	snd_csa
                                                  device	snd_emu10kx
                                                  device	snd_es137x
                                                  device	snd_hda
                                                  device	snd_ich
                                                  device	snd_via8233
                                                  device	mmc
                                                  device	mmcsd
                                                  device	sdhci
                                                  device	virtio
                                                  device	virtio_pci
                                                  device	vtnet
                                                  device	virtio_blk
                                                  device	virtio_scsi
                                                  device	virtio_balloon
                                                  device	hyperv
                                                  device	xenpci
                                                  device	vmx
                                                  device	netmap
                                                  device	crypto
                                                  device	wlan_rssadapt
                                                  device	wlan_xauth
                                                  device	wlan_acl
                                                  device	iwifw
                                                  device	ipwfw
                                                  device	wpifw
                                                  device	iwnfw
                                                  device	uath
                                                  device	ralfw
                                                  device	ural
                                                  device	urtw
                                                  device	rum
                                                  device	mwlfw
                                                  device	zyd
                                                  device	upgt
                                                  device	udav
                                                  device	axe
                                                  device	axge
                                                  device	aue
                                                  device	cue
                                                  device	kue
                                                  device	mos
                                                  device	rsu
                                                  device	rsufw
                                                  device	run
                                                  device	runfw
                                                  device	rue
                                                  device	rtwn
                                                  device	rtwnfw
                                                  device	siba_bwn
                                                  device	bwn
                                                  device	bwi
                                                  device	ufoma
                                                  device	ucom
                                                  device	uslcom
                                                  device	uplcom
                                                  device	umct
                                                  device	uvisor
                                                  device	uark
                                                  device	uftdi
                                                  device	uvscom
                                                  device	umodem
                                                  device	u3g
                                                  device	cdce
                                                  device	uhid
                                                  device	firewire
                                                  device	sbp
                                                  device	tap
                                                  device	gre
                                                  device	if_bridge
                                                  device	carp
                                                  device	lagg
                                                  device	vte
                                                  device	enc
                                                  device	pf
                                                  device	pflog
                                                  device	pfsync
                                                  device	rndtest
                                                  device	hifn
                                                  device	ubsec
                                                  device	safe
                                                  device	padlock
                                                  device	speaker
                                                  device	mxge
                                                  device	cxgb
                                                  device	cxgbe
                                                  device	oce
                                                  msgbuf.txt06000017726213115636557  7664 ustarrootwheelCopyright (c) 1992-2016 The FreeBSD Project.
                                                  Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
                                                  	The Regents of the University of California. All rights reserved.
                                                  FreeBSD is a registered trademark of The FreeBSD Foundation.
                                                  FreeBSD 11.0-RELEASE-p8 #78 3f54ce3814d(RELENG_2_4): Wed Mar 15 18:23:59 CDT 2017
                                                      root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense amd64
                                                  FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
                                                  VT(efifb): resolution 1920x1080
                                                  CPU: Intel(R) Celeron(R) CPU  N3150  @ 1.60GHz (1600.05-MHz K8-class CPU)
                                                    Origin="GenuineIntel"  Id=0x406c3  Family=0x6  Model=0x4c  Stepping=3
                                                    Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x43d8e3bf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand>AMD Features=0x28100800 <syscall,nx,rdtscp,lm>AMD Features2=0x101 <lahf,prefetch>Structured Extended Features=0x2282 <tscadj,smep,erms,nfpusg>VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID
                                                    TSC: P-state invariant, performance statistics
                                                  real memory  = 4294967296 (4096 MB)
                                                  avail memory = 3877691392 (3698 MB)
                                                  Event timer "LAPIC" quality 600
                                                  ACPI APIC Table: <alaska a="" m="" i="">WARNING: L1 data cache covers less APIC IDs than a core
                                                  0 < 1
                                                  FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
                                                  FreeBSD/SMP: 1 package(s) x 4 core(s)
                                                  random: unblocking device.
                                                  ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 128/32 (20160527/tbfadt-650)
                                                  ioapic0 <version 2.0="">irqs 0-114 on motherboard
                                                  random: entropy device external interface
                                                  wlan: mac acl policy registered
                                                  kbd0 at kbdmux0
                                                  netmap: loaded module
                                                  module_register_init: MOD_LOAD (vesa, 0xffffffff8122b980, 0) error 19
                                                  random: registering fast source Intel Secure Key RNG
                                                  random: fast provider: "Intel Secure Key RNG"
                                                  cryptosoft0: <software crypto="">on motherboard
                                                  padlock0: No ACE support.
                                                  acpi0: <alaska a="" m="" i="">on motherboard
                                                  acpi0: Power Button (fixed)
                                                  unknown: I/O range not supported
                                                  cpu0: <acpi cpu="">on acpi0
                                                  cpu1: <acpi cpu="">on acpi0
                                                  cpu2: <acpi cpu="">on acpi0
                                                  cpu3: <acpi cpu="">on acpi0
                                                  atrtc0: <at realtime="" clock="">port 0x70-0x77 on acpi0
                                                  atrtc0: Warning: Couldn't map I/O.
                                                  Event timer "RTC" frequency 32768 Hz quality 0
                                                  attimer0: <at timer="">port 0x40-0x43,0x50-0x53 irq 0 on acpi0
                                                  Timecounter "i8254" frequency 1193182 Hz quality 0
                                                  Event timer "i8254" frequency 1193182 Hz quality 100
                                                  Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
                                                  acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
                                                  pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
                                                  pci0: <acpi pci="" bus="">on pcib0
                                                  vgapci0: <vga-compatible display="">port 0xf000-0xf03f mem 0x80000000-0x80ffffff,0x90000000-0x9fffffff irq 16 at device 2.0 on pci0
                                                  vgapci0: Boot video device
                                                  ahci0: <ahci sata="" controller="">port 0xf060-0xf07f mem 0x81511000-0x815117ff irq 19 at device 19.0 on pci0
                                                  ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier not supported
                                                  ahcich0: <ahci channel="">at channel 0 on ahci0
                                                  ahcich1: <ahci channel="">at channel 1 on ahci0
                                                  xhci0: <xhci (generic)="" usb="" 3.0="" controller="">mem 0x81500000-0x8150ffff irq 20 at device 20.0 on pci0
                                                  xhci0: 32 bytes context size, 64-bit DMA
                                                  usbus0 on xhci0
                                                  pci0: <encrypt decrypt="">at device 26.0 (no driver attached)
                                                  pcib1: <acpi pci-pci="" bridge="">irq 16 at device 28.0 on pci0
                                                  pci1: <acpi pci="" bus="">on pcib1
                                                  re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet="">port 0xe000-0xe0ff mem 0xfffe04000-0xfffe04fff,0xfffe00000-0xfffe03fff irq 16 at device 0.0 on pci1
                                                  re0: Using 1 MSI-X message
                                                  re0: ASPM disabled
                                                  re0: Chip rev. 0x2c800000
                                                  re0: MAC rev. 0x00100000
                                                  miibus0: <mii bus="">on re0
                                                  rgephy0: <rtl8169s 8211="" 8110s="" 1000base-t="" media="" interface="">PHY 1 on miibus0
                                                  rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
                                                  re0: Using defaults for TSO: 65518/35/2048
                                                  re0: Ethernet address: 00:e0:4c:68:55:ae
                                                  re0: netmap queues/slots: TX 1/256, RX 1/256
                                                  pcib2: <acpi pci-pci="" bridge="">irq 17 at device 28.1 on pci0
                                                  pcib2: [GIANT-LOCKED]
                                                  pci2: <acpi pci="" bus="">on pcib2
                                                  re1: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet="">port 0xd000-0xd0ff mem 0xfffd04000-0xfffd04fff,0xfffd00000-0xfffd03fff irq 17 at device 0.0 on pci2
                                                  re1: Using 1 MSI-X message
                                                  re1: ASPM disabled
                                                  re1: Chip rev. 0x2c800000
                                                  re1: MAC rev. 0x00100000
                                                  miibus1: <mii bus="">on re1
                                                  rgephy1: <rtl8169s 8211="" 8110s="" 1000base-t="" media="" interface="">PHY 1 on miibus1
                                                  rgephy1:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
                                                  re1: Using defaults for TSO: 65518/35/2048
                                                  re1: Ethernet address: 00:e0:4c:68:55:af
                                                  re1: netmap queues/slots: TX 1/256, RX 1/256
                                                  pcib3: <acpi pci-pci="" bridge="">irq 18 at device 28.2 on pci0
                                                  pci3: <acpi pci="" bus="">on pcib3
                                                  igb0: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k="">port 0xc020-0xc03f mem 0x81380000-0x813fffff,0x81404000-0x81407fff irq 18 at device 0.0 on pci3
                                                  igb0: Using MSIX interrupts with 2 vectors
                                                  igb0: Ethernet address: 00:30:18:c3:d0:e5
                                                  igb0: netmap queues/slots: TX 1/4096, RX 1/4096
                                                  igb1: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k="">port 0xc000-0xc01f mem 0x81280000-0x812fffff,0x81400000-0x81403fff irq 19 at device 0.1 on pci3
                                                  igb1: Using MSIX interrupts with 2 vectors
                                                  igb1: Ethernet address: 00:30:18:c3:d0:e6
                                                  igb1: netmap queues/slots: TX 1/4096, RX 1/4096
                                                  isab0: <pci-isa bridge="">at device 31.0 on pci0
                                                  isa0: <isa bus="">on isab0
                                                  acpi_button0: <sleep button="">on acpi0
                                                  acpi_tz0: <thermal zone="">on acpi0
                                                  ppc0: cannot reserve I/O port range
                                                  est0: <enhanced speedstep="" frequency="" control="">on cpu0
                                                  est1: <enhanced speedstep="" frequency="" control="">on cpu1
                                                  est2: <enhanced speedstep="" frequency="" control="">on cpu2
                                                  est3: <enhanced speedstep="" frequency="" control="">on cpu3
                                                  usbus0: 5.0Gbps Super Speed USB v3.0
                                                  ZFS filesystem version: 5
                                                  ZFS storage pool version: features support (5000)
                                                  Timecounters tick every 1.000 msec
                                                  nvme cam probe device init
                                                  ugen0.1: <0x8086> at usbus0
                                                  uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
                                                  uhub0: 13 ports with 13 removable, self powered
                                                  ugen0.2: <american power="" conversion="">at usbus0
                                                  ugen0.3: <vendor 0x05e3="">at usbus0
                                                  uhub1: <vendor 2="" 9="" 0x05e3="" usb2.0="" hub,="" class="" 0,="" rev="" 2.00="" 85.36,="" addr="">on usbus0
                                                  uhub1: 4 ports with 4 removable, self powered
                                                  ugen0.4: <vendor 0x1997="">at usbus0
                                                  ukbd0: <vendor 0="" 3="" 0x1997="" mini="" keyboard,="" class="" 0,="" rev="" 1.10="" 20.12,="" addr="">on usbus0
                                                  kbd1 at ukbd0
                                                  ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
                                                  ada0: <kingston sms200s360g="" 608abbf0="">ATA8-ACS SATA 3.x device
                                                  ada0: Serial Number 50026B72630613B0
                                                  ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
                                                  ada0: Command Queueing enabled
                                                  ada0: 57241MB (117231408 512 byte sectors)
                                                  SMP: AP CPU #2 Launched!
                                                  SMP: AP CPU #1 Launched!
                                                  SMP: AP CPU #3 Launched!
                                                  Timecounter "TSC" frequency 1600052960 Hz quality 1000
                                                  Trying to mount root from zfs:zroot/ROOT/default []...
                                                  <118>Configuring crash dumps...
                                                  <118>Using /dev/ada0p3 for dump device.
                                                  <118>Filesystems are clean, continuing...
                                                  <118>Mounting filesystems...
                                                  <118>Restoring contents of RAM disk store... done.
                                                  <118>
                                                  <118>     ___
                                                  <118> ___/ f \
                                                  <118>/ p \___/ Sense
                                                  <118>\___/   \
                                                  <118>    \___/
                                                  <118>
                                                  <118>Welcome to pfSense 2.4.0-BETA...
                                                  <118>
                                                  <118>No core dumps found.
                                                  <118>chmod: /var/empty: Operation not permitted
                                                  <118>...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/dnscrypt-proxy /usr/local/lib/graphviz /usr/local/lib/ipsec /usr/local/lib/mysql /usr/local/lib/perl5/5.24/mach/CORE
                                                  <118>32-bit compatibility ldconfig path:
                                                  <118>done.
                                                  <118>External config loader 1.0 is now starting...
                                                  <118>Launching the init system....... done.
                                                  <118>Initializing.................. done.
                                                  <118>Starting device manager (devd)...kldload: can't load ums: No such file or directory
                                                  <118>done.
                                                  <118>Loading configuration......done.
                                                  <118>tar: Removing leading '/' from member names
                                                  <118>Updating configuration...done.
                                                  <118>Cleaning backup cache.................................done.
                                                  <118>Setting up extended sysctls...done.
                                                  padlock0: No ACE support.
                                                  aesni0: <aes-cbc,aes-xts,aes-gcm,aes-icm>on motherboard
                                                  coretemp0: <cpu on-die="" thermal="" sensors="">on cpu0
                                                  coretemp1: <cpu on-die="" thermal="" sensors="">on cpu1
                                                  coretemp2: <cpu on-die="" thermal="" sensors="">on cpu2
                                                  coretemp3: <cpu on-die="" thermal="" sensors="">on cpu3
                                                  <118>Setting timezone...done.
                                                  <118>Configuring loopback interface...done.
                                                  <118>Starting syslog...done.
                                                  <118>Starting Secure Shell Services...done.
                                                  <118>Setting up interfaces microcode...done.
                                                  <118>Configuring loopback interface...done.
                                                  <118>Creating wireless clone interfaces...done.
                                                  <118>Configuring LAGG interfaces...done.
                                                  <118>Configuring VLAN interfaces...done.
                                                  <118>Configuring QinQ interfaces...done.
                                                  <118>Configuring WAN interface...
                                                  <5>igb0: link state changed to UP
                                                  <118>done.
                                                  <118>Configuring LAN interface...done.
                                                  <118>Configuring CARP settings...done.
                                                  <118>Syncing OpenVPN settings...
                                                  <6>tun1: changing name to 'ovpnc1'
                                                  <6>tun2: changing name to 'ovpnc2'
                                                  <118>done.
                                                  <6>pflog0: promiscuous mode enabled
                                                  <118>Configuring firewall.....0 addresses deleted.
                                                  <118>0 addresses deleted.
                                                  <118>.done.
                                                  <118>Starting PFLOG...done.
                                                  <118>Setting up gateway monitors...done.
                                                  <5>ovpnc2: link state changed to UP
                                                  <5>ovpnc1: link state changed to UP
                                                  <5>igb1: link state changed to UP
                                                  <118>Starting DNS Resolver...done.
                                                  <118>Synchronizing user settings...done.
                                                  <118>Starting webConfigurator...done.
                                                  <118>Configuring CRON...done.
                                                  <118>Starting NTP time client...done.
                                                  <118>Starting DHCP service...done.
                                                  <118>Starting DHCPv6 service...done.
                                                  <118>Configuring firewall.....0 addresses deleted.
                                                  <118>0 addresses deleted.
                                                  <118>0 addresses deleted.
                                                  <118>0 addresses deleted.
                                                  <118>.done.
                                                  <5>igb0: link state changed to DOWN
                                                  <5>igb1: link state changed to DOWN
                                                  <5>igb0: link state changed to UP
                                                  <5>igb1: link state changed to UP
                                                  <118>Generating RRD graphs...done.
                                                  <118>Starting syslog...done.
                                                  <118>Starting CRON... done.
                                                  <118> Starting package Cron...done.
                                                  <118> Starting package iftop...done.
                                                  <118> Starting package iperf...done.
                                                  <118> Starting package mtr-nox11...done.
                                                  <118> Starting package nmap...done.
                                                  <118> Starting package pfBlockerNG...done.
                                                  <118> Starting package darkstat...done.
                                                  <118> Starting package ntopng...done.
                                                  <118> Starting package Shellcmd...done.
                                                  <118> Starting package System Patches...done.
                                                  <118> Starting package Backup...done.
                                                  <118> Starting package apcupsd...done.
                                                  <118> Starting package nut...done.
                                                  <118> Starting /usr/local/etc/rc.d/dnsbl.sh...done.
                                                  <118>pfSense 2.4.0-BETA amd64 Wed Mar 15 18:17:17 CDT 2017
                                                  <118>Bootup complete
                                                  <5>ovpnc1: link state changed to DOWN
                                                  <5>ovpnc2: link state changed to DOWN
                                                  <5>ovpnc2: link state changed to UP
                                                  <5>ovpnc1: link state changed to UP
                                                  <5>re0: link state changed to DOWN
                                                  <5>re1: link state changed to DOWN
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200a, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:401f:15::6, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4009:4c::a, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4009:4c::a, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4009:4c::a, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a01:578:3::36c2:327f, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:809::200a, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:816::200a, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200a, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:401f:31::6, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a01:578:3::3433:8b11, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:815::200a, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80a::200a, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:400c:c0a::5f, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:401f:15::6, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a01:578:3::3410:e6ed, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:809::200a, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:809::200a, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:811::200a, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:401f:15::6, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a01:578:3::34d2:92a8, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80a::200a, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
                                                  <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0</cpu></cpu></cpu></cpu></aes-cbc,aes-xts,aes-gcm,aes-icm></kingston></vendor></vendor></vendor></vendor></american></enhanced></enhanced></enhanced></enhanced></thermal></sleep></isa></pci-isa></intel(r)></intel(r)></acpi></acpi></rtl8169s></mii></realtek></acpi></acpi></rtl8169s></mii></realtek></acpi></acpi></encrypt></xhci></ahci></ahci></ahci></vga-compatible></acpi></acpi></at></at></acpi></acpi></acpi></acpi></alaska></software></version></alaska></tscadj,smep,erms,nfpusg></lahf,prefetch></syscall,nx,rdtscp,lm></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe> 
                                                  

                                                  pfSense 2.6.0 - ISP AAISP UK

                                                  1 Reply Last reply Reply Quote 0
                                                  • C
                                                    chrcoluk last edited by

                                                    and last bit

                                                    kernel trap 12 with interrupts disabled
                                                    
                                                    Fatal trap 12: page fault while in kernel mode
                                                    cpuid = 1; apic id = 02
                                                    fault virtual address	= 0x30
                                                    fault code		= supervisor read data, page not present
                                                    instruction pointer	= 0x20:0xffffffff80cb910c
                                                    stack pointer	        = 0x28:0xfffffe0113c6d450
                                                    frame pointer	        = 0x28:0xfffffe0113c6d480
                                                    code segment		= base 0x0, limit 0xfffff, type 0x1b
                                                    			= DPL 0, pres 1, long 1, def32 0, gran 1
                                                    processor eflags	= resume, IOPL = 0
                                                    current process		= 54120 (sh)
                                                    version.txt06000025413115636557  7630 ustarrootwheelFreeBSD 11.0-RELEASE-p8 #78 3f54ce3814d(RELENG_2_4): Wed Mar 15 18:23:59 CDT 2017
                                                        root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense
                                                    

                                                    pfSense 2.6.0 - ISP AAISP UK

                                                    1 Reply Last reply Reply Quote 0
                                                    • C
                                                      chrcoluk last edited by

                                                      another forum with same backtrace actually has led me to this unresolved bug report on FreeBSD,  has been confirmed to affect 11 as well as 12.

                                                      So I think this might be what I have

                                                      https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

                                                      Have compiled the pfsense kernel now with the patch, will report back here if all good or not, only just booted into it so need to give it some time.

                                                      pfSense 2.6.0 - ISP AAISP UK

                                                      1 Reply Last reply Reply Quote 0
                                                      • First post
                                                        Last post