Kernel lockup



  • Had a kernel lockup this morning, extract from logs below (note log is set to have newest line on top)

    Jan 4 10:46:17	kernel		Fatal trap 12: page fault while in kernel mode
    Jan 4 10:46:17	kernel		kernel trap 12 with interrupts disabled
    Jan 4 10:46:17	kernel		current process	= 97781 (sh)
    Jan 4 10:46:17	kernel		processor eflags	= resume, IOPL = 0
    Jan 4 10:46:17	kernel		= DPL 0, pres 1, long 1, def32 0, gran 1
    Jan 4 10:46:17	kernel		code segment	= base 0x0, limit 0xfffff, type 0x1b
    Jan 4 10:46:17	kernel		frame pointer	= 0x28:0xfffffe01181b2480
    Jan 4 10:46:17	kernel		stack pointer	= 0x28:0xfffffe01181b2450
    Jan 4 10:46:17	kernel		instruction pointer	= 0x20:0xffffffff80cb908c
    Jan 4 10:46:17	kernel		fault code	= supervisor read data, page not present
    Jan 4 10:46:17	kernel		fault virtual address	= 0x30
    Jan 4 10:46:17	kernel		cpuid = 0; apic id = 00
    

    Had no lockups on 2.2 but I also havent added load to the box until the past week.

    I undid some tunables I set for the nic in loader.conf.custom, and disabled MSIX.  Will see if it stabilises, if I get more issues, I will test the ram.

    If needed I can swap the ram as its the same exact sodimm that I had put in my laptop last year, so I can remove from laptop to put in the pfsense unit if I find the ram faulty.


  • Rebel Alliance Developer Netgate

    Was there no other data from the crash? No crash report or backtrace?

    There is not enough information in those log messages to make any guesses about a cause.



  • Hi Jimp

    Did you have any thoughts about the one I had and posted which was after the vpn fix.

    Thanks



  • jimp where do I look for the backtrace? I will check /var/crash if its same as default FreeBSD behaviour.

    right I read this

    https://doc.pfsense.org/index.php/Unexpected_Reboot_Troubleshooting

    yeah, I did disable the swap partition (it does exist tho), so I will enable it again just in case I get another panic.


  • Rebel Alliance Developer Netgate

    The firewall should present a box on top of the dashboard with a link to see the crash report if it was able to collect one.



  • edited my post, sorry I edited as you replied.



  • Well it looks like you are the second person having this issue.

    I posted a thread earlier and got NO reply.

    https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213856

    That is the bug and I have sent multiple crash dumps already.


  • Rebel Alliance Developer Netgate

    Just because you have a "Fatal Trap 12" doesn't mean it's the same bug. The stack trace doesn't look anywhere near the same.



  • @Dazog:

    Well it looks like you are the second person having this issue.

    I posted a thread earlier and got NO reply.

    https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213856

    That is the bug and I have sent multiple crash dumps already.

    panic can be caused by many things, this is why I didnt jump in claiming its a bug, as could be hardware or software caused.
    My fault I had disabled the swap partition (I had configured swap file instead for SSD trim reasons).  So the lack of backtrace is because of that.

    FreeBSD historically has needed issues resolving in a .0 release, this could be one of those issues, but it could be hardware as well.

    Also you posted directly to FreeBSD developers not on pfsense bug report.



  • no more panic's but given the number of people who reported panic's on FreeBSD 11 reporting it been fixed with the igb queue count set to 1 or 2, I adjusted it to 2 for safety.  I have enabled MSIX again.



  • @chrcoluk:

    no more panic's but given the number of people who reported panic's on FreeBSD 11 reporting it been fixed with the igb queue count set to 1 or 2, I adjusted it to 2 for safety.  I have enabled MSIX again.

    What "it" been fixed? Is there some ticket on freebsd community regarding this problem?



  • yeah there is a ticket but I dont have the url to hand sorry, everyone who was on the ticket reported panics been fixed with setting it to 1.

    Only 1 guy said it was also stable on 2 tho and he posted that on here somewhere.  The default is to match core count so in my case default is 4.

    hw.igb.num_queues=2



  • I reinstalled pfSense 2.4 2 days ago using zfs, I made sure I have the loader.conf.local in place to reduce igb queues to 1 but I have had a panic late last night, again no dump was made so no backtrace, this time tho a swap partition was enabled.

    Will scan my ram.
    Will disable adaptive mode for cpu so it stays at one speed only as cpu clock fluctuations can cause instabilities.
    Will check bios to see if anything in there that could cause compatibility issues.

    I do notice on bootup that the boot up log says its not using any dump device, so I will try manually specifying it in rc.conf.local to see if it reports in the boot log one is set.

    I wonder if it may be a good idea to track 11-STABLE instead of 11.0 as .0 releases of FreeBSD can be problematic.



  • @chrcoluk:

    I wonder if it may be a good idea to track 11-STABLE instead of 11.0 as .0 releases of FreeBSD can be problematic.

    The stable branch is a much more moving target than the releng branch, it would be hell a lot more work to keep up with the changes.



  • Have done the 2 ram tests I planned.

    Also have the dump device correctly showing during the boot process now so if I have another panic I should get a dump to backtrace.

    I have disabled turbo mode and EIST on the cpu and dynamic voltage on the ram.



  • chrcoluk what hardware are you running pfSense® on?  Your sig says "Braswell N3150" but that isn't specific.  Sorry if I missed it in another post.


  • Banned

    @luckman212:

    chrcoluk what hardware are you running pfSense® on?

    That is unacceptable usage. You must say "What hardware are you running pfSense® software on?"

    ;D ;D ::) ::)

    SCNR. Won't further comment on this new round of legal bleep. I hoped the unfortunate episode ~2+ years ago was enough of a lesson. Apparently not.



  • chrcoluk
    Did you add kern.ipc.nmbclusters="1000000"?
    What is your WAN bandwidth?



  • I dont know the exact hardware packaged as I purchased a compact unit from amazon which is simply labeled as a BRASWELL N3150 unit, which tells you I have a celeron N3150 on a braswell chipset.

    I think I am going to adjust the bios again to enable EIST and turbo mode but also configure powerd to never let the clocks drop below the stock max speed that means I get turbo mode back again for extra performance whilst not letting it drop to idle voltages, I will keep the dynamic voltages disabled on the ram.

    Also nmbclusters has been increased in the loader.conf from almost day one of using pfsense.

    WAN bandwidth at the time of the crash was just idle, the absolute max bandwidth usage possible is about 70mbit down and 20mbit up.



  • Ok, try to bump voltage on memory, +0.1V



  • I dont have that control in the bios.

    But I have disabled now dynamic voltage mode (DVFS) so it should always be at full voltage now instead of power saving lower voltages.



  • Am getting issues again, the good news is I am getting proper kernel dumps now, the last two I hit the submit button, but to let more eyes look at the problem I will dump the data here as well, the bit of research I have done seems to indicate its filesystem related, but I am most definitely not sure on this so appreciate an opinion from anyone.

    In the meantime since my laptop uses the same spec ram I am going to swap the ram with my laptop memory, even tho this existing ram has passed stability tests.

    					Crash report begins.  Anonymous machine information:
    
    amd64
    11.0-RELEASE-p8
    FreeBSD 11.0-RELEASE-p8 #78 3f54ce3814d(RELENG_2_4): Wed Mar 15 18:23:59 CDT 2017     root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense
    
    Crash report details:
    
    No PHP errors found.
    
    Filename: /var/crash/bounds
    1
    
    Filename: /var/crash/info.0
    Dump header from device: /dev/ada0p3
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 123392
      Blocksize: 512
      Dumptime: Wed Jun  7 00:40:31 2017
      Hostname: PFSENSE.home
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 11.0-RELEASE-p8 #78 3f54ce3814d(RELENG_2_4): Wed Mar 15 18:23:59 CDT 2017
        root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense
      Panic String: 
      Dump Parity: 318326786
      Bounds: 0
      Dump Status: good
    
    Filename: /var/crash/info.last
    Dump header from device: /dev/ada0p3
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 123392
      Blocksize: 512
      Dumptime: Wed Jun  7 00:40:31 2017
      Hostname: PFSENSE.home
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 11.0-RELEASE-p8 #78 3f54ce3814d(RELENG_2_4): Wed Mar 15 18:23:59 CDT 2017
        root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense
      Panic String: 
      Dump Parity: 318326786
      Bounds: 0
      Dump Status: good
    
    Filename: /var/crash/textdump.tar.0
    ddb.txt06000014000013115636557  7106 ustarrootwheeldb:0:kdb.enter.default>  run lockinfo
    db:1:lockinfo> show locks
    No such command
    db:1:locks>  show alllocks
    No such command
    db:1:alllocks>  show lockedvnods
    Locked vnodes
    db:0:kdb.enter.default>  show pcpu
    cpuid        = 1
    dynamic pcpu = 0xfffffe0167de2000
    curthread    = 0xfffff8015e162a00: pid 54120 "sh"
    curpcb       = 0xfffffe0113c6dcc0
    fpcurthread  = 0xfffff8015e162a00: pid 54120 "sh"
    idlethread   = 0xfffff8000490aa00: tid 100004 "idle: cpu1"
    curpmap      = 0xfffff801621b3138
    tssp         = 0xffffffff82a1dd78
    commontssp   = 0xffffffff82a1dd78
    rsp0         = 0xfffffe0113c6dcc0
    gs32p        = 0xffffffff82a245d0
    ldt          = 0xffffffff82a24610
    tss          = 0xffffffff82a24600
    db:0:kdb.enter.default>  bt
    Tracing pid 54120 tid 100774 td 0xfffff8015e162a00
    turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0113c6d480
    __rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0113c6d4b0
    vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0113c6d530
    vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0113c6d560
    exec_new_vmspace() at exec_new_vmspace+0x22f/frame 0xfffffe0113c6d5e0
    exec_elf64_imgact() at exec_elf64_imgact+0xa58/frame 0xfffffe0113c6d6f0
    kern_execve() at kern_execve+0x74d/frame 0xfffffe0113c6da50
    sys_execve() at sys_execve+0x4a/frame 0xfffffe0113c6dad0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c6dbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c6dbf0
    --- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x800b40d8a, rsp = 0x7fffffffe188, rbp = 0x7fffffffe2d0 ---
    db:0:kdb.enter.default>  ps
      pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
    54226 53957   334     0  R       CPU 0                       sh
    54120 53957   334     0  R       CPU 1                       sh
    54035 53957   334     0  R       CPU 2                       cat
    53957 33060   334     0  R       CPU 3                       sh
    39824   334   334     0  S       accept   0xfffff8003b01006c php-fpm
    43353 32987 43353     0  S+      select   0xfffff801629be7c0 top
    32987  2745 32987     0  S+      wait     0xfffff8003b194528 bash
     2745  2354  2745     0  Ss+     wait     0xfffff8003b03d528 sh
     2354  8180  2354     0  Ss      select   0xfffff80102b6a5c0 sshd
    21591     1   334     0  S       nanslp   0xffffffff82865d21 php
    21318     1 21318     0  Ss      select   0xfffff80102d507c0 upsd
    21063     1 21063    66  Ss      select   0xfffff8003b13e6c0 apcupsd-ups
    19999 19580 19580    66  S       nanslp   0xffffffff82865d21 upsmon
    19580     1 19580     0  Ss      piperd   0xfffff80162a03be0 upsmon
    33060     1   334     0  S       piperd   0xfffff8015e2bf8e8 sh
    15682     1 15682   136  Ss      select   0xfffff8001d948240 dhcpd
    88257     1 88257    59  Ss      (threaded)                  unbound
    100711                   S       kqread   0xfffff8015eee6c00 unbound
    100760                   S       kqread   0xfffff80011585800 unbound
    100761                   S       kqread   0xfffff80011585a00 unbound
    100762                   S       kqread   0xfffff800101aea00 unbound
    55992     1 55992     0  Ss      (threaded)                  dpinger
    100681                   S       uwait    0xfffff80162823e00 dpinger
    100729                   S       sbwait   0xfffff8016285c4a4 dpinger
    100730                   S       nanslp   0xffffffff82865d20 dpinger
    100731                   S       nanslp   0xffffffff82865d21 dpinger
    100732                   S       accept   0xfffff80162d3d72c dpinger
    55571     1 55571     0  Ss      (threaded)                  dpinger
    100701                   S       uwait    0xfffff8001d4c3b80 dpinger
    100725                   S       sbwait   0xfffff80162a2f144 dpinger
    100726                   S       nanslp   0xffffffff82865d22 dpinger
    100727                   S       nanslp   0xffffffff82865d20 dpinger
    100728                   S       accept   0xfffff801620e2a8c dpinger
    89259     1 89259     0  Ss      select   0xfffff8001d4c1e40 openvpn
    87846     1 87846     0  Ss      select   0xfffff8016220e7c0 openvpn
    68550     1 68550     0  Ss      select   0xfffff80102d520c0 powerd
    67165 67155 67165     0  S+      ttyin    0xfffff8001d3dfca8 sh
    67155 65512 67155     0  S+      wait     0xfffff8016212f528 sh
    66898 98897 66898     0  Ss      (threaded)                  sshlockout_pf
    100605                   S       piperd   0xfffff8001d9db000 sshlockout_pf
    100683                   S       nanslp   0xffffffff82865d20 sshlockout_pf
    66742     1 66742     0  Ss+     ttyin    0xfffff8001d46b0a8 getty
    66646     1 66646     0  Ss+     ttyin    0xfffff8001d46b4a8 getty
    66575     1 66575     0  Ss+     ttyin    0xfffff8001d46b8a8 getty
    66538     1 66538     0  Ss+     ttyin    0xfffff8001d46bca8 getty
    66211     1 66211     0  Ss+     ttyin    0xfffff8001d46c0a8 getty
    66158     1 66158     0  Ss+     ttyin    0xfffff80005179ca8 getty
    65836     1 65836     0  Ss+     ttyin    0xfffff8001d3df0a8 getty
    65512     1 65512     0  Ss+     wait     0xfffff8001d976a50 login
    50681 50482 50482     0  S       nanslp   0xffffffff82865d20 minicron
    50482     1 50482     0  Ss      wait     0xfffff8001df18a50 minicron
    50272 49938 49938     0  S       nanslp   0xffffffff82865d23 minicron
    49938     1 49938     0  Ss      wait     0xfffff8001d976528 minicron
    49476 49314 49314     0  S       nanslp   0xffffffff82865d23 minicron
    49314     1 49314     0  Ss      wait     0xfffff8016212aa50 minicron
    44216     1 44216     0  Ss      (threaded)                  apcupsd
    100658                   S       nanslp   0xffffffff82865d21 apcupsd
    100670                   S       sigwait  0xfffff80162a01000 apcupsd
    100671                   S       accept   0xfffff8016212206c apcupsd
     3948     1  3848     0  S       kqread   0xfffff8001dc25200 lighttpd_pfb
    98897     1 98897     0  Ss      select   0xfffff80102d509c0 syslogd
    53013     1 53013     0  Ss      kqread   0xfffff8001dbe1600 dnscrypt-proxy
    52232     1 52232     0  Ss      kqread   0xfffff8001db85000 dnscrypt-proxy
    50361     1 50361     0  Ss      kqread   0xfffff8001df2d900 dnscrypt-proxy
    47847     1 47764     0  S       select   0xfffff8003b3464c0 radvd
    40799     1 40799     0  Ss      (threaded)                  ntpd
    100646                   S       select   0xfffff8003b1408c0 ntpd
    40296     0     0     0  DL      mdwait   0xfffff80102bf3000 [md3]
    38103     1 38103     0  Ss      nanslp   0xffffffff82865d22 cron
    37771 37268 37268     0  S       kqread   0xfffff8001dc40600 nginx
    37735 37268 37268     0  S       kqread   0xfffff8001d5b0700 nginx
    37705 37268 37268     0  S       kqread   0xfffff8001dc1c900 nginx
    37554 37268 37268     0  S       kqread   0xfffff8001dbbaa00 nginx
    37494 37268 37268     0  S       kqread   0xfffff8001db43a00 nginx
    37268     1 37268     0  Ss      pause    0xfffff80102eecaf8 nginx
    20138     1 20138     0  Ss      bpf      0xfffff8003bd67000 filterlog
    15349     1 15349     0  Ss      select   0xfffff8003b13e1c0 dhcp6c
    14626     1 14626    65  Ss      select   0xfffff8001dc38ec0 dhclient
     9309     1  9309     0  Ss      select   0xfffff8003b3460c0 dhclient
     8519     1  8519     0  Ss      (threaded)                  sshlockout_pf
    100597                   S       uwait    0xfffff8001dc37600 sshlockout_pf
    100601                   S       nanslp   0xffffffff82865d20 sshlockout_pf
     8180     1  8180     0  Ss      select   0xfffff8001d4c2640 sshd
      393     0     0     0  DL      mdwait   0xfffff8001daf1800 [md2]
      383     0     0     0  DL      mdwait   0xfffff8003b132000 [md1]
      368     1   368     0  Ss      select   0xfffff8003b1411c0 devd
      350   348   348     0  S       kqread   0xfffff8001dcabd00 check_reload_status
      348     1   348     0  Ss      kqread   0xfffff8001db94800 check_reload_status
      334     1   334     0  Ss      kqread   0xfffff8001ddd2100 php-fpm
      103     0     0     0  DL      mdwait   0xfffff8001d565000 [md0]
       27     0     0     0  DL      syncer   0xffffffff8295f880 [syncer]
       26     0     0     0  DL      vlruwt   0xfffff8001d406528 [vnlru]
       25     0     0     0  DL      psleep   0xffffffff8295e104 [bufdaemon]
       24     0     0     0  DL      -        0xffffffff8295edb4 [bufspacedaemon]
       23     0     0     0  DL      pgzero   0xffffffff829734e4 [pagezero]
       22     0     0     0  DL      psleep   0xffffffff8296f98c [vmdaemon]
       21     0     0     0  DL      (threaded)                  [pagedaemon]
    100072                   D       psleep   0xffffffff82a1d105 [pagedaemon]
    100081                   D       umarcl   0xffffffff8296f2b8 [uma]
       20     0     0     0  DL      -        0xffffffff8295d914 [soaiod4]
       19     0     0     0  DL      -        0xffffffff8295d914 [soaiod3]
       18     0     0     0  DL      -        0xffffffff8295d914 [soaiod2]
        9     0     0     0  DL      -        0xffffffff8295d914 [soaiod1]
       17     0     0     0  DL      cooling  0xfffff80005139f58 [acpi_cooling0]
       16     0     0     0  DL      tzpoll   0xffffffff82621138 [acpi_thermal]
        8     0     0     0  DL      -        0xffffffff8273a4f0 [rand_harvestq]
        7     0     0     0  DL      pftm     0xffffffff80f4b6a0 [pf purge]
        6     0     0     0  DL      waiting_ 0xffffffff82a0c740 [sctp_iterator]
        5     0     0     0  DL      (threaded)                  [zfskern]
    100055                   D       arc_recl 0xffffffff82f3f4d0 [arc_reclaim_thread]
    100056                   D       arc_user 0xffffffff82f3f4f0 [arc_user_evicts_thr]
    100057                   D       arc_dnlc 0xffffffff82f3f480 [arc_dnlc_evicts_thr]
    100058                   D       l2arc_fe 0xffffffff82f3f888 [l2arc_feed_thread]
    100550                   D       spa->spa 0xfffff8001d55fbd0 [trim zroot]
    100564                   D       tx->tx_q 0xfffff8001d45fa30 [txg_thread_enter]
    100565                   D       tx->tx_s 0xfffff8001d45fa10 [txg_thread_enter]
       15     0     0     0  DL      (threaded)                  [usb]
    100035                   D       -        0xfffffe0001121460 [usbus0]
    100036                   D       -        0xfffffe00011214b8 [usbus0]
    100037                   D       -        0xfffffe0001121510 [usbus0]
    100038                   D       -        0xfffffe0001121568 [usbus0]
    100039                   D       -        0xfffffe00011215c0 [usbus0]
        4     0     0     0  DL      (threaded)                  [cam]
    100032                   D       -        0xffffffff8260d380 [doneq0]
    100067                   D       -        0xffffffff8260d1c8 [scanner]
        3     0     0     0  DL      crypto_r 0xffffffff8296de70 [crypto returns]
        2     0     0     0  DL      crypto_w 0xffffffff8296dd18 [crypto]
       14     0     0     0  DL      (threaded)                  [geom]
    100023                   D       -        0xffffffff829e4940 [g_event]
    100024                   D       -        0xffffffff829e4948 [g_up]
    100025                   D       -        0xffffffff829e4950 [g_down]
       13     0     0     0  DL      (threaded)                  [ng_queue]
    100019                   D       sleep    0xffffffff825cb230 [ng_queue0]
    100020                   D       sleep    0xffffffff825cb230 [ng_queue1]
    100021                   D       sleep    0xffffffff825cb230 [ng_queue2]
    100022                   D       sleep    0xffffffff825cb230 [ng_queue3]
       12     0     0     0  WL      (threaded)                  [intr]
    100007                   I                                   [swi3: vm]
    100008                   I                                   [swi4: clock (0)]
    100009                   I                                   [swi4: clock (1)]
    100010                   I                                   [swi4: clock (2)]
    100011                   I                                   [swi4: clock (3)]
    100012                   I                                   [swi1: netisr 0]
    100013                   I                                   [swi5: fast taskq]
    100015                   I                                   [swi6: task queue]
    100016                   I                                   [swi6: Giant taskq]
    100033                   I                                   [irq256: ahci0]
    100034                   I                                   [irq257: xhci0]
    100040                   I                                   [irq258: re0]
    100041                   I                                   [irq259: pcib2]
    100042                   I                                   [irq260: re1]
    100043                   I                                   [irq261: igb0:que 0]
    100045                   I                                   [irq262: igb0:link]
    100046                   I                                   [irq263: igb1:que 0]
    100048                   I                                   [irq264: igb1:link]
    100061                   I                                   [swi1: pf send]
    100062                   I                                   [swi1: pfsync]
    100079                   I                                   [swi1: netisr 1]
    100080                   I                                   [swi1: netisr 2]
    100082                   I                                   [swi1: netisr 3]
       11     0     0     0  RL      (threaded)                  [idle]
    100003                   CanRun                              [idle: cpu0]
    100004                   CanRun                              [idle: cpu1]
    100005                   CanRun                              [idle: cpu2]
    100006                   CanRun                              [idle: cpu3]
        1     0     1     0  SLs     wait     0xfffff80004908528 [init]
       10     0     0     0  DL      audit_wo 0xffffffff82a138c0 [audit]
        0     0     0     0  DLs     (threaded)                  [kernel]
    100000                   D       swapin   0xffffffff829e4978 [swapper]
    100014                   D       -        0xfffff8000491ee00 [kqueue_ctx taskq]
    100017                   D       -        0xfffff8000491e600 [aiod_kick taskq]
    100018                   D       -        0xfffff8000491e400 [thread]
    100026                   D       -        0xfffff8000491d600 [firmware taskq]
    100029                   D       -        0xfffff80004ae4d00 [acpi_task_0]
    100030                   D       -        0xfffff80004ae4d00 [acpi_task_1]
    100031                   D       -        0xfffff80004ae4d00 [acpi_task_2]
    100044                   D       -        0xfffff80005106b00 [igb0 que]
    100047                   D       -        0xfffff80004ba4d00 [igb1 que]
    100049                   D       -        0xfffff80005162800 [system_taskq_0]
    100050                   D       -        0xfffff80005162800 [system_taskq_1]
    100051                   D       -        0xfffff80005162800 [system_taskq_2]
    100052                   D       -        0xfffff80005162800 [system_taskq_3]
    100053                   D       -        0xfffff80004ae5000 [mca taskq]
    100054                   D       -        0xfffff80005161b00 [dbu_evict]
    100066                   D       -        0xfffff80004ae4900 [CAM taskq]
    100083                   D       -        0xfffff8001d467300 [if_io_tqg_0]
    100084                   D       -        0xfffff8001d467100 [if_io_tqg_1]
    100085                   D       -        0xfffff8001d4a4b00 [if_io_tqg_2]
    100086                   D       -        0xfffff8001d4a4900 [if_io_tqg_3]
    100087                   D       -        0xfffff8001d4a4700 [if_config_tqg_0]
    100322                   D       -        0xfffff8001d5fed00 [zio_null_issue]
    100323                   D       -        0xfffff8001d5b2600 [zio_null_intr]
    100324                   D       -        0xfffff8001d5b2300 [zio_read_issue_0]
    100325                   D       -        0xfffff8001d5b2300 [zio_read_issue_1]
    100326                   D       -        0xfffff8001d5b2300 [zio_read_issue_2]
    100327                   D       -        0xfffff8001d5b2300 [zio_read_issue_3]
    100328                   D       -        0xfffff8001d5b2300 [zio_read_issue_4]
    100329                   D       -        0xfffff8001d5b2300 [zio_read_issue_5]
    100330                   D       -        0xfffff8001d5b2300 [zio_read_issue_6]
    100331                   D       -        0xfffff8001d5b2300 [zio_read_issue_7]
    100332                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_0]
    100333                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_1]
    100334                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_2]
    100335                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_3]
    100336                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_4]
    100337                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_5]
    100338                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_6]
    100339                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_7]
    100340                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_8]
    100341                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_9]
    100342                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_10]
    100343                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_11]
    100344                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_0]
    100345                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_1]
    100346                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_2]
    100347                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_3]
    100348                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_4]
    100349                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_5]
    100350                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_6]
    100351                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_7]
    100352                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_8]
    100353                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_9]
    100354                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_10]
    100355                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_11]
    100356                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_0]
    100357                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_1]
    100358                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_2]
    100359                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_3]
    100360                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_4]
    100361                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_5]
    100362                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_6]
    100363                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_7]
    100364                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_8]
    100365                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_9]
    100366                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_10]
    100367                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_11]
    100368                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_0]
    100369                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_1]
    100370                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_2]
    100371                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_3]
    100372                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_4]
    100373                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_5]
    100374                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_6]
    100375                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_7]
    100376                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_8]
    100377                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_9]
    100378                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_10]
    100379                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_11]
    100380                   D       -        0xfffff8001d552500 [zio_read_intr_4_0]
    100381                   D       -        0xfffff8001d552500 [zio_read_intr_4_1]
    100382                   D       -        0xfffff8001d552500 [zio_read_intr_4_2]
    100383                   D       -        0xfffff8001d552500 [zio_read_intr_4_3]
    100384                   D       -        0xfffff8001d552500 [zio_read_intr_4_4]
    100385                   D       -        0xfffff8001d552500 [zio_read_intr_4_5]
    100386                   D       -        0xfffff8001d552500 [zio_read_intr_4_6]
    100387                   D       -        0xfffff8001d552500 [zio_read_intr_4_7]
    100388                   D       -        0xfffff8001d552500 [zio_read_intr_4_8]
    100389                   D       -        0xfffff8001d552500 [zio_read_intr_4_9]
    100390                   D       -        0xfffff8001d552500 [zio_read_intr_4_10]
    100391                   D       -        0xfffff8001d552500 [zio_read_intr_4_11]
    100392                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_0]
    100393                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_1]
    100394                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_2]
    100395                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_3]
    100396                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_4]
    100397                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_5]
    100398                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_6]
    100399                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_7]
    100400                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_8]
    100401                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_9]
    100402                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_10]
    100403                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_11]
    100404                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_0]
    100405                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_1]
    100406                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_2]
    100407                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_3]
    100408                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_4]
    100409                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_5]
    100410                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_6]
    100411                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_7]
    100412                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_8]
    100413                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_9]
    100414                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_10]
    100415                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_11]
    100416                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_0]
    100417                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_1]
    100418                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_2]
    100419                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_3]
    100420                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_4]
    100421                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_5]
    100422                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_6]
    100423                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_7]
    100424                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_8]
    100425                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_9]
    100426                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_10]
    100427                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_11]
    100428                   D       -        0xfffff8001d589300 [zio_write_issue_0]
    100429                   D       -        0xfffff8001d589300 [zio_write_issue_1]
    100430                   D       -        0xfffff8001d589300 [zio_write_issue_2]
    100431                   D       -        0xfffff8001d589200 [zio_write_issue_hig]
    100432                   D       -        0xfffff8001d589200 [zio_write_issue_hig]
    100433                   D       -        0xfffff8001d589200 [zio_write_issue_hig]
    100434                   D       -        0xfffff8001d589200 [zio_write_issue_hig]
    100435                   D       -        0xfffff8001d589200 [zio_write_issue_hig]
    100436                   D       -        0xfffff8001d589000 [zio_write_intr_0]
    100437                   D       -        0xfffff8001d589000 [zio_write_intr_1]
    100438                   D       -        0xfffff8001d589000 [zio_write_intr_2]
    100439                   D       -        0xfffff8001d589000 [zio_write_intr_3]
    100440                   D       -        0xfffff8001d589000 [zio_write_intr_4]
    100441                   D       -        0xfffff8001d589000 [zio_write_intr_5]
    100442                   D       -        0xfffff8001d589000 [zio_write_intr_6]
    100443                   D       -        0xfffff8001d589000 [zio_write_intr_7]
    100444                   D       -        0xfffff8001d588100 [zio_write_intr_high]
    100445                   D       -        0xfffff8001d588100 [zio_write_intr_high]
    100446                   D       -        0xfffff8001d588100 [zio_write_intr_high]
    100447                   D       -        0xfffff8001d588100 [zio_write_intr_high]
    100448                   D       -        0xfffff8001d588100 [zio_write_intr_high]
    100449                   D       -        0xfffff8001d587800 [zio_free_issue_0_0]
    100450                   D       -        0xfffff8001d587800 [zio_free_issue_0_1]
    100451                   D       -        0xfffff8001d587800 [zio_free_issue_0_2]
    100452                   D       -        0xfffff8001d587800 [zio_free_issue_0_3]
    100453                   D       -        0xfffff8001d587800 [zio_free_issue_0_4]
    100454                   D       -        0xfffff8001d587800 [zio_free_issue_0_5]
    100455                   D       -        0xfffff8001d587800 [zio_free_issue_0_6]
    100456                   D       -        0xfffff8001d587800 [zio_free_issue_0_7]
    100457                   D       -        0xfffff8001d587800 [zio_free_issue_0_8]
    100458                   D       -        0xfffff8001d587800 [zio_free_issue_0_9]
    100459                   D       -        0xfffff8001d587800 [zio_free_issue_0_10]
    100460                   D       -        0xfffff8001d587800 [zio_free_issue_0_11]
    100461                   D       -        0xfffff8001d587100 [zio_free_issue_1_0]
    100462                   D       -        0xfffff8001d587100 [zio_free_issue_1_1]
    100463                   D       -        0xfffff8001d587100 [zio_free_issue_1_2]
    100464                   D       -        0xfffff8001d587100 [zio_free_issue_1_3]
    100465                   D       -        0xfffff8001d587100 [zio_free_issue_1_4]
    100466                   D       -        0xfffff8001d587100 [zio_free_issue_1_5]
    100467                   D       -        0xfffff8001d587100 [zio_free_issue_1_6]
    100468                   D       -        0xfffff8001d587100 [zio_free_issue_1_7]
    100469                   D       -        0xfffff8001d587100 [zio_free_issue_1_8]
    100470                   D       -        0xfffff8001d587100 [zio_free_issue_1_9]
    100471                   D       -        0xfffff8001d587100 [zio_free_issue_1_10]
    100472                   D       -        0xfffff8001d587100 [zio_free_issue_1_11]
    100473                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_0]
    100474                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_1]
    100475                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_2]
    100476                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_3]
    100477                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_4]
    100478                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_5]
    100479                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_6]
    100480                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_7]
    100481                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_8]
    100482                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_9]
    100483                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_10]
    100484                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_11]
    100485                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_0]
    100486                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_1]
    100487                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_2]
    100488                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_3]
    100489                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_4]
    100490                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_5]
    100491                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_6]
    100492                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_7]
    100493                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_8]
    100494                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_9]
    100495                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_10]
    100496                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_11]
    100497                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_0]
    100498                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_1]
    100499                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_2]
    100500                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_3]
    100501                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_4]
    100502                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_5]
    100503                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_6]
    100504                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_7]
    100505                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_8]
    100506                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_9]
    100507                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_10]
    100508                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_11]
    100509                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_0]
    100510                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_1]
    100511                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_2]
    100512                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_3]
    100513                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_4]
    100514                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_5]
    100515                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_6]
    [/thread]
    


  • continued here - hit post limit

    100516                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_7]
    100517                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_8]
    100518                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_9]
    100519                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_10]
    100520                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_11]
    100521                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_0]
    100522                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_1]
    100523                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_2]
    100524                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_3]
    100525                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_4]
    100526                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_5]
    100527                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_6]
    100528                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_7]
    100529                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_8]
    100530                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_9]
    100531                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_10]
    100532                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_11]
    100533                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_0]
    100534                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_1]
    100535                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_2]
    100536                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_3]
    100537                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_4]
    100538                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_5]
    100539                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_6]
    100540                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_7]
    100541                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_8]
    100542                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_9]
    100543                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_10]
    100544                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_11]
    100545                   D       -        0xfffff8001d5db700 [zio_free_intr]
    100546                   D       -        0xfffff8001d5db300 [zio_claim_issue]
    100547                   D       -        0xfffff8001d5db600 [zio_claim_intr]
    100548                   D       -        0xfffff8001d5dae00 [zio_ioctl_issue]
    100549                   D       -        0xfffff8001d5dab00 [zio_ioctl_intr]
    100551                   D       -        0xfffff8001d5dac00 [metaslab_group_task]
    100552                   D       -        0xfffff8001d5dac00 [metaslab_group_task]
    100553                   D       -        0xfffff8001d5da000 [zfs_vn_rele_taskq]
    100566                   D       -        0xfffff8001d8dd200 [zil_clean]
    100572                   D       -        0xfffff8001daa6600 [zil_clean]
    100573                   D       -        0xfffff8001d9d8a00 [zil_clean]
    100574                   D       -        0xfffff8001d6fd100 [zil_clean]
    oops, ran out of processes early!
    db:0:kdb.enter.default>  alltrace
    
    Tracing command sh pid 54226 tid 100593 td 0xfffff8001dece000
    cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff82977730
    ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xffffffff82977750
    trap() at trap+0x3a/frame 0xffffffff82977960
    nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff82977960
    --- trap 0x13, rip = 0xffffffff811a4a14, rsp = 0xfffffe0113a4f490, rbp = 0xfffffe0113a4f560 ---
    pmap_remove_pages() at pmap_remove_pages+0x384/frame 0xfffffe0113a4f560
    exec_new_vmspace() at exec_new_vmspace+0x21d/frame 0xfffffe0113a4f5e0
    exec_elf64_imgact() at exec_elf64_imgact+0xa58/frame 0xfffffe0113a4f6f0
    kern_execve() at kern_execve+0x74d/frame 0xfffffe0113a4fa50
    sys_execve() at sys_execve+0x4a/frame 0xfffffe0113a4fad0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113a4fbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113a4fbf0
    --- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x800b40d8a, rsp = 0x7fffffffe188, rbp = 0x7fffffffe2d0 ---
    
    Tracing command sh pid 54120 tid 100774 td 0xfffff8015e162a00
    turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0113c6d480
    __rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0113c6d4b0
    vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0113c6d530
    vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0113c6d560
    exec_new_vmspace() at exec_new_vmspace+0x22f/frame 0xfffffe0113c6d5e0
    exec_elf64_imgact() at exec_elf64_imgact+0xa58/frame 0xfffffe0113c6d6f0
    kern_execve() at kern_execve+0x74d/frame 0xfffffe0113c6da50
    sys_execve() at sys_execve+0x4a/frame 0xfffffe0113c6dad0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c6dbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c6dbf0
    --- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x800b40d8a, rsp = 0x7fffffffe188, rbp = 0x7fffffffe2d0 ---
    
    Tracing command cat pid 54035 tid 100570 td 0xfffff8001d601000
    cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00ea3f4cf0
    ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe00ea3f4d10
    trap() at trap+0x3a/frame 0xfffffe00ea3f4f20
    nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00ea3f4f20
    --- trap 0x13, rip = 0xffffffff82e91de0, rsp = 0xfffffe01137de588, rbp = 0xfffffe01137de5b0 ---
    zrl_add_impl() at zrl_add_impl/frame 0xfffffe01137de5b0
    zfs_freebsd_getattr() at zfs_freebsd_getattr+0xa2e/frame 0xfffffe01137de820
    VOP_GETATTR_APV() at VOP_GETATTR_APV+0x87/frame 0xfffffe01137de850
    vm_mmap_vnode() at vm_mmap_vnode+0x164/frame 0xfffffe01137de990
    vn_mmap() at vn_mmap+0xe5/frame 0xfffffe01137dea20
    sys_mmap() at sys_mmap+0x40c/frame 0xfffffe01137dead0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe01137debf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe01137debf0
    --- syscall (477, FreeBSD ELF64, sys_mmap), rip = 0x800613ffa, rsp = 0x7fffffffde68, rbp = 0x7fffffffdfc0 ---
    
    Tracing command sh pid 53957 tid 101116 td 0xfffff80162b5a500
    cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00ea3fccf0
    ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe00ea3fcd10
    trap() at trap+0x3a/frame 0xfffffe00ea3fcf20
    nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00ea3fcf20
    --- trap 0x13, rip = 0xffffffff811a3b4b, rsp = 0xfffffe0113c9a850, rbp = 0xfffffe0113c9a890 ---
    pmap_try_insert_pv_entry() at pmap_try_insert_pv_entry+0x13b/frame 0xfffffe0113c9a890
    pmap_copy() at pmap_copy+0x65c/frame 0xfffffe0113c9a930
    vmspace_fork() at vmspace_fork+0x8d9/frame 0xfffffe0113c9a9b0
    fork1() at fork1+0x5ec/frame 0xfffffe0113c9aa70
    sys_fork() at sys_fork+0x39/frame 0xfffffe0113c9aad0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c9abf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c9abf0
    --- syscall (2, FreeBSD ELF64, sys_fork), rip = 0x800b5d77a, rsp = 0x7fffffffe508, rbp = 0x7fffffffe540 ---
    
    Tracing command php-fpm pid 39824 tid 100686 td 0xfffff8003b242000
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113bb2850
    mi_switch() at mi_switch+0xd2/frame 0xfffffe0113bb2880
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113bb2900
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113bb2930
    _sleep() at _sleep+0x294/frame 0xfffffe0113bb29c0
    kern_accept4() at kern_accept4+0x1f2/frame 0xfffffe0113bb2a70
    accept1() at accept1+0x60/frame 0xfffffe0113bb2ad0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113bb2bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113bb2bf0
    --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x801d947da, rsp = 0x7ffffffec848, rbp = 0x7ffffffec880 ---
    
    Tracing command top pid 43353 tid 100695 td 0xfffff80162d08a00
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113b35700
    mi_switch() at mi_switch+0xd2/frame 0xfffffe0113b35730
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113b357b0
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113b357e0
    _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe0113b35850
    seltdwait() at seltdwait+0xc7/frame 0xfffffe0113b358a0
    kern_select() at kern_select+0x979/frame 0xfffffe0113b35a90
    sys_select() at sys_select+0x54/frame 0xfffffe0113b35ad0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113b35bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113b35bf0
    --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80121000a, rsp = 0x7fffffffe888, rbp = 0x7fffffffea60 ---
    
    Tracing command bash pid 32987 tid 100680 td 0xfffff801628d1000
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113b946a0
    mi_switch() at mi_switch+0xd2/frame 0xfffffe0113b946d0
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113b94750
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113b94780
    _sleep() at _sleep+0x294/frame 0xfffffe0113b94810
    kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe0113b948e0
    sys_wait4() at sys_wait4+0x73/frame 0xfffffe0113b94ad0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113b94bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113b94bf0
    --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800e5d57a, rsp = 0x7fffffffe7a8, rbp = 0x7fffffffe7f0 ---
    
    Tracing command sh pid 2745 tid 100748 td 0xfffff8015e029a00
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113c366a0
    mi_switch() at mi_switch+0xd2/frame 0xfffffe0113c366d0
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113c36750
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113c36780
    _sleep() at _sleep+0x294/frame 0xfffffe0113c36810
    kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe0113c368e0
    sys_wait4() at sys_wait4+0x73/frame 0xfffffe0113c36ad0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c36bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c36bf0
    --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800b5d57a, rsp = 0x7fffffffe768, rbp = 0x7fffffffe820 ---
    
    Tracing command sshd pid 2354 tid 100699 td 0xfffff8001dd13000
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113bbc700
    mi_switch() at mi_switch+0xd2/frame 0xfffffe0113bbc730
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113bbc7b0
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113bbc7e0
    _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe0113bbc850
    seltdwait() at seltdwait+0xc7/frame 0xfffffe0113bbc8a0
    kern_select() at kern_select+0x979/frame 0xfffffe0113bbca90
    sys_select() at sys_select+0x54/frame 0xfffffe0113bbcad0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113bbcbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113bbcbf0
    --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x801db400a, rsp = 0x7fffffffe5a8, rbp = 0x7fffffffe5e0 ---
    
    Tracing command php pid 21591 tid 100704 td 0xfffff8001d4b5500
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113bd58c0
    mi_switch() at mi_switch+0xd2/frame 0xfffffe0113bd58f0
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113bd5970
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113bd59a0
    _sleep() at _sleep+0x26f/frame 0xfffffe0113bd5a30
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe0113bd5a90
    sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe0113bd5ad0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113bd5bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113bd5bf0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x801d8571a, rsp = 0x7fffffffc7e8, rbp = 0x7fffffffc810 ---
    
    Tracing command upsd pid 21318 tid 100739 td 0xfffff80162cd4500
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113c09770
    mi_switch() at mi_switch+0xd2/frame 0xfffffe0113c097a0
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113c09820
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113c09850
    _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe0113c098c0
    seltdwait() at seltdwait+0xc7/frame 0xfffffe0113c09910
    kern_poll() at kern_poll+0x296/frame 0xfffffe0113c09ab0
    sys_poll() at sys_poll+0x61/frame 0xfffffe0113c09ad0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c09bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c09bf0
    --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x800d4fd9a, rsp = 0x7fffffffe668, rbp = 0x7fffffffe6a0 ---
    
    Tracing command apcupsd-ups pid 21063 tid 100611 td 0xfffff8003be18500
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113aa9700
    mi_switch() at mi_switch+0xd2/frame 0xfffffe0113aa9730
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113aa97b0
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113aa97e0
    _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe0113aa9850
    seltdwait() at seltdwait+0xc7/frame 0xfffffe0113aa98a0
    kern_select() at kern_select+0x979/frame 0xfffffe0113aa9a90
    sys_select() at sys_select+0x54/frame 0xfffffe0113aa9ad0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113aa9bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113aa9bf0
    --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800b9b00a, rsp = 0x7fffffffe678, rbp = 0x7fffffffe6b0 ---
    
    Tracing command upsmon pid 19999 tid 100703 td 0xfffff8001d4b5a00
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113bd08c0
    mi_switch() at mi_switch+0xd2/frame 0xfffffe0113bd08f0
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113bd0970
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113bd09a0
    _sleep() at _sleep+0x26f/frame 0xfffffe0113bd0a30
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe0113bd0a90
    sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe0113bd0ad0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113bd0bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113bd0bf0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800d3c71a, rsp = 0x7fffffffe5c8, rbp = 0x7fffffffe5f0 ---
    
    Tracing command upsmon pid 19580 tid 100685 td 0xfffff80162d0a500
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113bad810
    mi_switch() at mi_switch+0xd2/frame 0xfffffe0113bad840
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113bad8c0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113bad8f0
    _sleep() at _sleep+0x294/frame 0xfffffe0113bad980
    pipe_read() at pipe_read+0x36d/frame 0xfffffe0113bad9e0
    dofileread() at dofileread+0x98/frame 0xfffffe0113bada30
    kern_readv() at kern_readv+0x68/frame 0xfffffe0113bada80
    sys_read() at sys_read+0x84/frame 0xfffffe0113badad0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113badbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113badbf0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800da502a, rsp = 0x7fffffffe628, rbp = 0x7fffffffe660 ---
    
    Tracing command sh pid 33060 tid 100578 td 0xfffff8001d4e5a00
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113a04810
    mi_switch() at mi_switch+0xd2/frame 0xfffffe0113a04840
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113a048c0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113a048f0
    _sleep() at _sleep+0x294/frame 0xfffffe0113a04980
    pipe_read() at pipe_read+0x36d/frame 0xfffffe0113a049e0
    dofileread() at dofileread+0x98/frame 0xfffffe0113a04a30
    kern_readv() at kern_readv+0x68/frame 0xfffffe0113a04a80
    sys_read() at sys_read+0x84/frame 0xfffffe0113a04ad0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113a04bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113a04bf0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800bc602a, rsp = 0x7fffffffe6a8, rbp = 0x7fffffffe880 ---
    
    Tracing command dhcpd pid 15682 tid 100713 td 0xfffff80162d08000
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113c04700
    mi_switch() at mi_switch+0xd2/frame 0xfffffe0113c04730
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113c047b0
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113c047e0
    _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe0113c04850
    seltdwait() at seltdwait+0xc7/frame 0xfffffe0113c048a0
    kern_select() at kern_select+0x979/frame 0xfffffe0113c04a90
    sys_select() at sys_select+0x54/frame 0xfffffe0113c04ad0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c04bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c04bf0
    --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800d6400a, rsp = 0x7fffffffea38, rbp = 0x7fffffffea60 ---
    
    Tracing command unbound pid 88257 tid 100711 td 0xfffff80162cc2a00
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113bf8660
    mi_switch() at mi_switch+0xd2/frame 0xfffffe0113bf8690
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113bf8710
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113bf8740
    _sleep() at _sleep+0x294/frame 0xfffffe0113bf87d0
    kqueue_kevent() at kqueue_kevent+0x336/frame 0xfffffe0113bf89b0
    kern_kevent() at kern_kevent+0xb5/frame 0xfffffe0113bf8a20
    sys_kevent() at sys_kevent+0x11c/frame 0xfffffe0113bf8ad0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113bf8bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113bf8bf0
    --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x80177275a, rsp = 0x7fffffffe9f8, rbp = 0x7fffffffea40 ---
    
    Tracing command unbound pid 88257 tid 100760 td 0xfffff80162da8a00
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113c40660
    mi_switch() at mi_switch+0xd2/frame 0xfffffe0113c40690
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113c40710
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113c40740
    _sleep() at _sleep+0x294/frame 0xfffffe0113c407d0
    kqueue_kevent() at kqueue_kevent+0x336/frame 0xfffffe0113c409b0
    kern_kevent() at kern_kevent+0xb5/frame 0xfffffe0113c40a20
    sys_kevent() at sys_kevent+0x11c/frame 0xfffffe0113c40ad0
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c40bf0
    Xfast_syscall() at Xfast_syscall+0config.txt0600001341413115636557  7552 ustarrootwheeloptions	CONFIG_AUTOGENERATED
    ident	pfSense
    machine	amd64
    cpu	HAMMER
    makeoptions	WITH_CTF=1
    makeoptions	DEBUG=-g
    options	CONSPEED=115200
    options	MROUTING
    options	HIFN_RNDTEST
    options	HIFN_DEBUG
    options	MSGTQL=2048
    options	MSGSSZ=32
    options	MSGSEG=512
    options	MSGMNI=40
    options	MSGMNB=8192
    options	ALTQ_CODEL
    options	ALTQ_NOPCC
    options	ALTQ_FAIRQ
    options	ALTQ_PRIQ
    options	ALTQ_HFSC
    options	ALTQ_RIO
    options	ALTQ_RED
    options	ALTQ_CBQ
    options	ALTQ
    options	TCP_SIGNATURE
    options	IPSEC_NAT_T
    options	NETGRAPH_PRED1
    options	NETGRAPH_DEFLATE
    options	NETGRAPH_CAR
    options	NETGRAPH_PIPE
    options	NETGRAPH_TCPMSS
    options	NETGRAPH_TEE
    options	NETGRAPH_HOLE
    options	NETGRAPH_FRAME_RELAY
    options	NETGRAPH_ASYNC
    options	NETGRAPH_ECHO
    options	NETGRAPH_CISCO
    options	NETGRAPH_BRIDGE
    options	NETGRAPH_ONE2MANY
    options	NETGRAPH_LMI
    options	NETGRAPH_KSOCKET
    options	NETGRAPH_VJC
    options	NETGRAPH_UI
    options	NETGRAPH_MPPC_ENCRYPTION
    options	NETGRAPH_TTY
    options	NETGRAPH_SOCKET
    options	NETGRAPH_RFC1490
    options	NETGRAPH_PPTPGRE
    options	NETGRAPH_PPPOE
    options	NETGRAPH_PPP
    options	NETGRAPH_EIFACE
    options	NETGRAPH_IFACE
    options	NETGRAPH_ETHER
    options	NETGRAPH_BPF
    options	NETGRAPH_L2TP
    options	NETGRAPH_VLAN
    options	NETGRAPH
    options	IPSTEALTH
    options	IPFIREWALL_VERBOSE
    options	IPFIREWALL_DEFAULT_TO_ACCEPT
    options	PPS_SYNC
    options	NULLFS
    options	UNIONFS
    options	TMPFS
    options	GEOM_BDE
    options	GEOM_ELI
    options	GEOM_UZIP
    options	GEOM_MIRROR
    options	DDB
    options	XENHVM
    options	ATH_ENABLE_11N
    options	AH_AR5416_INTERRUPT_MITIGATION
    options	AH_SUPPORT_AR5416
    options	IEEE80211_SUPPORT_MESH
    options	IEEE80211_AMPDU_AGE
    options	IEEE80211_DEBUG
    options	SC_PIXEL_MODE
    options	VESA
    options	AHD_REG_PRETTY_PRINT
    options	AHC_REG_PRETTY_PRINT
    options	PCI_IOV
    options	PCI_HP
    options	ACPI_DMAR
    options	DEVICE_NUMA
    options	SMP
    options	KDB
    options	RCTL
    options	RACCT_DEFAULT_TO_DISABLED
    options	RACCT
    options	INCLUDE_CONFIG_FILE
    options	DDB_CTF
    options	KDTRACE_HOOKS
    options	KDTRACE_FRAME
    options	CAPABILITIES
    options	CAPABILITY_MODE
    options	AUDIT
    options	HWPMC_HOOKS
    options	KBD_INSTALL_CDEV
    options	PRINTF_BUFR_SIZE=128
    options	_KPOSIX_PRIORITY_SCHEDULING
    options	SYSVSEM
    options	SYSVMSG
    options	SYSVSHM
    options	STACK
    options	KTRACE
    options	SCSI_DELAY=5000
    options	COMPAT_FREEBSD10
    options	COMPAT_FREEBSD9
    options	COMPAT_FREEBSD32
    options	GEOM_LABEL
    options	GEOM_RAID
    options	GEOM_PART_GPT
    options	PSEUDOFS
    options	PROCFS
    options	CD9660
    options	MSDOSFS
    options	NFS_ROOT
    options	NFSLOCKD
    options	NFSD
    options	NFSCL
    options	MD_ROOT
    options	QUOTA
    options	UFS_GJOURNAL
    options	UFS_DIRHASH
    options	UFS_ACL
    options	SOFTUPDATES
    options	FFS
    options	SCTP
    options	TCP_OFFLOAD
    options	IPSEC
    options	INET6
    options	INET
    options	PREEMPTION
    options	SCHED_ULE
    options	NEW_PCIB
    options	GEOM_PART_MBR
    options	GEOM_PART_EBR_COMPAT
    options	GEOM_PART_EBR
    options	GEOM_PART_BSD
    device	isa
    device	mem
    device	io
    device	uart_ns8250
    device	cpufreq
    device	acpi
    device	pci
    device	fdc
    device	ahci
    device	ata
    device	mvs
    device	siis
    device	ahc
    device	ahd
    device	esp
    device	hptiop
    device	isp
    device	mpt
    device	mps
    device	mpr
    device	sym
    device	trm
    device	adv
    device	adw
    device	aic
    device	bt
    device	isci
    device	scbus
    device	ch
    device	da
    device	sa
    device	cd
    device	pass
    device	ses
    device	amr
    device	arcmsr
    device	ciss
    device	dpt
    device	hptmv
    device	hptnr
    device	hptrr
    device	hpt27xx
    device	iir
    device	ips
    device	mly
    device	twa
    device	tws
    device	aac
    device	aacp
    device	aacraid
    device	ida
    device	mfi
    device	mlx
    device	mrsas
    device	pmspcv
    device	twe
    device	nvme
    device	nvd
    device	atkbdc
    device	atkbd
    device	psm
    device	kbdmux
    device	vga
    device	splash
    device	sc
    device	vt
    device	vt_vga
    device	vt_efifb
    device	agp
    device	cbb
    device	pccard
    device	cardbus
    device	uart
    device	ppc
    device	ppbus
    device	lpt
    device	ppi
    device	puc
    device	bxe
    device	de
    device	em
    device	igb
    device	ix
    device	ixv
    device	ixl
    device	ixlv
    device	le
    device	ti
    device	txp
    device	vx
    device	miibus
    device	ae
    device	age
    device	alc
    device	ale
    device	bce
    device	bfe
    device	bge
    device	cas
    device	dc
    device	et
    device	fxp
    device	gem
    device	hme
    device	jme
    device	lge
    device	msk
    device	nfe
    device	nge
    device	pcn
    device	re
    device	rl
    device	sf
    device	sge
    device	sis
    device	sk
    device	ste
    device	stge
    device	tl
    device	tx
    device	vge
    device	vr
    device	wb
    device	xl
    device	wlan
    device	wlan_wep
    device	wlan_ccmp
    device	wlan_tkip
    device	wlan_amrr
    device	an
    device	ath
    device	ath_pci
    device	ath_hal
    device	ath_rate_sample
    device	ipw
    device	iwi
    device	iwn
    device	malo
    device	mwl
    device	ral
    device	wi
    device	wpi
    device	loop
    device	random
    device	padlock_rng
    device	rdrand_rng
    device	ether
    device	vlan
    device	tun
    device	md
    device	gif
    device	firmware
    device	bpf
    device	uhci
    device	ohci
    device	ehci
    device	xhci
    device	usb
    device	ukbd
    device	umass
    device	sound
    device	snd_cmi
    device	snd_csa
    device	snd_emu10kx
    device	snd_es137x
    device	snd_hda
    device	snd_ich
    device	snd_via8233
    device	mmc
    device	mmcsd
    device	sdhci
    device	virtio
    device	virtio_pci
    device	vtnet
    device	virtio_blk
    device	virtio_scsi
    device	virtio_balloon
    device	hyperv
    device	xenpci
    device	vmx
    device	netmap
    device	crypto
    device	wlan_rssadapt
    device	wlan_xauth
    device	wlan_acl
    device	iwifw
    device	ipwfw
    device	wpifw
    device	iwnfw
    device	uath
    device	ralfw
    device	ural
    device	urtw
    device	rum
    device	mwlfw
    device	zyd
    device	upgt
    device	udav
    device	axe
    device	axge
    device	aue
    device	cue
    device	kue
    device	mos
    device	rsu
    device	rsufw
    device	run
    device	runfw
    device	rue
    device	rtwn
    device	rtwnfw
    device	siba_bwn
    device	bwn
    device	bwi
    device	ufoma
    device	ucom
    device	uslcom
    device	uplcom
    device	umct
    device	uvisor
    device	uark
    device	uftdi
    device	uvscom
    device	umodem
    device	u3g
    device	cdce
    device	uhid
    device	firewire
    device	sbp
    device	tap
    device	gre
    device	if_bridge
    device	carp
    device	lagg
    device	vte
    device	enc
    device	pf
    device	pflog
    device	pfsync
    device	rndtest
    device	hifn
    device	ubsec
    device	safe
    device	padlock
    device	speaker
    device	mxge
    device	cxgb
    device	cxgbe
    device	oce
    msgbuf.txt06000017726213115636557  7664 ustarrootwheelCopyright (c) 1992-2016 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    	The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 11.0-RELEASE-p8 #78 3f54ce3814d(RELENG_2_4): Wed Mar 15 18:23:59 CDT 2017
        root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense amd64
    FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
    VT(efifb): resolution 1920x1080
    CPU: Intel(R) Celeron(R) CPU  N3150  @ 1.60GHz (1600.05-MHz K8-class CPU)
      Origin="GenuineIntel"  Id=0x406c3  Family=0x6  Model=0x4c  Stepping=3
      Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x43d8e3bf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand>AMD Features=0x28100800 <syscall,nx,rdtscp,lm>AMD Features2=0x101 <lahf,prefetch>Structured Extended Features=0x2282 <tscadj,smep,erms,nfpusg>VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID
      TSC: P-state invariant, performance statistics
    real memory  = 4294967296 (4096 MB)
    avail memory = 3877691392 (3698 MB)
    Event timer "LAPIC" quality 600
    ACPI APIC Table: <alaska a="" m="" i="">WARNING: L1 data cache covers less APIC IDs than a core
    0 < 1
    FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
    FreeBSD/SMP: 1 package(s) x 4 core(s)
    random: unblocking device.
    ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 128/32 (20160527/tbfadt-650)
    ioapic0 <version 2.0="">irqs 0-114 on motherboard
    random: entropy device external interface
    wlan: mac acl policy registered
    kbd0 at kbdmux0
    netmap: loaded module
    module_register_init: MOD_LOAD (vesa, 0xffffffff8122b980, 0) error 19
    random: registering fast source Intel Secure Key RNG
    random: fast provider: "Intel Secure Key RNG"
    cryptosoft0: <software crypto="">on motherboard
    padlock0: No ACE support.
    acpi0: <alaska a="" m="" i="">on motherboard
    acpi0: Power Button (fixed)
    unknown: I/O range not supported
    cpu0: <acpi cpu="">on acpi0
    cpu1: <acpi cpu="">on acpi0
    cpu2: <acpi cpu="">on acpi0
    cpu3: <acpi cpu="">on acpi0
    atrtc0: <at realtime="" clock="">port 0x70-0x77 on acpi0
    atrtc0: Warning: Couldn't map I/O.
    Event timer "RTC" frequency 32768 Hz quality 0
    attimer0: <at timer="">port 0x40-0x43,0x50-0x53 irq 0 on acpi0
    Timecounter "i8254" frequency 1193182 Hz quality 0
    Event timer "i8254" frequency 1193182 Hz quality 100
    Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
    acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
    pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
    pci0: <acpi pci="" bus="">on pcib0
    vgapci0: <vga-compatible display="">port 0xf000-0xf03f mem 0x80000000-0x80ffffff,0x90000000-0x9fffffff irq 16 at device 2.0 on pci0
    vgapci0: Boot video device
    ahci0: <ahci sata="" controller="">port 0xf060-0xf07f mem 0x81511000-0x815117ff irq 19 at device 19.0 on pci0
    ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier not supported
    ahcich0: <ahci channel="">at channel 0 on ahci0
    ahcich1: <ahci channel="">at channel 1 on ahci0
    xhci0: <xhci (generic)="" usb="" 3.0="" controller="">mem 0x81500000-0x8150ffff irq 20 at device 20.0 on pci0
    xhci0: 32 bytes context size, 64-bit DMA
    usbus0 on xhci0
    pci0: <encrypt decrypt="">at device 26.0 (no driver attached)
    pcib1: <acpi pci-pci="" bridge="">irq 16 at device 28.0 on pci0
    pci1: <acpi pci="" bus="">on pcib1
    re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet="">port 0xe000-0xe0ff mem 0xfffe04000-0xfffe04fff,0xfffe00000-0xfffe03fff irq 16 at device 0.0 on pci1
    re0: Using 1 MSI-X message
    re0: ASPM disabled
    re0: Chip rev. 0x2c800000
    re0: MAC rev. 0x00100000
    miibus0: <mii bus="">on re0
    rgephy0: <rtl8169s 8211="" 8110s="" 1000base-t="" media="" interface="">PHY 1 on miibus0
    rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
    re0: Using defaults for TSO: 65518/35/2048
    re0: Ethernet address: 00:e0:4c:68:55:ae
    re0: netmap queues/slots: TX 1/256, RX 1/256
    pcib2: <acpi pci-pci="" bridge="">irq 17 at device 28.1 on pci0
    pcib2: [GIANT-LOCKED]
    pci2: <acpi pci="" bus="">on pcib2
    re1: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet="">port 0xd000-0xd0ff mem 0xfffd04000-0xfffd04fff,0xfffd00000-0xfffd03fff irq 17 at device 0.0 on pci2
    re1: Using 1 MSI-X message
    re1: ASPM disabled
    re1: Chip rev. 0x2c800000
    re1: MAC rev. 0x00100000
    miibus1: <mii bus="">on re1
    rgephy1: <rtl8169s 8211="" 8110s="" 1000base-t="" media="" interface="">PHY 1 on miibus1
    rgephy1:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
    re1: Using defaults for TSO: 65518/35/2048
    re1: Ethernet address: 00:e0:4c:68:55:af
    re1: netmap queues/slots: TX 1/256, RX 1/256
    pcib3: <acpi pci-pci="" bridge="">irq 18 at device 28.2 on pci0
    pci3: <acpi pci="" bus="">on pcib3
    igb0: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k="">port 0xc020-0xc03f mem 0x81380000-0x813fffff,0x81404000-0x81407fff irq 18 at device 0.0 on pci3
    igb0: Using MSIX interrupts with 2 vectors
    igb0: Ethernet address: 00:30:18:c3:d0:e5
    igb0: netmap queues/slots: TX 1/4096, RX 1/4096
    igb1: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k="">port 0xc000-0xc01f mem 0x81280000-0x812fffff,0x81400000-0x81403fff irq 19 at device 0.1 on pci3
    igb1: Using MSIX interrupts with 2 vectors
    igb1: Ethernet address: 00:30:18:c3:d0:e6
    igb1: netmap queues/slots: TX 1/4096, RX 1/4096
    isab0: <pci-isa bridge="">at device 31.0 on pci0
    isa0: <isa bus="">on isab0
    acpi_button0: <sleep button="">on acpi0
    acpi_tz0: <thermal zone="">on acpi0
    ppc0: cannot reserve I/O port range
    est0: <enhanced speedstep="" frequency="" control="">on cpu0
    est1: <enhanced speedstep="" frequency="" control="">on cpu1
    est2: <enhanced speedstep="" frequency="" control="">on cpu2
    est3: <enhanced speedstep="" frequency="" control="">on cpu3
    usbus0: 5.0Gbps Super Speed USB v3.0
    ZFS filesystem version: 5
    ZFS storage pool version: features support (5000)
    Timecounters tick every 1.000 msec
    nvme cam probe device init
    ugen0.1: <0x8086> at usbus0
    uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
    uhub0: 13 ports with 13 removable, self powered
    ugen0.2: <american power="" conversion="">at usbus0
    ugen0.3: <vendor 0x05e3="">at usbus0
    uhub1: <vendor 2="" 9="" 0x05e3="" usb2.0="" hub,="" class="" 0,="" rev="" 2.00="" 85.36,="" addr="">on usbus0
    uhub1: 4 ports with 4 removable, self powered
    ugen0.4: <vendor 0x1997="">at usbus0
    ukbd0: <vendor 0="" 3="" 0x1997="" mini="" keyboard,="" class="" 0,="" rev="" 1.10="" 20.12,="" addr="">on usbus0
    kbd1 at ukbd0
    ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
    ada0: <kingston sms200s360g="" 608abbf0="">ATA8-ACS SATA 3.x device
    ada0: Serial Number 50026B72630613B0
    ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
    ada0: Command Queueing enabled
    ada0: 57241MB (117231408 512 byte sectors)
    SMP: AP CPU #2 Launched!
    SMP: AP CPU #1 Launched!
    SMP: AP CPU #3 Launched!
    Timecounter "TSC" frequency 1600052960 Hz quality 1000
    Trying to mount root from zfs:zroot/ROOT/default []...
    <118>Configuring crash dumps...
    <118>Using /dev/ada0p3 for dump device.
    <118>Filesystems are clean, continuing...
    <118>Mounting filesystems...
    <118>Restoring contents of RAM disk store... done.
    <118>
    <118>     ___
    <118> ___/ f \
    <118>/ p \___/ Sense
    <118>\___/   \
    <118>    \___/
    <118>
    <118>Welcome to pfSense 2.4.0-BETA...
    <118>
    <118>No core dumps found.
    <118>chmod: /var/empty: Operation not permitted
    <118>...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/dnscrypt-proxy /usr/local/lib/graphviz /usr/local/lib/ipsec /usr/local/lib/mysql /usr/local/lib/perl5/5.24/mach/CORE
    <118>32-bit compatibility ldconfig path:
    <118>done.
    <118>External config loader 1.0 is now starting...
    <118>Launching the init system....... done.
    <118>Initializing.................. done.
    <118>Starting device manager (devd)...kldload: can't load ums: No such file or directory
    <118>done.
    <118>Loading configuration......done.
    <118>tar: Removing leading '/' from member names
    <118>Updating configuration...done.
    <118>Cleaning backup cache.................................done.
    <118>Setting up extended sysctls...done.
    padlock0: No ACE support.
    aesni0: <aes-cbc,aes-xts,aes-gcm,aes-icm>on motherboard
    coretemp0: <cpu on-die="" thermal="" sensors="">on cpu0
    coretemp1: <cpu on-die="" thermal="" sensors="">on cpu1
    coretemp2: <cpu on-die="" thermal="" sensors="">on cpu2
    coretemp3: <cpu on-die="" thermal="" sensors="">on cpu3
    <118>Setting timezone...done.
    <118>Configuring loopback interface...done.
    <118>Starting syslog...done.
    <118>Starting Secure Shell Services...done.
    <118>Setting up interfaces microcode...done.
    <118>Configuring loopback interface...done.
    <118>Creating wireless clone interfaces...done.
    <118>Configuring LAGG interfaces...done.
    <118>Configuring VLAN interfaces...done.
    <118>Configuring QinQ interfaces...done.
    <118>Configuring WAN interface...
    <5>igb0: link state changed to UP
    <118>done.
    <118>Configuring LAN interface...done.
    <118>Configuring CARP settings...done.
    <118>Syncing OpenVPN settings...
    <6>tun1: changing name to 'ovpnc1'
    <6>tun2: changing name to 'ovpnc2'
    <118>done.
    <6>pflog0: promiscuous mode enabled
    <118>Configuring firewall.....0 addresses deleted.
    <118>0 addresses deleted.
    <118>.done.
    <118>Starting PFLOG...done.
    <118>Setting up gateway monitors...done.
    <5>ovpnc2: link state changed to UP
    <5>ovpnc1: link state changed to UP
    <5>igb1: link state changed to UP
    <118>Starting DNS Resolver...done.
    <118>Synchronizing user settings...done.
    <118>Starting webConfigurator...done.
    <118>Configuring CRON...done.
    <118>Starting NTP time client...done.
    <118>Starting DHCP service...done.
    <118>Starting DHCPv6 service...done.
    <118>Configuring firewall.....0 addresses deleted.
    <118>0 addresses deleted.
    <118>0 addresses deleted.
    <118>0 addresses deleted.
    <118>.done.
    <5>igb0: link state changed to DOWN
    <5>igb1: link state changed to DOWN
    <5>igb0: link state changed to UP
    <5>igb1: link state changed to UP
    <118>Generating RRD graphs...done.
    <118>Starting syslog...done.
    <118>Starting CRON... done.
    <118> Starting package Cron...done.
    <118> Starting package iftop...done.
    <118> Starting package iperf...done.
    <118> Starting package mtr-nox11...done.
    <118> Starting package nmap...done.
    <118> Starting package pfBlockerNG...done.
    <118> Starting package darkstat...done.
    <118> Starting package ntopng...done.
    <118> Starting package Shellcmd...done.
    <118> Starting package System Patches...done.
    <118> Starting package Backup...done.
    <118> Starting package apcupsd...done.
    <118> Starting package nut...done.
    <118> Starting /usr/local/etc/rc.d/dnsbl.sh...done.
    <118>pfSense 2.4.0-BETA amd64 Wed Mar 15 18:17:17 CDT 2017
    <118>Bootup complete
    <5>ovpnc1: link state changed to DOWN
    <5>ovpnc2: link state changed to DOWN
    <5>ovpnc2: link state changed to UP
    <5>ovpnc1: link state changed to UP
    <5>re0: link state changed to DOWN
    <5>re1: link state changed to DOWN
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200a, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:401f:15::6, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4009:4c::a, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4009:4c::a, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4009:4c::a, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a01:578:3::36c2:327f, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:809::200a, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:816::200a, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200a, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:401f:31::6, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a01:578:3::3433:8b11, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:815::200a, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80a::200a, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:400c:c0a::5f, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:401f:15::6, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a01:578:3::3410:e6ed, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:809::200a, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:809::200a, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:811::200a, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:401f:15::6, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a01:578:3::34d2:92a8, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80a::200a, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
    <7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0</cpu></cpu></cpu></cpu></aes-cbc,aes-xts,aes-gcm,aes-icm></kingston></vendor></vendor></vendor></vendor></american></enhanced></enhanced></enhanced></enhanced></thermal></sleep></isa></pci-isa></intel(r)></intel(r)></acpi></acpi></rtl8169s></mii></realtek></acpi></acpi></rtl8169s></mii></realtek></acpi></acpi></encrypt></xhci></ahci></ahci></ahci></vga-compatible></acpi></acpi></at></at></acpi></acpi></acpi></acpi></alaska></software></version></alaska></tscadj,smep,erms,nfpusg></lahf,prefetch></syscall,nx,rdtscp,lm></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe> 
    


  • and last bit

    kernel trap 12 with interrupts disabled
    
    Fatal trap 12: page fault while in kernel mode
    cpuid = 1; apic id = 02
    fault virtual address	= 0x30
    fault code		= supervisor read data, page not present
    instruction pointer	= 0x20:0xffffffff80cb910c
    stack pointer	        = 0x28:0xfffffe0113c6d450
    frame pointer	        = 0x28:0xfffffe0113c6d480
    code segment		= base 0x0, limit 0xfffff, type 0x1b
    			= DPL 0, pres 1, long 1, def32 0, gran 1
    processor eflags	= resume, IOPL = 0
    current process		= 54120 (sh)
    version.txt06000025413115636557  7630 ustarrootwheelFreeBSD 11.0-RELEASE-p8 #78 3f54ce3814d(RELENG_2_4): Wed Mar 15 18:23:59 CDT 2017
        root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense
    


  • another forum with same backtrace actually has led me to this unresolved bug report on FreeBSD,  has been confirmed to affect 11 as well as 12.

    So I think this might be what I have

    https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

    Have compiled the pfsense kernel now with the patch, will report back here if all good or not, only just booted into it so need to give it some time.


Log in to reply