Kernel lockup

[chrcoluk]

Had a kernel lockup this morning, extract from logs below (note log is set to have newest line on top)

Jan 4 10:46:17	kernel		Fatal trap 12: page fault while in kernel mode
Jan 4 10:46:17	kernel		kernel trap 12 with interrupts disabled
Jan 4 10:46:17	kernel		current process	= 97781 (sh)
Jan 4 10:46:17	kernel		processor eflags	= resume, IOPL = 0
Jan 4 10:46:17	kernel		= DPL 0, pres 1, long 1, def32 0, gran 1
Jan 4 10:46:17	kernel		code segment	= base 0x0, limit 0xfffff, type 0x1b
Jan 4 10:46:17	kernel		frame pointer	= 0x28:0xfffffe01181b2480
Jan 4 10:46:17	kernel		stack pointer	= 0x28:0xfffffe01181b2450
Jan 4 10:46:17	kernel		instruction pointer	= 0x20:0xffffffff80cb908c
Jan 4 10:46:17	kernel		fault code	= supervisor read data, page not present
Jan 4 10:46:17	kernel		fault virtual address	= 0x30
Jan 4 10:46:17	kernel		cpuid = 0; apic id = 00

Had no lockups on 2.2 but I also havent added load to the box until the past week.

I undid some tunables I set for the nic in loader.conf.custom, and disabled MSIX.  Will see if it stabilises, if I get more issues, I will test the ram.

If needed I can swap the ram as its the same exact sodimm that I had put in my laptop last year, so I can remove from laptop to put in the pfsense unit if I find the ram faulty.

[jimp]

Was there no other data from the crash? No crash report or backtrace?

There is not enough information in those log messages to make any guesses about a cause.

[skyeci]

Hi Jimp

Did you have any thoughts about the one I had and posted which was after the vpn fix.

Thanks

[chrcoluk]

jimp where do I look for the backtrace? I will check /var/crash if its same as default FreeBSD behaviour.

right I read this

https://doc.pfsense.org/index.php/Unexpected_Reboot_Troubleshooting

yeah, I did disable the swap partition (it does exist tho), so I will enable it again just in case I get another panic.

[jimp]

The firewall should present a box on top of the dashboard with a link to see the crash report if it was able to collect one.

[chrcoluk]

edited my post, sorry I edited as you replied.

[Dazog]

Well it looks like you are the second person having this issue.

I posted a thread earlier and got NO reply.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213856

That is the bug and I have sent multiple crash dumps already.

[jimp]

Just because you have a "Fatal Trap 12" doesn't mean it's the same bug. The stack trace doesn't look anywhere near the same.

[chrcoluk]

@Dazog:

Well it looks like you are the second person having this issue.

I posted a thread earlier and got NO reply.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213856

That is the bug and I have sent multiple crash dumps already.

panic can be caused by many things, this is why I didnt jump in claiming its a bug, as could be hardware or software caused.
My fault I had disabled the swap partition (I had configured swap file instead for SSD trim reasons).  So the lack of backtrace is because of that.

FreeBSD historically has needed issues resolving in a .0 release, this could be one of those issues, but it could be hardware as well.

Also you posted directly to FreeBSD developers not on pfsense bug report.

[chrcoluk]

no more panic's but given the number of people who reported panic's on FreeBSD 11 reporting it been fixed with the igb queue count set to 1 or 2, I adjusted it to 2 for safety.  I have enabled MSIX again.

[w0w]

@chrcoluk:

no more panic's but given the number of people who reported panic's on FreeBSD 11 reporting it been fixed with the igb queue count set to 1 or 2, I adjusted it to 2 for safety.  I have enabled MSIX again.

What "it" been fixed? Is there some ticket on freebsd community regarding this problem?

[chrcoluk]

yeah there is a ticket but I dont have the url to hand sorry, everyone who was on the ticket reported panics been fixed with setting it to 1.

Only 1 guy said it was also stable on 2 tho and he posted that on here somewhere.  The default is to match core count so in my case default is 4.

hw.igb.num_queues=2

[chrcoluk]

I reinstalled pfSense 2.4 2 days ago using zfs, I made sure I have the loader.conf.local in place to reduce igb queues to 1 but I have had a panic late last night, again no dump was made so no backtrace, this time tho a swap partition was enabled.

Will scan my ram.
Will disable adaptive mode for cpu so it stays at one speed only as cpu clock fluctuations can cause instabilities.
Will check bios to see if anything in there that could cause compatibility issues.

I do notice on bootup that the boot up log says its not using any dump device, so I will try manually specifying it in rc.conf.local to see if it reports in the boot log one is set.

I wonder if it may be a good idea to track 11-STABLE instead of 11.0 as .0 releases of FreeBSD can be problematic.

[kpa]

@chrcoluk:

I wonder if it may be a good idea to track 11-STABLE instead of 11.0 as .0 releases of FreeBSD can be problematic.

The stable branch is a much more moving target than the releng branch, it would be hell a lot more work to keep up with the changes.

[chrcoluk]

Have done the 2 ram tests I planned.

Also have the dump device correctly showing during the boot process now so if I have another panic I should get a dump to backtrace.

I have disabled turbo mode and EIST on the cpu and dynamic voltage on the ram.

[luckman212]

chrcoluk what hardware are you running pfSense on?  Your sig says "Braswell N3150" but that isn't specific.  Sorry if I missed it in another post.

[doktornotor]

@luckman212:

chrcoluk what hardware are you running pfSense on?

That is unacceptable usage. You must say "What hardware are you running pfSense software on?"

;D ;D ::) ::)

SCNR. Won't further comment on this new round of legal bleep. I hoped the unfortunate episode ~2+ years ago was enough of a lesson. Apparently not.

[w0w]

chrcoluk
Did you add kern.ipc.nmbclusters="1000000"?
What is your WAN bandwidth?

[chrcoluk]

I dont know the exact hardware packaged as I purchased a compact unit from amazon which is simply labeled as a BRASWELL N3150 unit, which tells you I have a celeron N3150 on a braswell chipset.

I think I am going to adjust the bios again to enable EIST and turbo mode but also configure powerd to never let the clocks drop below the stock max speed that means I get turbo mode back again for extra performance whilst not letting it drop to idle voltages, I will keep the dynamic voltages disabled on the ram.

Also nmbclusters has been increased in the loader.conf from almost day one of using pfsense.

WAN bandwidth at the time of the crash was just idle, the absolute max bandwidth usage possible is about 70mbit down and 20mbit up.

[w0w]

Ok, try to bump voltage on memory, +0.1V

[chrcoluk]

I dont have that control in the bios.

But I have disabled now dynamic voltage mode (DVFS) so it should always be at full voltage now instead of power saving lower voltages.

[chrcoluk]

Am getting issues again, the good news is I am getting proper kernel dumps now, the last two I hit the submit button, but to let more eyes look at the problem I will dump the data here as well, the bit of research I have done seems to indicate its filesystem related, but I am most definitely not sure on this so appreciate an opinion from anyone.

In the meantime since my laptop uses the same spec ram I am going to swap the ram with my laptop memory, even tho this existing ram has passed stability tests.

					Crash report begins.  Anonymous machine information:

amd64
11.0-RELEASE-p8
FreeBSD 11.0-RELEASE-p8 #78 3f54ce3814d(RELENG_2_4): Wed Mar 15 18:23:59 CDT 2017     root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense

Crash report details:

No PHP errors found.

Filename: /var/crash/bounds
1

Filename: /var/crash/info.0
Dump header from device: /dev/ada0p3
  Architecture: amd64
  Architecture Version: 1
  Dump Length: 123392
  Blocksize: 512
  Dumptime: Wed Jun  7 00:40:31 2017
  Hostname: PFSENSE.home
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 11.0-RELEASE-p8 #78 3f54ce3814d(RELENG_2_4): Wed Mar 15 18:23:59 CDT 2017
    root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense
  Panic String: 
  Dump Parity: 318326786
  Bounds: 0
  Dump Status: good

Filename: /var/crash/info.last
Dump header from device: /dev/ada0p3
  Architecture: amd64
  Architecture Version: 1
  Dump Length: 123392
  Blocksize: 512
  Dumptime: Wed Jun  7 00:40:31 2017
  Hostname: PFSENSE.home
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 11.0-RELEASE-p8 #78 3f54ce3814d(RELENG_2_4): Wed Mar 15 18:23:59 CDT 2017
    root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense
  Panic String: 
  Dump Parity: 318326786
  Bounds: 0
  Dump Status: good

Filename: /var/crash/textdump.tar.0
ddb.txt06000014000013115636557  7106 ustarrootwheeldb:0:kdb.enter.default>  run lockinfo
db:1:lockinfo> show locks
No such command
db:1:locks>  show alllocks
No such command
db:1:alllocks>  show lockedvnods
Locked vnodes
db:0:kdb.enter.default>  show pcpu
cpuid        = 1
dynamic pcpu = 0xfffffe0167de2000
curthread    = 0xfffff8015e162a00: pid 54120 "sh"
curpcb       = 0xfffffe0113c6dcc0
fpcurthread  = 0xfffff8015e162a00: pid 54120 "sh"
idlethread   = 0xfffff8000490aa00: tid 100004 "idle: cpu1"
curpmap      = 0xfffff801621b3138
tssp         = 0xffffffff82a1dd78
commontssp   = 0xffffffff82a1dd78
rsp0         = 0xfffffe0113c6dcc0
gs32p        = 0xffffffff82a245d0
ldt          = 0xffffffff82a24610
tss          = 0xffffffff82a24600
db:0:kdb.enter.default>  bt
Tracing pid 54120 tid 100774 td 0xfffff8015e162a00
turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0113c6d480
__rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0113c6d4b0
vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0113c6d530
vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0113c6d560
exec_new_vmspace() at exec_new_vmspace+0x22f/frame 0xfffffe0113c6d5e0
exec_elf64_imgact() at exec_elf64_imgact+0xa58/frame 0xfffffe0113c6d6f0
kern_execve() at kern_execve+0x74d/frame 0xfffffe0113c6da50
sys_execve() at sys_execve+0x4a/frame 0xfffffe0113c6dad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c6dbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c6dbf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x800b40d8a, rsp = 0x7fffffffe188, rbp = 0x7fffffffe2d0 ---
db:0:kdb.enter.default>  ps
  pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
54226 53957   334     0  R       CPU 0                       sh
54120 53957   334     0  R       CPU 1                       sh
54035 53957   334     0  R       CPU 2                       cat
53957 33060   334     0  R       CPU 3                       sh
39824   334   334     0  S       accept   0xfffff8003b01006c php-fpm
43353 32987 43353     0  S+      select   0xfffff801629be7c0 top
32987  2745 32987     0  S+      wait     0xfffff8003b194528 bash
 2745  2354  2745     0  Ss+     wait     0xfffff8003b03d528 sh
 2354  8180  2354     0  Ss      select   0xfffff80102b6a5c0 sshd
21591     1   334     0  S       nanslp   0xffffffff82865d21 php
21318     1 21318     0  Ss      select   0xfffff80102d507c0 upsd
21063     1 21063    66  Ss      select   0xfffff8003b13e6c0 apcupsd-ups
19999 19580 19580    66  S       nanslp   0xffffffff82865d21 upsmon
19580     1 19580     0  Ss      piperd   0xfffff80162a03be0 upsmon
33060     1   334     0  S       piperd   0xfffff8015e2bf8e8 sh
15682     1 15682   136  Ss      select   0xfffff8001d948240 dhcpd
88257     1 88257    59  Ss      (threaded)                  unbound
100711                   S       kqread   0xfffff8015eee6c00 unbound
100760                   S       kqread   0xfffff80011585800 unbound
100761                   S       kqread   0xfffff80011585a00 unbound
100762                   S       kqread   0xfffff800101aea00 unbound
55992     1 55992     0  Ss      (threaded)                  dpinger
100681                   S       uwait    0xfffff80162823e00 dpinger
100729                   S       sbwait   0xfffff8016285c4a4 dpinger
100730                   S       nanslp   0xffffffff82865d20 dpinger
100731                   S       nanslp   0xffffffff82865d21 dpinger
100732                   S       accept   0xfffff80162d3d72c dpinger
55571     1 55571     0  Ss      (threaded)                  dpinger
100701                   S       uwait    0xfffff8001d4c3b80 dpinger
100725                   S       sbwait   0xfffff80162a2f144 dpinger
100726                   S       nanslp   0xffffffff82865d22 dpinger
100727                   S       nanslp   0xffffffff82865d20 dpinger
100728                   S       accept   0xfffff801620e2a8c dpinger
89259     1 89259     0  Ss      select   0xfffff8001d4c1e40 openvpn
87846     1 87846     0  Ss      select   0xfffff8016220e7c0 openvpn
68550     1 68550     0  Ss      select   0xfffff80102d520c0 powerd
67165 67155 67165     0  S+      ttyin    0xfffff8001d3dfca8 sh
67155 65512 67155     0  S+      wait     0xfffff8016212f528 sh
66898 98897 66898     0  Ss      (threaded)                  sshlockout_pf
100605                   S       piperd   0xfffff8001d9db000 sshlockout_pf
100683                   S       nanslp   0xffffffff82865d20 sshlockout_pf
66742     1 66742     0  Ss+     ttyin    0xfffff8001d46b0a8 getty
66646     1 66646     0  Ss+     ttyin    0xfffff8001d46b4a8 getty
66575     1 66575     0  Ss+     ttyin    0xfffff8001d46b8a8 getty
66538     1 66538     0  Ss+     ttyin    0xfffff8001d46bca8 getty
66211     1 66211     0  Ss+     ttyin    0xfffff8001d46c0a8 getty
66158     1 66158     0  Ss+     ttyin    0xfffff80005179ca8 getty
65836     1 65836     0  Ss+     ttyin    0xfffff8001d3df0a8 getty
65512     1 65512     0  Ss+     wait     0xfffff8001d976a50 login
50681 50482 50482     0  S       nanslp   0xffffffff82865d20 minicron
50482     1 50482     0  Ss      wait     0xfffff8001df18a50 minicron
50272 49938 49938     0  S       nanslp   0xffffffff82865d23 minicron
49938     1 49938     0  Ss      wait     0xfffff8001d976528 minicron
49476 49314 49314     0  S       nanslp   0xffffffff82865d23 minicron
49314     1 49314     0  Ss      wait     0xfffff8016212aa50 minicron
44216     1 44216     0  Ss      (threaded)                  apcupsd
100658                   S       nanslp   0xffffffff82865d21 apcupsd
100670                   S       sigwait  0xfffff80162a01000 apcupsd
100671                   S       accept   0xfffff8016212206c apcupsd
 3948     1  3848     0  S       kqread   0xfffff8001dc25200 lighttpd_pfb
98897     1 98897     0  Ss      select   0xfffff80102d509c0 syslogd
53013     1 53013     0  Ss      kqread   0xfffff8001dbe1600 dnscrypt-proxy
52232     1 52232     0  Ss      kqread   0xfffff8001db85000 dnscrypt-proxy
50361     1 50361     0  Ss      kqread   0xfffff8001df2d900 dnscrypt-proxy
47847     1 47764     0  S       select   0xfffff8003b3464c0 radvd
40799     1 40799     0  Ss      (threaded)                  ntpd
100646                   S       select   0xfffff8003b1408c0 ntpd
40296     0     0     0  DL      mdwait   0xfffff80102bf3000 [md3]
38103     1 38103     0  Ss      nanslp   0xffffffff82865d22 cron
37771 37268 37268     0  S       kqread   0xfffff8001dc40600 nginx
37735 37268 37268     0  S       kqread   0xfffff8001d5b0700 nginx
37705 37268 37268     0  S       kqread   0xfffff8001dc1c900 nginx
37554 37268 37268     0  S       kqread   0xfffff8001dbbaa00 nginx
37494 37268 37268     0  S       kqread   0xfffff8001db43a00 nginx
37268     1 37268     0  Ss      pause    0xfffff80102eecaf8 nginx
20138     1 20138     0  Ss      bpf      0xfffff8003bd67000 filterlog
15349     1 15349     0  Ss      select   0xfffff8003b13e1c0 dhcp6c
14626     1 14626    65  Ss      select   0xfffff8001dc38ec0 dhclient
 9309     1  9309     0  Ss      select   0xfffff8003b3460c0 dhclient
 8519     1  8519     0  Ss      (threaded)                  sshlockout_pf
100597                   S       uwait    0xfffff8001dc37600 sshlockout_pf
100601                   S       nanslp   0xffffffff82865d20 sshlockout_pf
 8180     1  8180     0  Ss      select   0xfffff8001d4c2640 sshd
  393     0     0     0  DL      mdwait   0xfffff8001daf1800 [md2]
  383     0     0     0  DL      mdwait   0xfffff8003b132000 [md1]
  368     1   368     0  Ss      select   0xfffff8003b1411c0 devd
  350   348   348     0  S       kqread   0xfffff8001dcabd00 check_reload_status
  348     1   348     0  Ss      kqread   0xfffff8001db94800 check_reload_status
  334     1   334     0  Ss      kqread   0xfffff8001ddd2100 php-fpm
  103     0     0     0  DL      mdwait   0xfffff8001d565000 [md0]
   27     0     0     0  DL      syncer   0xffffffff8295f880 [syncer]
   26     0     0     0  DL      vlruwt   0xfffff8001d406528 [vnlru]
   25     0     0     0  DL      psleep   0xffffffff8295e104 [bufdaemon]
   24     0     0     0  DL      -        0xffffffff8295edb4 [bufspacedaemon]
   23     0     0     0  DL      pgzero   0xffffffff829734e4 [pagezero]
   22     0     0     0  DL      psleep   0xffffffff8296f98c [vmdaemon]
   21     0     0     0  DL      (threaded)                  [pagedaemon]
100072                   D       psleep   0xffffffff82a1d105 [pagedaemon]
100081                   D       umarcl   0xffffffff8296f2b8 [uma]
   20     0     0     0  DL      -        0xffffffff8295d914 [soaiod4]
   19     0     0     0  DL      -        0xffffffff8295d914 [soaiod3]
   18     0     0     0  DL      -        0xffffffff8295d914 [soaiod2]
    9     0     0     0  DL      -        0xffffffff8295d914 [soaiod1]
   17     0     0     0  DL      cooling  0xfffff80005139f58 [acpi_cooling0]
   16     0     0     0  DL      tzpoll   0xffffffff82621138 [acpi_thermal]
    8     0     0     0  DL      -        0xffffffff8273a4f0 [rand_harvestq]
    7     0     0     0  DL      pftm     0xffffffff80f4b6a0 [pf purge]
    6     0     0     0  DL      waiting_ 0xffffffff82a0c740 [sctp_iterator]
    5     0     0     0  DL      (threaded)                  [zfskern]
100055                   D       arc_recl 0xffffffff82f3f4d0 [arc_reclaim_thread]
100056                   D       arc_user 0xffffffff82f3f4f0 [arc_user_evicts_thr]
100057                   D       arc_dnlc 0xffffffff82f3f480 [arc_dnlc_evicts_thr]
100058                   D       l2arc_fe 0xffffffff82f3f888 [l2arc_feed_thread]
100550                   D       spa->spa 0xfffff8001d55fbd0 [trim zroot]
100564                   D       tx->tx_q 0xfffff8001d45fa30 [txg_thread_enter]
100565                   D       tx->tx_s 0xfffff8001d45fa10 [txg_thread_enter]
   15     0     0     0  DL      (threaded)                  [usb]
100035                   D       -        0xfffffe0001121460 [usbus0]
100036                   D       -        0xfffffe00011214b8 [usbus0]
100037                   D       -        0xfffffe0001121510 [usbus0]
100038                   D       -        0xfffffe0001121568 [usbus0]
100039                   D       -        0xfffffe00011215c0 [usbus0]
    4     0     0     0  DL      (threaded)                  [cam]
100032                   D       -        0xffffffff8260d380 [doneq0]
100067                   D       -        0xffffffff8260d1c8 [scanner]
    3     0     0     0  DL      crypto_r 0xffffffff8296de70 [crypto returns]
    2     0     0     0  DL      crypto_w 0xffffffff8296dd18 [crypto]
   14     0     0     0  DL      (threaded)                  [geom]
100023                   D       -        0xffffffff829e4940 [g_event]
100024                   D       -        0xffffffff829e4948 [g_up]
100025                   D       -        0xffffffff829e4950 [g_down]
   13     0     0     0  DL      (threaded)                  [ng_queue]
100019                   D       sleep    0xffffffff825cb230 [ng_queue0]
100020                   D       sleep    0xffffffff825cb230 [ng_queue1]
100021                   D       sleep    0xffffffff825cb230 [ng_queue2]
100022                   D       sleep    0xffffffff825cb230 [ng_queue3]
   12     0     0     0  WL      (threaded)                  [intr]
100007                   I                                   [swi3: vm]
100008                   I                                   [swi4: clock (0)]
100009                   I                                   [swi4: clock (1)]
100010                   I                                   [swi4: clock (2)]
100011                   I                                   [swi4: clock (3)]
100012                   I                                   [swi1: netisr 0]
100013                   I                                   [swi5: fast taskq]
100015                   I                                   [swi6: task queue]
100016                   I                                   [swi6: Giant taskq]
100033                   I                                   [irq256: ahci0]
100034                   I                                   [irq257: xhci0]
100040                   I                                   [irq258: re0]
100041                   I                                   [irq259: pcib2]
100042                   I                                   [irq260: re1]
100043                   I                                   [irq261: igb0:que 0]
100045                   I                                   [irq262: igb0:link]
100046                   I                                   [irq263: igb1:que 0]
100048                   I                                   [irq264: igb1:link]
100061                   I                                   [swi1: pf send]
100062                   I                                   [swi1: pfsync]
100079                   I                                   [swi1: netisr 1]
100080                   I                                   [swi1: netisr 2]
100082                   I                                   [swi1: netisr 3]
   11     0     0     0  RL      (threaded)                  [idle]
100003                   CanRun                              [idle: cpu0]
100004                   CanRun                              [idle: cpu1]
100005                   CanRun                              [idle: cpu2]
100006                   CanRun                              [idle: cpu3]
    1     0     1     0  SLs     wait     0xfffff80004908528 [init]
   10     0     0     0  DL      audit_wo 0xffffffff82a138c0 [audit]
    0     0     0     0  DLs     (threaded)                  [kernel]
100000                   D       swapin   0xffffffff829e4978 [swapper]
100014                   D       -        0xfffff8000491ee00 [kqueue_ctx taskq]
100017                   D       -        0xfffff8000491e600 [aiod_kick taskq]
100018                   D       -        0xfffff8000491e400 [thread]
100026                   D       -        0xfffff8000491d600 [firmware taskq]
100029                   D       -        0xfffff80004ae4d00 [acpi_task_0]
100030                   D       -        0xfffff80004ae4d00 [acpi_task_1]
100031                   D       -        0xfffff80004ae4d00 [acpi_task_2]
100044                   D       -        0xfffff80005106b00 [igb0 que]
100047                   D       -        0xfffff80004ba4d00 [igb1 que]
100049                   D       -        0xfffff80005162800 [system_taskq_0]
100050                   D       -        0xfffff80005162800 [system_taskq_1]
100051                   D       -        0xfffff80005162800 [system_taskq_2]
100052                   D       -        0xfffff80005162800 [system_taskq_3]
100053                   D       -        0xfffff80004ae5000 [mca taskq]
100054                   D       -        0xfffff80005161b00 [dbu_evict]
100066                   D       -        0xfffff80004ae4900 [CAM taskq]
100083                   D       -        0xfffff8001d467300 [if_io_tqg_0]
100084                   D       -        0xfffff8001d467100 [if_io_tqg_1]
100085                   D       -        0xfffff8001d4a4b00 [if_io_tqg_2]
100086                   D       -        0xfffff8001d4a4900 [if_io_tqg_3]
100087                   D       -        0xfffff8001d4a4700 [if_config_tqg_0]
100322                   D       -        0xfffff8001d5fed00 [zio_null_issue]
100323                   D       -        0xfffff8001d5b2600 [zio_null_intr]
100324                   D       -        0xfffff8001d5b2300 [zio_read_issue_0]
100325                   D       -        0xfffff8001d5b2300 [zio_read_issue_1]
100326                   D       -        0xfffff8001d5b2300 [zio_read_issue_2]
100327                   D       -        0xfffff8001d5b2300 [zio_read_issue_3]
100328                   D       -        0xfffff8001d5b2300 [zio_read_issue_4]
100329                   D       -        0xfffff8001d5b2300 [zio_read_issue_5]
100330                   D       -        0xfffff8001d5b2300 [zio_read_issue_6]
100331                   D       -        0xfffff8001d5b2300 [zio_read_issue_7]
100332                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_0]
100333                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_1]
100334                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_2]
100335                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_3]
100336                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_4]
100337                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_5]
100338                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_6]
100339                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_7]
100340                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_8]
100341                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_9]
100342                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_10]
100343                   D       -        0xfffff8001d5ff700 [zio_read_intr_0_11]
100344                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_0]
100345                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_1]
100346                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_2]
100347                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_3]
100348                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_4]
100349                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_5]
100350                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_6]
100351                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_7]
100352                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_8]
100353                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_9]
100354                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_10]
100355                   D       -        0xfffff8001d5b2200 [zio_read_intr_1_11]
100356                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_0]
100357                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_1]
100358                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_2]
100359                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_3]
100360                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_4]
100361                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_5]
100362                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_6]
100363                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_7]
100364                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_8]
100365                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_9]
100366                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_10]
100367                   D       -        0xfffff8001d5fda00 [zio_read_intr_2_11]
100368                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_0]
100369                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_1]
100370                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_2]
100371                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_3]
100372                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_4]
100373                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_5]
100374                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_6]
100375                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_7]
100376                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_8]
100377                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_9]
100378                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_10]
100379                   D       -        0xfffff8001d5ddc00 [zio_read_intr_3_11]
100380                   D       -        0xfffff8001d552500 [zio_read_intr_4_0]
100381                   D       -        0xfffff8001d552500 [zio_read_intr_4_1]
100382                   D       -        0xfffff8001d552500 [zio_read_intr_4_2]
100383                   D       -        0xfffff8001d552500 [zio_read_intr_4_3]
100384                   D       -        0xfffff8001d552500 [zio_read_intr_4_4]
100385                   D       -        0xfffff8001d552500 [zio_read_intr_4_5]
100386                   D       -        0xfffff8001d552500 [zio_read_intr_4_6]
100387                   D       -        0xfffff8001d552500 [zio_read_intr_4_7]
100388                   D       -        0xfffff8001d552500 [zio_read_intr_4_8]
100389                   D       -        0xfffff8001d552500 [zio_read_intr_4_9]
100390                   D       -        0xfffff8001d552500 [zio_read_intr_4_10]
100391                   D       -        0xfffff8001d552500 [zio_read_intr_4_11]
100392                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_0]
100393                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_1]
100394                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_2]
100395                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_3]
100396                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_4]
100397                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_5]
100398                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_6]
100399                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_7]
100400                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_8]
100401                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_9]
100402                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_10]
100403                   D       -        0xfffff8001d4a2800 [zio_read_intr_5_11]
100404                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_0]
100405                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_1]
100406                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_2]
100407                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_3]
100408                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_4]
100409                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_5]
100410                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_6]
100411                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_7]
100412                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_8]
100413                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_9]
100414                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_10]
100415                   D       -        0xfffff8001d4a2300 [zio_read_intr_6_11]
100416                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_0]
100417                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_1]
100418                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_2]
100419                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_3]
100420                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_4]
100421                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_5]
100422                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_6]
100423                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_7]
100424                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_8]
100425                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_9]
100426                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_10]
100427                   D       -        0xfffff8001d4a1000 [zio_read_intr_7_11]
100428                   D       -        0xfffff8001d589300 [zio_write_issue_0]
100429                   D       -        0xfffff8001d589300 [zio_write_issue_1]
100430                   D       -        0xfffff8001d589300 [zio_write_issue_2]
100431                   D       -        0xfffff8001d589200 [zio_write_issue_hig]
100432                   D       -        0xfffff8001d589200 [zio_write_issue_hig]
100433                   D       -        0xfffff8001d589200 [zio_write_issue_hig]
100434                   D       -        0xfffff8001d589200 [zio_write_issue_hig]
100435                   D       -        0xfffff8001d589200 [zio_write_issue_hig]
100436                   D       -        0xfffff8001d589000 [zio_write_intr_0]
100437                   D       -        0xfffff8001d589000 [zio_write_intr_1]
100438                   D       -        0xfffff8001d589000 [zio_write_intr_2]
100439                   D       -        0xfffff8001d589000 [zio_write_intr_3]
100440                   D       -        0xfffff8001d589000 [zio_write_intr_4]
100441                   D       -        0xfffff8001d589000 [zio_write_intr_5]
100442                   D       -        0xfffff8001d589000 [zio_write_intr_6]
100443                   D       -        0xfffff8001d589000 [zio_write_intr_7]
100444                   D       -        0xfffff8001d588100 [zio_write_intr_high]
100445                   D       -        0xfffff8001d588100 [zio_write_intr_high]
100446                   D       -        0xfffff8001d588100 [zio_write_intr_high]
100447                   D       -        0xfffff8001d588100 [zio_write_intr_high]
100448                   D       -        0xfffff8001d588100 [zio_write_intr_high]
100449                   D       -        0xfffff8001d587800 [zio_free_issue_0_0]
100450                   D       -        0xfffff8001d587800 [zio_free_issue_0_1]
100451                   D       -        0xfffff8001d587800 [zio_free_issue_0_2]
100452                   D       -        0xfffff8001d587800 [zio_free_issue_0_3]
100453                   D       -        0xfffff8001d587800 [zio_free_issue_0_4]
100454                   D       -        0xfffff8001d587800 [zio_free_issue_0_5]
100455                   D       -        0xfffff8001d587800 [zio_free_issue_0_6]
100456                   D       -        0xfffff8001d587800 [zio_free_issue_0_7]
100457                   D       -        0xfffff8001d587800 [zio_free_issue_0_8]
100458                   D       -        0xfffff8001d587800 [zio_free_issue_0_9]
100459                   D       -        0xfffff8001d587800 [zio_free_issue_0_10]
100460                   D       -        0xfffff8001d587800 [zio_free_issue_0_11]
100461                   D       -        0xfffff8001d587100 [zio_free_issue_1_0]
100462                   D       -        0xfffff8001d587100 [zio_free_issue_1_1]
100463                   D       -        0xfffff8001d587100 [zio_free_issue_1_2]
100464                   D       -        0xfffff8001d587100 [zio_free_issue_1_3]
100465                   D       -        0xfffff8001d587100 [zio_free_issue_1_4]
100466                   D       -        0xfffff8001d587100 [zio_free_issue_1_5]
100467                   D       -        0xfffff8001d587100 [zio_free_issue_1_6]
100468                   D       -        0xfffff8001d587100 [zio_free_issue_1_7]
100469                   D       -        0xfffff8001d587100 [zio_free_issue_1_8]
100470                   D       -        0xfffff8001d587100 [zio_free_issue_1_9]
100471                   D       -        0xfffff8001d587100 [zio_free_issue_1_10]
100472                   D       -        0xfffff8001d587100 [zio_free_issue_1_11]
100473                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_0]
100474                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_1]
100475                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_2]
100476                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_3]
100477                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_4]
100478                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_5]
100479                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_6]
100480                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_7]
100481                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_8]
100482                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_9]
100483                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_10]
100484                   D       -        0xfffff8001d5b1d00 [zio_free_issue_2_11]
100485                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_0]
100486                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_1]
100487                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_2]
100488                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_3]
100489                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_4]
100490                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_5]
100491                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_6]
100492                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_7]
100493                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_8]
100494                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_9]
100495                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_10]
100496                   D       -        0xfffff8001d5b1000 [zio_free_issue_3_11]
100497                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_0]
100498                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_1]
100499                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_2]
100500                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_3]
100501                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_4]
100502                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_5]
100503                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_6]
100504                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_7]
100505                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_8]
100506                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_9]
100507                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_10]
100508                   D       -        0xfffff8001d5afb00 [zio_free_issue_4_11]
100509                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_0]
100510                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_1]
100511                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_2]
100512                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_3]
100513                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_4]
100514                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_5]
100515                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_6]
[/thread]

[chrcoluk]

continued here - hit post limit

100516                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_7]
100517                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_8]
100518                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_9]
100519                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_10]
100520                   D       -        0xfffff8001d58ab00 [zio_free_issue_5_11]
100521                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_0]
100522                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_1]
100523                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_2]
100524                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_3]
100525                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_4]
100526                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_5]
100527                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_6]
100528                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_7]
100529                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_8]
100530                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_9]
100531                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_10]
100532                   D       -        0xfffff8001d5dd200 [zio_free_issue_6_11]
100533                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_0]
100534                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_1]
100535                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_2]
100536                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_3]
100537                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_4]
100538                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_5]
100539                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_6]
100540                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_7]
100541                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_8]
100542                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_9]
100543                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_10]
100544                   D       -        0xfffff8001d5dc400 [zio_free_issue_7_11]
100545                   D       -        0xfffff8001d5db700 [zio_free_intr]
100546                   D       -        0xfffff8001d5db300 [zio_claim_issue]
100547                   D       -        0xfffff8001d5db600 [zio_claim_intr]
100548                   D       -        0xfffff8001d5dae00 [zio_ioctl_issue]
100549                   D       -        0xfffff8001d5dab00 [zio_ioctl_intr]
100551                   D       -        0xfffff8001d5dac00 [metaslab_group_task]
100552                   D       -        0xfffff8001d5dac00 [metaslab_group_task]
100553                   D       -        0xfffff8001d5da000 [zfs_vn_rele_taskq]
100566                   D       -        0xfffff8001d8dd200 [zil_clean]
100572                   D       -        0xfffff8001daa6600 [zil_clean]
100573                   D       -        0xfffff8001d9d8a00 [zil_clean]
100574                   D       -        0xfffff8001d6fd100 [zil_clean]
oops, ran out of processes early!
db:0:kdb.enter.default>  alltrace

Tracing command sh pid 54226 tid 100593 td 0xfffff8001dece000
cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff82977730
ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xffffffff82977750
trap() at trap+0x3a/frame 0xffffffff82977960
nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff82977960
--- trap 0x13, rip = 0xffffffff811a4a14, rsp = 0xfffffe0113a4f490, rbp = 0xfffffe0113a4f560 ---
pmap_remove_pages() at pmap_remove_pages+0x384/frame 0xfffffe0113a4f560
exec_new_vmspace() at exec_new_vmspace+0x21d/frame 0xfffffe0113a4f5e0
exec_elf64_imgact() at exec_elf64_imgact+0xa58/frame 0xfffffe0113a4f6f0
kern_execve() at kern_execve+0x74d/frame 0xfffffe0113a4fa50
sys_execve() at sys_execve+0x4a/frame 0xfffffe0113a4fad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113a4fbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113a4fbf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x800b40d8a, rsp = 0x7fffffffe188, rbp = 0x7fffffffe2d0 ---

Tracing command sh pid 54120 tid 100774 td 0xfffff8015e162a00
turnstile_broadcast() at turnstile_broadcast+0x9c/frame 0xfffffe0113c6d480
__rw_wunlock_hard() at __rw_wunlock_hard+0x8f/frame 0xfffffe0113c6d4b0
vm_map_delete() at vm_map_delete+0x3dc/frame 0xfffffe0113c6d530
vm_map_remove() at vm_map_remove+0x47/frame 0xfffffe0113c6d560
exec_new_vmspace() at exec_new_vmspace+0x22f/frame 0xfffffe0113c6d5e0
exec_elf64_imgact() at exec_elf64_imgact+0xa58/frame 0xfffffe0113c6d6f0
kern_execve() at kern_execve+0x74d/frame 0xfffffe0113c6da50
sys_execve() at sys_execve+0x4a/frame 0xfffffe0113c6dad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c6dbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c6dbf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x800b40d8a, rsp = 0x7fffffffe188, rbp = 0x7fffffffe2d0 ---

Tracing command cat pid 54035 tid 100570 td 0xfffff8001d601000
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00ea3f4cf0
ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe00ea3f4d10
trap() at trap+0x3a/frame 0xfffffe00ea3f4f20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00ea3f4f20
--- trap 0x13, rip = 0xffffffff82e91de0, rsp = 0xfffffe01137de588, rbp = 0xfffffe01137de5b0 ---
zrl_add_impl() at zrl_add_impl/frame 0xfffffe01137de5b0
zfs_freebsd_getattr() at zfs_freebsd_getattr+0xa2e/frame 0xfffffe01137de820
VOP_GETATTR_APV() at VOP_GETATTR_APV+0x87/frame 0xfffffe01137de850
vm_mmap_vnode() at vm_mmap_vnode+0x164/frame 0xfffffe01137de990
vn_mmap() at vn_mmap+0xe5/frame 0xfffffe01137dea20
sys_mmap() at sys_mmap+0x40c/frame 0xfffffe01137dead0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe01137debf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe01137debf0
--- syscall (477, FreeBSD ELF64, sys_mmap), rip = 0x800613ffa, rsp = 0x7fffffffde68, rbp = 0x7fffffffdfc0 ---

Tracing command sh pid 53957 tid 101116 td 0xfffff80162b5a500
cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe00ea3fccf0
ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xfffffe00ea3fcd10
trap() at trap+0x3a/frame 0xfffffe00ea3fcf20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00ea3fcf20
--- trap 0x13, rip = 0xffffffff811a3b4b, rsp = 0xfffffe0113c9a850, rbp = 0xfffffe0113c9a890 ---
pmap_try_insert_pv_entry() at pmap_try_insert_pv_entry+0x13b/frame 0xfffffe0113c9a890
pmap_copy() at pmap_copy+0x65c/frame 0xfffffe0113c9a930
vmspace_fork() at vmspace_fork+0x8d9/frame 0xfffffe0113c9a9b0
fork1() at fork1+0x5ec/frame 0xfffffe0113c9aa70
sys_fork() at sys_fork+0x39/frame 0xfffffe0113c9aad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c9abf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c9abf0
--- syscall (2, FreeBSD ELF64, sys_fork), rip = 0x800b5d77a, rsp = 0x7fffffffe508, rbp = 0x7fffffffe540 ---

Tracing command php-fpm pid 39824 tid 100686 td 0xfffff8003b242000
sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113bb2850
mi_switch() at mi_switch+0xd2/frame 0xfffffe0113bb2880
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113bb2900
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113bb2930
_sleep() at _sleep+0x294/frame 0xfffffe0113bb29c0
kern_accept4() at kern_accept4+0x1f2/frame 0xfffffe0113bb2a70
accept1() at accept1+0x60/frame 0xfffffe0113bb2ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113bb2bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113bb2bf0
--- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x801d947da, rsp = 0x7ffffffec848, rbp = 0x7ffffffec880 ---

Tracing command top pid 43353 tid 100695 td 0xfffff80162d08a00
sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113b35700
mi_switch() at mi_switch+0xd2/frame 0xfffffe0113b35730
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113b357b0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113b357e0
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe0113b35850
seltdwait() at seltdwait+0xc7/frame 0xfffffe0113b358a0
kern_select() at kern_select+0x979/frame 0xfffffe0113b35a90
sys_select() at sys_select+0x54/frame 0xfffffe0113b35ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113b35bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113b35bf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80121000a, rsp = 0x7fffffffe888, rbp = 0x7fffffffea60 ---

Tracing command bash pid 32987 tid 100680 td 0xfffff801628d1000
sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113b946a0
mi_switch() at mi_switch+0xd2/frame 0xfffffe0113b946d0
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113b94750
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113b94780
_sleep() at _sleep+0x294/frame 0xfffffe0113b94810
kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe0113b948e0
sys_wait4() at sys_wait4+0x73/frame 0xfffffe0113b94ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113b94bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113b94bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800e5d57a, rsp = 0x7fffffffe7a8, rbp = 0x7fffffffe7f0 ---

Tracing command sh pid 2745 tid 100748 td 0xfffff8015e029a00
sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113c366a0
mi_switch() at mi_switch+0xd2/frame 0xfffffe0113c366d0
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113c36750
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113c36780
_sleep() at _sleep+0x294/frame 0xfffffe0113c36810
kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe0113c368e0
sys_wait4() at sys_wait4+0x73/frame 0xfffffe0113c36ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c36bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c36bf0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800b5d57a, rsp = 0x7fffffffe768, rbp = 0x7fffffffe820 ---

Tracing command sshd pid 2354 tid 100699 td 0xfffff8001dd13000
sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113bbc700
mi_switch() at mi_switch+0xd2/frame 0xfffffe0113bbc730
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113bbc7b0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113bbc7e0
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe0113bbc850
seltdwait() at seltdwait+0xc7/frame 0xfffffe0113bbc8a0
kern_select() at kern_select+0x979/frame 0xfffffe0113bbca90
sys_select() at sys_select+0x54/frame 0xfffffe0113bbcad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113bbcbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113bbcbf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x801db400a, rsp = 0x7fffffffe5a8, rbp = 0x7fffffffe5e0 ---

Tracing command php pid 21591 tid 100704 td 0xfffff8001d4b5500
sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113bd58c0
mi_switch() at mi_switch+0xd2/frame 0xfffffe0113bd58f0
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113bd5970
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113bd59a0
_sleep() at _sleep+0x26f/frame 0xfffffe0113bd5a30
kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe0113bd5a90
sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe0113bd5ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113bd5bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113bd5bf0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x801d8571a, rsp = 0x7fffffffc7e8, rbp = 0x7fffffffc810 ---

Tracing command upsd pid 21318 tid 100739 td 0xfffff80162cd4500
sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113c09770
mi_switch() at mi_switch+0xd2/frame 0xfffffe0113c097a0
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113c09820
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113c09850
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe0113c098c0
seltdwait() at seltdwait+0xc7/frame 0xfffffe0113c09910
kern_poll() at kern_poll+0x296/frame 0xfffffe0113c09ab0
sys_poll() at sys_poll+0x61/frame 0xfffffe0113c09ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c09bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c09bf0
--- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x800d4fd9a, rsp = 0x7fffffffe668, rbp = 0x7fffffffe6a0 ---

Tracing command apcupsd-ups pid 21063 tid 100611 td 0xfffff8003be18500
sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113aa9700
mi_switch() at mi_switch+0xd2/frame 0xfffffe0113aa9730
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113aa97b0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113aa97e0
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe0113aa9850
seltdwait() at seltdwait+0xc7/frame 0xfffffe0113aa98a0
kern_select() at kern_select+0x979/frame 0xfffffe0113aa9a90
sys_select() at sys_select+0x54/frame 0xfffffe0113aa9ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113aa9bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113aa9bf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800b9b00a, rsp = 0x7fffffffe678, rbp = 0x7fffffffe6b0 ---

Tracing command upsmon pid 19999 tid 100703 td 0xfffff8001d4b5a00
sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113bd08c0
mi_switch() at mi_switch+0xd2/frame 0xfffffe0113bd08f0
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113bd0970
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113bd09a0
_sleep() at _sleep+0x26f/frame 0xfffffe0113bd0a30
kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe0113bd0a90
sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe0113bd0ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113bd0bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113bd0bf0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800d3c71a, rsp = 0x7fffffffe5c8, rbp = 0x7fffffffe5f0 ---

Tracing command upsmon pid 19580 tid 100685 td 0xfffff80162d0a500
sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113bad810
mi_switch() at mi_switch+0xd2/frame 0xfffffe0113bad840
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113bad8c0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113bad8f0
_sleep() at _sleep+0x294/frame 0xfffffe0113bad980
pipe_read() at pipe_read+0x36d/frame 0xfffffe0113bad9e0
dofileread() at dofileread+0x98/frame 0xfffffe0113bada30
kern_readv() at kern_readv+0x68/frame 0xfffffe0113bada80
sys_read() at sys_read+0x84/frame 0xfffffe0113badad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113badbf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113badbf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800da502a, rsp = 0x7fffffffe628, rbp = 0x7fffffffe660 ---

Tracing command sh pid 33060 tid 100578 td 0xfffff8001d4e5a00
sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113a04810
mi_switch() at mi_switch+0xd2/frame 0xfffffe0113a04840
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113a048c0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113a048f0
_sleep() at _sleep+0x294/frame 0xfffffe0113a04980
pipe_read() at pipe_read+0x36d/frame 0xfffffe0113a049e0
dofileread() at dofileread+0x98/frame 0xfffffe0113a04a30
kern_readv() at kern_readv+0x68/frame 0xfffffe0113a04a80
sys_read() at sys_read+0x84/frame 0xfffffe0113a04ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113a04bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113a04bf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800bc602a, rsp = 0x7fffffffe6a8, rbp = 0x7fffffffe880 ---

Tracing command dhcpd pid 15682 tid 100713 td 0xfffff80162d08000
sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113c04700
mi_switch() at mi_switch+0xd2/frame 0xfffffe0113c04730
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113c047b0
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0113c047e0
_cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe0113c04850
seltdwait() at seltdwait+0xc7/frame 0xfffffe0113c048a0
kern_select() at kern_select+0x979/frame 0xfffffe0113c04a90
sys_select() at sys_select+0x54/frame 0xfffffe0113c04ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c04bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113c04bf0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800d6400a, rsp = 0x7fffffffea38, rbp = 0x7fffffffea60 ---

Tracing command unbound pid 88257 tid 100711 td 0xfffff80162cc2a00
sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113bf8660
mi_switch() at mi_switch+0xd2/frame 0xfffffe0113bf8690
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113bf8710
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113bf8740
_sleep() at _sleep+0x294/frame 0xfffffe0113bf87d0
kqueue_kevent() at kqueue_kevent+0x336/frame 0xfffffe0113bf89b0
kern_kevent() at kern_kevent+0xb5/frame 0xfffffe0113bf8a20
sys_kevent() at sys_kevent+0x11c/frame 0xfffffe0113bf8ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113bf8bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0113bf8bf0
--- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x80177275a, rsp = 0x7fffffffe9f8, rbp = 0x7fffffffea40 ---

Tracing command unbound pid 88257 tid 100760 td 0xfffff80162da8a00
sched_switch() at sched_switch+0x6cb/frame 0xfffffe0113c40660
mi_switch() at mi_switch+0xd2/frame 0xfffffe0113c40690
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0113c40710
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0113c40740
_sleep() at _sleep+0x294/frame 0xfffffe0113c407d0
kqueue_kevent() at kqueue_kevent+0x336/frame 0xfffffe0113c409b0
kern_kevent() at kern_kevent+0xb5/frame 0xfffffe0113c40a20
sys_kevent() at sys_kevent+0x11c/frame 0xfffffe0113c40ad0
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0113c40bf0
Xfast_syscall() at Xfast_syscall+0config.txt0600001341413115636557  7552 ustarrootwheeloptions	CONFIG_AUTOGENERATED
ident	pfSense
machine	amd64
cpu	HAMMER
makeoptions	WITH_CTF=1
makeoptions	DEBUG=-g
options	CONSPEED=115200
options	MROUTING
options	HIFN_RNDTEST
options	HIFN_DEBUG
options	MSGTQL=2048
options	MSGSSZ=32
options	MSGSEG=512
options	MSGMNI=40
options	MSGMNB=8192
options	ALTQ_CODEL
options	ALTQ_NOPCC
options	ALTQ_FAIRQ
options	ALTQ_PRIQ
options	ALTQ_HFSC
options	ALTQ_RIO
options	ALTQ_RED
options	ALTQ_CBQ
options	ALTQ
options	TCP_SIGNATURE
options	IPSEC_NAT_T
options	NETGRAPH_PRED1
options	NETGRAPH_DEFLATE
options	NETGRAPH_CAR
options	NETGRAPH_PIPE
options	NETGRAPH_TCPMSS
options	NETGRAPH_TEE
options	NETGRAPH_HOLE
options	NETGRAPH_FRAME_RELAY
options	NETGRAPH_ASYNC
options	NETGRAPH_ECHO
options	NETGRAPH_CISCO
options	NETGRAPH_BRIDGE
options	NETGRAPH_ONE2MANY
options	NETGRAPH_LMI
options	NETGRAPH_KSOCKET
options	NETGRAPH_VJC
options	NETGRAPH_UI
options	NETGRAPH_MPPC_ENCRYPTION
options	NETGRAPH_TTY
options	NETGRAPH_SOCKET
options	NETGRAPH_RFC1490
options	NETGRAPH_PPTPGRE
options	NETGRAPH_PPPOE
options	NETGRAPH_PPP
options	NETGRAPH_EIFACE
options	NETGRAPH_IFACE
options	NETGRAPH_ETHER
options	NETGRAPH_BPF
options	NETGRAPH_L2TP
options	NETGRAPH_VLAN
options	NETGRAPH
options	IPSTEALTH
options	IPFIREWALL_VERBOSE
options	IPFIREWALL_DEFAULT_TO_ACCEPT
options	PPS_SYNC
options	NULLFS
options	UNIONFS
options	TMPFS
options	GEOM_BDE
options	GEOM_ELI
options	GEOM_UZIP
options	GEOM_MIRROR
options	DDB
options	XENHVM
options	ATH_ENABLE_11N
options	AH_AR5416_INTERRUPT_MITIGATION
options	AH_SUPPORT_AR5416
options	IEEE80211_SUPPORT_MESH
options	IEEE80211_AMPDU_AGE
options	IEEE80211_DEBUG
options	SC_PIXEL_MODE
options	VESA
options	AHD_REG_PRETTY_PRINT
options	AHC_REG_PRETTY_PRINT
options	PCI_IOV
options	PCI_HP
options	ACPI_DMAR
options	DEVICE_NUMA
options	SMP
options	KDB
options	RCTL
options	RACCT_DEFAULT_TO_DISABLED
options	RACCT
options	INCLUDE_CONFIG_FILE
options	DDB_CTF
options	KDTRACE_HOOKS
options	KDTRACE_FRAME
options	CAPABILITIES
options	CAPABILITY_MODE
options	AUDIT
options	HWPMC_HOOKS
options	KBD_INSTALL_CDEV
options	PRINTF_BUFR_SIZE=128
options	_KPOSIX_PRIORITY_SCHEDULING
options	SYSVSEM
options	SYSVMSG
options	SYSVSHM
options	STACK
options	KTRACE
options	SCSI_DELAY=5000
options	COMPAT_FREEBSD10
options	COMPAT_FREEBSD9
options	COMPAT_FREEBSD32
options	GEOM_LABEL
options	GEOM_RAID
options	GEOM_PART_GPT
options	PSEUDOFS
options	PROCFS
options	CD9660
options	MSDOSFS
options	NFS_ROOT
options	NFSLOCKD
options	NFSD
options	NFSCL
options	MD_ROOT
options	QUOTA
options	UFS_GJOURNAL
options	UFS_DIRHASH
options	UFS_ACL
options	SOFTUPDATES
options	FFS
options	SCTP
options	TCP_OFFLOAD
options	IPSEC
options	INET6
options	INET
options	PREEMPTION
options	SCHED_ULE
options	NEW_PCIB
options	GEOM_PART_MBR
options	GEOM_PART_EBR_COMPAT
options	GEOM_PART_EBR
options	GEOM_PART_BSD
device	isa
device	mem
device	io
device	uart_ns8250
device	cpufreq
device	acpi
device	pci
device	fdc
device	ahci
device	ata
device	mvs
device	siis
device	ahc
device	ahd
device	esp
device	hptiop
device	isp
device	mpt
device	mps
device	mpr
device	sym
device	trm
device	adv
device	adw
device	aic
device	bt
device	isci
device	scbus
device	ch
device	da
device	sa
device	cd
device	pass
device	ses
device	amr
device	arcmsr
device	ciss
device	dpt
device	hptmv
device	hptnr
device	hptrr
device	hpt27xx
device	iir
device	ips
device	mly
device	twa
device	tws
device	aac
device	aacp
device	aacraid
device	ida
device	mfi
device	mlx
device	mrsas
device	pmspcv
device	twe
device	nvme
device	nvd
device	atkbdc
device	atkbd
device	psm
device	kbdmux
device	vga
device	splash
device	sc
device	vt
device	vt_vga
device	vt_efifb
device	agp
device	cbb
device	pccard
device	cardbus
device	uart
device	ppc
device	ppbus
device	lpt
device	ppi
device	puc
device	bxe
device	de
device	em
device	igb
device	ix
device	ixv
device	ixl
device	ixlv
device	le
device	ti
device	txp
device	vx
device	miibus
device	ae
device	age
device	alc
device	ale
device	bce
device	bfe
device	bge
device	cas
device	dc
device	et
device	fxp
device	gem
device	hme
device	jme
device	lge
device	msk
device	nfe
device	nge
device	pcn
device	re
device	rl
device	sf
device	sge
device	sis
device	sk
device	ste
device	stge
device	tl
device	tx
device	vge
device	vr
device	wb
device	xl
device	wlan
device	wlan_wep
device	wlan_ccmp
device	wlan_tkip
device	wlan_amrr
device	an
device	ath
device	ath_pci
device	ath_hal
device	ath_rate_sample
device	ipw
device	iwi
device	iwn
device	malo
device	mwl
device	ral
device	wi
device	wpi
device	loop
device	random
device	padlock_rng
device	rdrand_rng
device	ether
device	vlan
device	tun
device	md
device	gif
device	firmware
device	bpf
device	uhci
device	ohci
device	ehci
device	xhci
device	usb
device	ukbd
device	umass
device	sound
device	snd_cmi
device	snd_csa
device	snd_emu10kx
device	snd_es137x
device	snd_hda
device	snd_ich
device	snd_via8233
device	mmc
device	mmcsd
device	sdhci
device	virtio
device	virtio_pci
device	vtnet
device	virtio_blk
device	virtio_scsi
device	virtio_balloon
device	hyperv
device	xenpci
device	vmx
device	netmap
device	crypto
device	wlan_rssadapt
device	wlan_xauth
device	wlan_acl
device	iwifw
device	ipwfw
device	wpifw
device	iwnfw
device	uath
device	ralfw
device	ural
device	urtw
device	rum
device	mwlfw
device	zyd
device	upgt
device	udav
device	axe
device	axge
device	aue
device	cue
device	kue
device	mos
device	rsu
device	rsufw
device	run
device	runfw
device	rue
device	rtwn
device	rtwnfw
device	siba_bwn
device	bwn
device	bwi
device	ufoma
device	ucom
device	uslcom
device	uplcom
device	umct
device	uvisor
device	uark
device	uftdi
device	uvscom
device	umodem
device	u3g
device	cdce
device	uhid
device	firewire
device	sbp
device	tap
device	gre
device	if_bridge
device	carp
device	lagg
device	vte
device	enc
device	pf
device	pflog
device	pfsync
device	rndtest
device	hifn
device	ubsec
device	safe
device	padlock
device	speaker
device	mxge
device	cxgb
device	cxgbe
device	oce
msgbuf.txt06000017726213115636557  7664 ustarrootwheelCopyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-RELEASE-p8 #78 3f54ce3814d(RELENG_2_4): Wed Mar 15 18:23:59 CDT 2017
    root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense amd64
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
VT(efifb): resolution 1920x1080
CPU: Intel(R) Celeron(R) CPU  N3150  @ 1.60GHz (1600.05-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406c3  Family=0x6  Model=0x4c  Stepping=3
  Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x43d8e3bf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand>AMD Features=0x28100800 <syscall,nx,rdtscp,lm>AMD Features2=0x101 <lahf,prefetch>Structured Extended Features=0x2282 <tscadj,smep,erms,nfpusg>VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 4294967296 (4096 MB)
avail memory = 3877691392 (3698 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <alaska a="" m="" i="">WARNING: L1 data cache covers less APIC IDs than a core
0 < 1
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
random: unblocking device.
ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 128/32 (20160527/tbfadt-650)
ioapic0 <version 2.0="">irqs 0-114 on motherboard
random: entropy device external interface
wlan: mac acl policy registered
kbd0 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff8122b980, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
cryptosoft0: <software crypto="">on motherboard
padlock0: No ACE support.
acpi0: <alaska a="" m="" i="">on motherboard
acpi0: Power Button (fixed)
unknown: I/O range not supported
cpu0: <acpi cpu="">on acpi0
cpu1: <acpi cpu="">on acpi0
cpu2: <acpi cpu="">on acpi0
cpu3: <acpi cpu="">on acpi0
atrtc0: <at realtime="" clock="">port 0x70-0x77 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <at timer="">port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
pci0: <acpi pci="" bus="">on pcib0
vgapci0: <vga-compatible display="">port 0xf000-0xf03f mem 0x80000000-0x80ffffff,0x90000000-0x9fffffff irq 16 at device 2.0 on pci0
vgapci0: Boot video device
ahci0: <ahci sata="" controller="">port 0xf060-0xf07f mem 0x81511000-0x815117ff irq 19 at device 19.0 on pci0
ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier not supported
ahcich0: <ahci channel="">at channel 0 on ahci0
ahcich1: <ahci channel="">at channel 1 on ahci0
xhci0: <xhci (generic)="" usb="" 3.0="" controller="">mem 0x81500000-0x8150ffff irq 20 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
pci0: <encrypt decrypt="">at device 26.0 (no driver attached)
pcib1: <acpi pci-pci="" bridge="">irq 16 at device 28.0 on pci0
pci1: <acpi pci="" bus="">on pcib1
re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet="">port 0xe000-0xe0ff mem 0xfffe04000-0xfffe04fff,0xfffe00000-0xfffe03fff irq 16 at device 0.0 on pci1
re0: Using 1 MSI-X message
re0: ASPM disabled
re0: Chip rev. 0x2c800000
re0: MAC rev. 0x00100000
miibus0: <mii bus="">on re0
rgephy0: <rtl8169s 8211="" 8110s="" 1000base-t="" media="" interface="">PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 00:e0:4c:68:55:ae
re0: netmap queues/slots: TX 1/256, RX 1/256
pcib2: <acpi pci-pci="" bridge="">irq 17 at device 28.1 on pci0
pcib2: [GIANT-LOCKED]
pci2: <acpi pci="" bus="">on pcib2
re1: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet="">port 0xd000-0xd0ff mem 0xfffd04000-0xfffd04fff,0xfffd00000-0xfffd03fff irq 17 at device 0.0 on pci2
re1: Using 1 MSI-X message
re1: ASPM disabled
re1: Chip rev. 0x2c800000
re1: MAC rev. 0x00100000
miibus1: <mii bus="">on re1
rgephy1: <rtl8169s 8211="" 8110s="" 1000base-t="" media="" interface="">PHY 1 on miibus1
rgephy1:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re1: Using defaults for TSO: 65518/35/2048
re1: Ethernet address: 00:e0:4c:68:55:af
re1: netmap queues/slots: TX 1/256, RX 1/256
pcib3: <acpi pci-pci="" bridge="">irq 18 at device 28.2 on pci0
pci3: <acpi pci="" bus="">on pcib3
igb0: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k="">port 0xc020-0xc03f mem 0x81380000-0x813fffff,0x81404000-0x81407fff irq 18 at device 0.0 on pci3
igb0: Using MSIX interrupts with 2 vectors
igb0: Ethernet address: 00:30:18:c3:d0:e5
igb0: netmap queues/slots: TX 1/4096, RX 1/4096
igb1: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k="">port 0xc000-0xc01f mem 0x81280000-0x812fffff,0x81400000-0x81403fff irq 19 at device 0.1 on pci3
igb1: Using MSIX interrupts with 2 vectors
igb1: Ethernet address: 00:30:18:c3:d0:e6
igb1: netmap queues/slots: TX 1/4096, RX 1/4096
isab0: <pci-isa bridge="">at device 31.0 on pci0
isa0: <isa bus="">on isab0
acpi_button0: <sleep button="">on acpi0
acpi_tz0: <thermal zone="">on acpi0
ppc0: cannot reserve I/O port range
est0: <enhanced speedstep="" frequency="" control="">on cpu0
est1: <enhanced speedstep="" frequency="" control="">on cpu1
est2: <enhanced speedstep="" frequency="" control="">on cpu2
est3: <enhanced speedstep="" frequency="" control="">on cpu3
usbus0: 5.0Gbps Super Speed USB v3.0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
nvme cam probe device init
ugen0.1: <0x8086> at usbus0
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub0: 13 ports with 13 removable, self powered
ugen0.2: <american power="" conversion="">at usbus0
ugen0.3: <vendor 0x05e3="">at usbus0
uhub1: <vendor 2="" 9="" 0x05e3="" usb2.0="" hub,="" class="" 0,="" rev="" 2.00="" 85.36,="" addr="">on usbus0
uhub1: 4 ports with 4 removable, self powered
ugen0.4: <vendor 0x1997="">at usbus0
ukbd0: <vendor 0="" 3="" 0x1997="" mini="" keyboard,="" class="" 0,="" rev="" 1.10="" 20.12,="" addr="">on usbus0
kbd1 at ukbd0
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <kingston sms200s360g="" 608abbf0="">ATA8-ACS SATA 3.x device
ada0: Serial Number 50026B72630613B0
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 57241MB (117231408 512 byte sectors)
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
Timecounter "TSC" frequency 1600052960 Hz quality 1000
Trying to mount root from zfs:zroot/ROOT/default []...
<118>Configuring crash dumps...
<118>Using /dev/ada0p3 for dump device.
<118>Filesystems are clean, continuing...
<118>Mounting filesystems...
<118>Restoring contents of RAM disk store... done.
<118>
<118>     ___
<118> ___/ f \
<118>/ p \___/ Sense
<118>\___/   \
<118>    \___/
<118>
<118>Welcome to pfSense 2.4.0-BETA...
<118>
<118>No core dumps found.
<118>chmod: /var/empty: Operation not permitted
<118>...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/dnscrypt-proxy /usr/local/lib/graphviz /usr/local/lib/ipsec /usr/local/lib/mysql /usr/local/lib/perl5/5.24/mach/CORE
<118>32-bit compatibility ldconfig path:
<118>done.
<118>External config loader 1.0 is now starting...
<118>Launching the init system....... done.
<118>Initializing.................. done.
<118>Starting device manager (devd)...kldload: can't load ums: No such file or directory
<118>done.
<118>Loading configuration......done.
<118>tar: Removing leading '/' from member names
<118>Updating configuration...done.
<118>Cleaning backup cache.................................done.
<118>Setting up extended sysctls...done.
padlock0: No ACE support.
aesni0: <aes-cbc,aes-xts,aes-gcm,aes-icm>on motherboard
coretemp0: <cpu on-die="" thermal="" sensors="">on cpu0
coretemp1: <cpu on-die="" thermal="" sensors="">on cpu1
coretemp2: <cpu on-die="" thermal="" sensors="">on cpu2
coretemp3: <cpu on-die="" thermal="" sensors="">on cpu3
<118>Setting timezone...done.
<118>Configuring loopback interface...done.
<118>Starting syslog...done.
<118>Starting Secure Shell Services...done.
<118>Setting up interfaces microcode...done.
<118>Configuring loopback interface...done.
<118>Creating wireless clone interfaces...done.
<118>Configuring LAGG interfaces...done.
<118>Configuring VLAN interfaces...done.
<118>Configuring QinQ interfaces...done.
<118>Configuring WAN interface...
<5>igb0: link state changed to UP
<118>done.
<118>Configuring LAN interface...done.
<118>Configuring CARP settings...done.
<118>Syncing OpenVPN settings...
<6>tun1: changing name to 'ovpnc1'
<6>tun2: changing name to 'ovpnc2'
<118>done.
<6>pflog0: promiscuous mode enabled
<118>Configuring firewall.....0 addresses deleted.
<118>0 addresses deleted.
<118>.done.
<118>Starting PFLOG...done.
<118>Setting up gateway monitors...done.
<5>ovpnc2: link state changed to UP
<5>ovpnc1: link state changed to UP
<5>igb1: link state changed to UP
<118>Starting DNS Resolver...done.
<118>Synchronizing user settings...done.
<118>Starting webConfigurator...done.
<118>Configuring CRON...done.
<118>Starting NTP time client...done.
<118>Starting DHCP service...done.
<118>Starting DHCPv6 service...done.
<118>Configuring firewall.....0 addresses deleted.
<118>0 addresses deleted.
<118>0 addresses deleted.
<118>0 addresses deleted.
<118>.done.
<5>igb0: link state changed to DOWN
<5>igb1: link state changed to DOWN
<5>igb0: link state changed to UP
<5>igb1: link state changed to UP
<118>Generating RRD graphs...done.
<118>Starting syslog...done.
<118>Starting CRON... done.
<118> Starting package Cron...done.
<118> Starting package iftop...done.
<118> Starting package iperf...done.
<118> Starting package mtr-nox11...done.
<118> Starting package nmap...done.
<118> Starting package pfBlockerNG...done.
<118> Starting package darkstat...done.
<118> Starting package ntopng...done.
<118> Starting package Shellcmd...done.
<118> Starting package System Patches...done.
<118> Starting package Backup...done.
<118> Starting package apcupsd...done.
<118> Starting package nut...done.
<118> Starting /usr/local/etc/rc.d/dnsbl.sh...done.
<118>pfSense 2.4.0-BETA amd64 Wed Mar 15 18:17:17 CDT 2017
<118>Bootup complete
<5>ovpnc1: link state changed to DOWN
<5>ovpnc2: link state changed to DOWN
<5>ovpnc2: link state changed to UP
<5>ovpnc1: link state changed to UP
<5>re0: link state changed to DOWN
<5>re1: link state changed to DOWN
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200a, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:401f:15::6, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4009:4c::a, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4009:4c::a, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4009:4c::a, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a01:578:3::36c2:327f, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:809::200a, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:816::200a, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200a, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:401f:31::6, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a01:578:3::3433:8b11, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:815::200a, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80a::200a, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:400c:c0a::5f, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:401f:15::6, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a01:578:3::3410:e6ed, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:809::200a, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:809::200a, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:811::200a, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:401f:15::6, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a01:578:3::34d2:92a8, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80a::200a, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0
<7>cannot forward src fe80:4::c6e9:84ff:fe59:770, dst 2a00:1450:4007:80c::200e, nxt 6, rcvif igb1, outif igb0</cpu></cpu></cpu></cpu></aes-cbc,aes-xts,aes-gcm,aes-icm></kingston></vendor></vendor></vendor></vendor></american></enhanced></enhanced></enhanced></enhanced></thermal></sleep></isa></pci-isa></intel(r)></intel(r)></acpi></acpi></rtl8169s></mii></realtek></acpi></acpi></rtl8169s></mii></realtek></acpi></acpi></encrypt></xhci></ahci></ahci></ahci></vga-compatible></acpi></acpi></at></at></acpi></acpi></acpi></acpi></alaska></software></version></alaska></tscadj,smep,erms,nfpusg></lahf,prefetch></syscall,nx,rdtscp,lm></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe> 

[chrcoluk]

and last bit

kernel trap 12 with interrupts disabled

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 02
fault virtual address	= 0x30
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80cb910c
stack pointer	        = 0x28:0xfffffe0113c6d450
frame pointer	        = 0x28:0xfffffe0113c6d480
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= resume, IOPL = 0
current process		= 54120 (sh)
version.txt06000025413115636557  7630 ustarrootwheelFreeBSD 11.0-RELEASE-p8 #78 3f54ce3814d(RELENG_2_4): Wed Mar 15 18:23:59 CDT 2017
    root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense

[chrcoluk]

another forum with same backtrace actually has led me to this unresolved bug report on FreeBSD,  has been confirmed to affect 11 as well as 12.

So I think this might be what I have

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213903

Have compiled the pfsense kernel now with the patch, will report back here if all good or not, only just booted into it so need to give it some time.