Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Possibly not-pfSense related kernel panic

    Scheduled Pinned Locked Moved 2.4 Development Snapshots
    4 Posts 2 Posters 1.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • ?
      Guest
      last edited by

      Just got a KP on a Xen VM, probably not something pfSense can do anything about, but it's somewhat relevant as it may affect future Xen+pfSense users.

      
      panic: HYPERVISOR_memory_op failed to map gnttab
      cpuid = 2
      KDB: enter: panic
      [ thread pid 26 tid 100093 ]
      Stopped at      kdb_enter+0x3b: movq    $0,kdb_why
      db:0:kdb.enter.default> textdump set
      textdump set
      db:0:kdb.enter.default>  capture on
      db:0:kdb.enter.default>  run lockinfo
      db:1:lockinfo> show locks
      No such command
      db:1:locks>  show alllocks
      No such command
      db:1:alllocks>  show lockedvnods
      Locked vnodes
      
      0xfffff800039da3b0: tag devfs, type VCHR
          usecount 1, writecount 0, refcount 71 mountedhere 0xfffff8000902bc00
          flags (VI_ACTIVE)
          v_object 0xfffff80005d94a50 ref 0 pages 530 cleanbuf 66 dirtybuf 2
          lock type devfs: EXCL by thread 0xfffff80005d9b000 (pid 26, syncer, tid 100093)
      	dev gptid/e6091acf-7471-11e7-a3b1-00163e4568fd
      db:0:kdb.enter.default>  show pcpu
      cpuid        = 2
      dynamic pcpu = 0xfffffe010d13d200
      curthread    = 0xfffff80005d9b000: pid 26 "syncer"
      curpcb       = 0xfffffe0093cacb80
      fpcurthread  = none
      idlethread   = 0xfffff80003507500: tid 100005 "idle: cpu2"
      curpmap      = 0xffffffff829e6300
      tssp         = 0xffffffff82a1ebe0
      commontssp   = 0xffffffff82a1ebe0
      rsp0         = 0xfffffe0093cacb80
      gs32p        = 0xffffffff82a25438
      ldt          = 0xffffffff82a25478
      tss          = 0xffffffff82a25468
      db:0:kdb.enter.default>  bt
      Tracing pid 26 tid 100093 td 0xfffff80005d9b000
      kdb_enter() at kdb_enter+0x3b/frame 0xfffffe0093cac3f0
      vpanic() at vpanic+0x19f/frame 0xfffffe0093cac470
      panic() at panic+0x43/frame 0xfffffe0093cac4d0
      gnttab_map() at gnttab_map+0x131/frame 0xfffffe0093cac520
      get_free_entries() at get_free_entries+0x113/frame 0xfffffe0093cac580
      gnttab_alloc_grant_references() at gnttab_alloc_grant_references+0x12/frame 0xfffffe0093cac5a0
      xbd_startio() at xbd_startio+0x1f3/frame 0xfffffe0093cac5e0
      xbd_strategy() at xbd_strategy+0xc2/frame 0xfffffe0093cac620
      g_disk_start() at g_disk_start+0x34e/frame 0xfffffe0093cac680
      g_io_request() at g_io_request+0x4a7/frame 0xfffffe0093cac6e0
      g_part_start() at g_part_start+0x2a4/frame 0xfffffe0093cac760
      g_io_request() at g_io_request+0x4a7/frame 0xfffffe0093cac7c0
      g_io_request() at g_io_request+0x4a7/frame 0xfffffe0093cac820
      bufwrite() at bufwrite+0x267/frame 0xfffffe0093cac860
      vfs_bio_awrite() at vfs_bio_awrite+0x3ca/frame 0xfffffe0093cac900
      vop_stdfsync() at vop_stdfsync+0x277/frame 0xfffffe0093cac970
      devfs_fsync() at devfs_fsync+0x26/frame 0xfffffe0093cac9a0
      VOP_FSYNC_APV() at VOP_FSYNC_APV+0x8d/frame 0xfffffe0093cac9d0
      sched_sync() at sched_sync+0x3be/frame 0xfffffe0093caca70
      fork_exit() at fork_exit+0x85/frame 0xfffffe0093cacab0
      fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0093cacab0
      --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
      db:0:kdb.enter.default>  ps
        pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
      96866   292   292     0  S       accept   0xfffff800094343cc php-fpm
      95134 94268 95134     0  S+      ttyin    0xfffff80003a3d0a8 sh
      94828 94560 94828     0  S+      ttyin    0xfffff800037c9ca8 sh
      94560 94182 94560     0  S+      wait     0xfffff80064987a50 sh
      94268 92794 94268     0  S+      wait     0xfffff80009139528 sh
      94182     1 94182     0  Ss+     wait     0xfffff80009139a50 login
      94028     1 94028     0  Ss+     ttyin    0xfffff80005d82ca8 getty
      93774     1 93774     0  Ss+     ttyin    0xfffff80005d830a8 getty
      93724     1 93724     0  Ss+     ttyin    0xfffff80005d834a8 getty
      93563     1 93563     0  Ss+     ttyin    0xfffff80005d838a8 getty
      93277     1 93277     0  Ss+     ttyin    0xfffff80005d83ca8 getty
      92995     1 92995     0  Ss+     ttyin    0xfffff80005d840a8 getty
      92927     1 92927     0  Ss+     ttyin    0xfffff80005d844a8 getty
      92794     1 92794     0  Ss+     wait     0xfffff80003a4e000 login
      79583 79293 79293     0  S       nanslp   0xffffffff82866b30 minicron
      79293     1 79293     0  Ss      wait     0xfffff8000967f000 minicron
      79085 78636 78636     0  S       nanslp   0xffffffff82866b32 minicron
      78636     1 78636     0  Ss      wait     0xfffff8005990c000 minicron
      78429 77978 77978     0  S       nanslp   0xffffffff82866b31 minicron
      77978     1 77978     0  Ss      wait     0xfffff80009165528 minicron
      77255 76709 77255     0  Ss      (threaded)                  sshlockout_pf
      100118                   S       piperd   0xfffff800091445f0 sshlockout_pf
      100174                   S       nanslp   0xffffffff82866b33 sshlockout_pf
      76709     1 76709     0  Ss      select   0xfffff80005dcb540 syslogd
      74485 43738    27     0  S+      nanslp   0xffffffff82866b33 sleep
      43738     1    27     0  S+      wait     0xfffff8000967f528 sh
      36608     1 36608   136  Ss      select   0xfffff80009010e40 dhcpd
      30448     1 30448     0  Ss      (threaded)                  ntpd
      100156                   S       select   0xfffff800098da2c0 ntpd
      100162                   S       usem     0xfffff800091d1600 ntpd
      30006     1 30006     0  Ss      nanslp   0xffffffff82866b32 cron
      29449 29124 29124     0  S       kqread   0xfffff80009883400 nginx
      29441 29124 29124     0  S       kqread   0xfffff8000924be00 nginx
      29124     1 29124     0  Ss      pause    0xfffff800096805d0 nginx
      22719     1 22719    59  Ss      (threaded)                  unbound
      100126                   S       kqread   0xfffff80059739c00 unbound
      100151                   S       kqread   0xfffff80009883800 unbound
      100152                   S       kqread   0xfffff800597e3700 unbound
      100153                   S       kqread   0xfffff800597e3400 unbound
      15927     1 15927     0  Ss      (threaded)                  dpinger
      100145                   S       uwait    0xfffff8000535cc00 dpinger
      100146                   S       sbwait   0xfffff80009888804 dpinger
      100147                   S       nanslp   0xffffffff82866b30 dpinger
      100148                   S       nanslp   0xffffffff82866b30 dpinger
      100149                   S       accept   0xfffff800098883cc dpinger
      13491     1 13491     0  Ss      bpf      0xfffff80009003c00 filterlog
      12932     1 12932     0  Ss      select   0xfffff80005dca540 openvpn
      11268     1 11268     0  Ss      select   0xfffff80005df0a40 openvpn
       7987     1  7987     0  Ss      (threaded)                  sshlockout_pf
      100119                   S       uwait    0xfffff80005df2400 sshlockout_pf
      100141                   S       nanslp   0xffffffff82866b32 sshlockout_pf
       7885     1  7885     0  Ss      select   0xfffff80005def0c0 sshd
        320     1   320     0  Ss      select   0xfffff80005df1240 devd
        307   306   306     0  S       kqread   0xfffff8000901f200 check_reload_status
        306     1   306     0  Ss      kqread   0xfffff80009229500 check_reload_status
        292     1   292     0  Ss      kqread   0xfffff8000901f900 php-fpm
         57     0     0     0  DL      mdwait   0xfffff80003a45000 [md0]
         26     0     0     0  RL      CPU 2                       [syncer]
         25     0     0     0  DL      vlruwt   0xfffff80003a4ea50 [vnlru]
         24     0     0     0  DL      (threaded)                  [bufdaemon]
      100091                   D       psleep   0xffffffff8295ef04 [bufdaemon]
      100107                   D       sdflush  0xfffff80005d800e8 [/ worker]
         23     0     0     0  DL      -        0xffffffff8295fbb4 [bufspacedaemon]
         22     0     0     0  DL      pgzero   0xffffffff829742e4 [pagezero]
         21     0     0     0  DL      psleep   0xffffffff8297078c [vmdaemon]
         20     0     0     0  DL      (threaded)                  [pagedaemon]
      100087                   D       psleep   0xffffffff82a1df05 [pagedaemon]
      100094                   D       umarcl   0xffffffff829700b8 [uma]
         19     0     0     0  DL      -        0xffffffff8295e714 [soaiod4]
         18     0     0     0  DL      -        0xffffffff8295e714 [soaiod3]
         17     0     0     0  DL      -        0xffffffff8295e714 [soaiod2]
          9     0     0     0  DL      -        0xffffffff8295e714 [soaiod1]
          8     0     0     0  DL      balloon  0xffffffff80a9cef0 [balloon]
          7     0     0     0  DL      -        0xffffffff8273b2f0 [rand_harvestq]
          6     0     0     0  DL      pftm     0xffffffff80f4c4a0 [pf purge]
          5     0     0     0  DL      waiting_ 0xffffffff82a0d540 [sctp_iterator]
         16     0     0     0  SL      xbread   0xfffff800feffc000 [xenstore_rcv]
         15     0     0     0  SL      waitev   0xffffffff82858dc0 [xenwatch]
          4     0     0     0  DL      (threaded)                  [cam]
      100032                   D       -        0xffffffff8260e180 [doneq0]
      100048                   D       -        0xffffffff8260dfc8 [scanner]
          3     0     0     0  DL      crypto_r 0xffffffff8296ec70 [crypto returns]
          2     0     0     0  DL      crypto_w 0xffffffff8296eb18 [crypto]
         14     0     0     0  DL      (threaded)                  [geom]
      100023                   D       -        0xffffffff829e5740 [g_event]
      100024                   D       -        0xffffffff829e5748 [g_up]
      100025                   D       -        0xffffffff829e5750 [g_down]
         13     0     0     0  DL      (threaded)                  [ng_queue]
      100019                   D       sleep    0xffffffff825cbfd0 [ng_queue0]
      100020                   D       sleep    0xffffffff825cbfd0 [ng_queue1]
      100021                   D       sleep    0xffffffff825cbfd0 [ng_queue2]
      100022                   D       sleep    0xffffffff825cbfd0 [ng_queue3]
         12     0     0     0  WL      (threaded)                  [intr]
      100007                   I                                   [swi3: vm]
      100008                   I                                   [swi4: clock (0)]
      100009                   I                                   [swi4: clock (1)]
      100010                   I                                   [swi4: clock (2)]
      100011                   I                                   [swi4: clock (3)]
      100012                   I                                   [swi1: netisr 0]
      100013                   I                                   [swi6: task queue]
      100014                   I                                   [swi6: Giant taskq]
      100017                   I                                   [swi5: fast taskq]
      100033                   I                                   [irq14: ata0]
      100034                   I                                   [irq15: ata1]
      100035                   I                                   [irq1: atkbd0]
      100036                   I                                   [irq12: psm0]
      100037                   I                                   [swi0: uart]
      100038                   I                                   [irq772: xenstore0]
      100044                   I                                   [swi1: pf send]
      100045                   I                                   [swi1: pfsync]
      100054                   I                                   [irq777: xbd0]
      100056                   I                                   [irq778: xn0]
      100058                   I                                   [irq779: xn0]
      100060                   I                                   [irq780: xn0]
      100062                   I                                   [irq781: xn0]
      100064                   I                                   [irq782: xn1]
      100066                   I                                   [irq783: xn1]
      100068                   I                                   [irq784: xn1]
      100070                   I                                   [irq785: xn1]
      100072                   I                                   [irq786: xn2]
      100074                   I                                   [irq787: xn2]
      100076                   I                                   [irq788: xn2]
      100078                   I                                   [irq789: xn2]
      100080                   I                                   [irq790: xn3]
      100082                   I                                   [irq791: xn3]
      100084                   I                                   [irq792: xn3]
      100086                   I                                   [irq793: xn3]
      100095                   I                                   [swi1: netisr 1]
      100096                   I                                   [swi1: netisr 2]
      100097                   I                                   [swi1: netisr 3]
         11     0     0     0  RL      (threaded)                  [idle]
      100003                   Run     CPU 0                       [idle: cpu0]
      100004                   Run     CPU 1                       [idle: cpu1]
      100005                   CanRun                              [idle: cpu2]
      100006                   Run     CPU 3                       [idle: cpu3]
          1     0     1     0  SLs     wait     0xfffff80003505528 [init]
         10     0     0     0  DL      audit_wo 0xffffffff82a146e0 [audit]
          0     0     0     0  DLs     (threaded)                  [kernel]
      100000                   D       swapin   0xffffffff829e5778 [swapper]
      100015                   D       -        0xfffff8000351a100 [aiod_kick taskq]
      100016                   D       -        0xfffff80003519e00 [thread]
      100018                   D       -        0xfffff80003519900 [kqueue_ctx taskq]
      100026                   D       -        0xfffff80003518a00 [firmware taskq]
      100029                   D       -        0xfffff800034f8300 [acpi_task_0]
      100030                   D       -        0xfffff800034f8300 [acpi_task_1]
      100031                   D       -        0xfffff800034f8300 [acpi_task_2]
      100041                   D       -        0xfffff800037c2c00 [mca taskq]
      100047                   D       -        0xfffff800036e6100 [CAM taskq]
      100071                   D       -        0xfffff80003eea000 [xn2 txq 0]
      100073                   D       -        0xfffff80003ee9b00 [xn2 txq 1]
      100075                   D       -        0xfffff80003ee9700 [xn2 txq 2]
      100077                   D       -        0xfffff80003ee9300 [xn2 txq 3]
      100079                   D       -        0xfffff80003ee8e00 [xn3 txq 0]
      100081                   D       -        0xfffff80003ee8a00 [xn3 txq 1]
      100083                   D       -        0xfffff80003ee8600 [xn3 txq 2]
      100085                   D       -        0xfffff80003ee8200 [xn3 txq 3]
      100098                   D       -        0xfffff80005da1900 [if_config_tqg_0]
      100099                   D       -        0xfffff80005da1700 [if_io_tqg_0]
      100100                   D       -        0xfffff80005da1500 [if_io_tqg_1]
      100101                   D       -        0xfffff80005da1300 [if_io_tqg_2]
      100102                   D       -        0xfffff80005da1100 [if_io_tqg_3]
      100132                   D       -        0xfffff80003a54100 [xn0 txq 0]
      100133                   D       -        0xfffff80003a54800 [xn0 txq 1]
      100134                   D       -        0xfffff80003a54000 [xn0 txq 2]
      100135                   D       -        0xfffff80003a54900 [xn0 txq 3]
      100136                   D       -        0xfffff80003eea400 [xn1 txq 0]
      100137                   D       -        0xfffff800039e1700 [xn1 txq 1]
      100138                   D       -        0xfffff80003eea300 [xn1 txq 2]
      100139                   D       -        0xfffff800039e1800 [xn1 txq 3]
      db:0:kdb.enter.default>  alltrace
      
      Tracing command php-fpm pid 96866 tid 100169 td 0xfffff800090ef000
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093e04710
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093e04740
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093e047c0
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093e047f0
      _sleep() at _sleep+0x294/frame 0xfffffe0093e04880
      kern_accept4() at kern_accept4+0x1f2/frame 0xfffffe0093e04930
      accept1() at accept1+0x60/frame 0xfffffe0093e04990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093e04ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093e04ab0
      --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x801d939ba, rsp = 0x7ffffffec848, rbp = 0x7ffffffec880 ---
      
      Tracing command sh pid 95134 tid 100173 td 0xfffff800649d8000
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093e18660
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093e18690
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093e18710
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093e18740
      _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe0093e18790
      tty_wait() at tty_wait+0x1c/frame 0xfffffe0093e187b0
      ttydisc_read() at ttydisc_read+0x244/frame 0xfffffe0093e18810
      ttydev_read() at ttydev_read+0x34/frame 0xfffffe0093e18840
      devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe0093e188a0
      dofileread() at dofileread+0x98/frame 0xfffffe0093e188f0
      kern_readv() at kern_readv+0x68/frame 0xfffffe0093e18940
      sys_read() at sys_read+0x84/frame 0xfffffe0093e18990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093e18ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093e18ab0
      --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800bc61ea, rsp = 0x7fffffffe768, rbp = 0x7fffffffe870 ---
      
      Tracing command sh pid 94828 tid 100177 td 0xfffff80009981500
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093e2c660
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093e2c690
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093e2c710
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093e2c740
      _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe0093e2c790
      tty_wait() at tty_wait+0x1c/frame 0xfffffe0093e2c7b0
      ttydisc_read() at ttydisc_read+0x244/frame 0xfffffe0093e2c810
      ttydev_read() at ttydev_read+0x34/frame 0xfffffe0093e2c840
      devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe0093e2c8a0
      dofileread() at dofileread+0x98/frame 0xfffffe0093e2c8f0
      kern_readv() at kern_readv+0x68/frame 0xfffffe0093e2c940
      sys_read() at sys_read+0x84/frame 0xfffffe0093e2c990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093e2cab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093e2cab0
      --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800bc61ea, rsp = 0x7fffffffe768, rbp = 0x7fffffffe870 ---
      
      Tracing command sh pid 94560 tid 100176 td 0xfffff80009981a00
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093e27560
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093e27590
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093e27610
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093e27640
      _sleep() at _sleep+0x294/frame 0xfffffe0093e276d0
      kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe0093e277a0
      sys_wait4() at sys_wait4+0x73/frame 0xfffffe0093e27990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093e27ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093e27ab0
      --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800b5d75a, rsp = 0x7fffffffe818, rbp = 0x7fffffffe8d0 ---
      
      Tracing command sh pid 94268 tid 100112 td 0xfffff8000916f500
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093d0b560
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093d0b590
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093d0b610
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093d0b640
      _sleep() at _sleep+0x294/frame 0xfffffe0093d0b6d0
      kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe0093d0b7a0
      sys_wait4() at sys_wait4+0x73/frame 0xfffffe0093d0b990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093d0bab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093d0bab0
      --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800b5d75a, rsp = 0x7fffffffe818, rbp = 0x7fffffffe8d0 ---
      
      Tracing command login pid 94182 tid 100106 td 0xfffff8000916fa00
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093ced560
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093ced590
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093ced610
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093ced640
      _sleep() at _sleep+0x294/frame 0xfffffe0093ced6d0
      kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe0093ced7a0
      sys_wait4() at sys_wait4+0x73/frame 0xfffffe0093ced990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093cedab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093cedab0
      --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d2475a, rsp = 0x7fffffffebe8, rbp = 0x7fffffffed40 ---
      
      Tracing command getty pid 94028 tid 100108 td 0xfffff800090f8500
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093cf7660
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093cf7690
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093cf7710
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093cf7740
      _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe0093cf7790
      tty_wait() at tty_wait+0x1c/frame 0xfffffe0093cf77b0
      ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe0093cf7810
      ttydev_read() at ttydev_read+0x34/frame 0xfffffe0093cf7840
      devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe0093cf78a0
      dofileread() at dofileread+0x98/frame 0xfffffe0093cf78f0
      kern_readv() at kern_readv+0x68/frame 0xfffffe0093cf7940
      sys_read() at sys_read+0x84/frame 0xfffffe0093cf7990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093cf7ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093cf7ab0
      --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---
      
      Tracing command getty pid 93774 tid 100130 td 0xfffff80009365000
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093d65660
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093d65690
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093d65710
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093d65740
      _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe0093d65790
      tty_wait() at tty_wait+0x1c/frame 0xfffffe0093d657b0
      ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe0093d65810
      ttydev_read() at ttydev_read+0x34/frame 0xfffffe0093d65840
      devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe0093d658a0
      dofileread() at dofileread+0x98/frame 0xfffffe0093d658f0
      kern_readv() at kern_readv+0x68/frame 0xfffffe0093d65940
      sys_read() at sys_read+0x84/frame 0xfffffe0093d65990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093d65ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093d65ab0
      --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---
      
      Tracing command getty pid 93724 tid 100175 td 0xfffff800098dd000
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093e22660
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093e22690
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093e22710
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093e22740
      _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe0093e22790
      tty_wait() at tty_wait+0x1c/frame 0xfffffe0093e227b0
      ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe0093e22810
      ttydev_read() at ttydev_read+0x34/frame 0xfffffe0093e22840
      devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe0093e228a0
      dofileread() at dofileread+0x98/frame 0xfffffe0093e228f0
      kern_readv() at kern_readv+0x68/frame 0xfffffe0093e22940
      sys_read() at sys_read+0x84/frame 0xfffffe0093e22990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093e22ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093e22ab0
      --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---
      
      Tracing command getty pid 93563 tid 100113 td 0xfffff80005df6000
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093d10660
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093d10690
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093d10710
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093d10740
      _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe0093d10790
      tty_wait() at tty_wait+0x1c/frame 0xfffffe0093d107b0
      ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe0093d10810
      ttydev_read() at ttydev_read+0x34/frame 0xfffffe0093d10840
      devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe0093d108a0
      dofileread() at dofileread+0x98/frame 0xfffffe0093d108f0
      kern_readv() at kern_readv+0x68/frame 0xfffffe0093d10940
      sys_read() at sys_read+0x84/frame 0xfffffe0093d10990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093d10ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093d10ab0
      --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---
      
      Tracing command getty pid 93277 tid 100125 td 0xfffff8000935ca00
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093d4c660
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093d4c690
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093d4c710
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093d4c740
      _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe0093d4c790
      tty_wait() at tty_wait+0x1c/frame 0xfffffe0093d4c7b0
      ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe0093d4c810
      ttydev_read() at ttydev_read+0x34/frame 0xfffffe0093d4c840
      devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe0093d4c8a0
      dofileread() at dofileread+0x98/frame 0xfffffe0093d4c8f0
      kern_readv() at kern_readv+0x68/frame 0xfffffe0093d4c940
      sys_read() at sys_read+0x84/frame 0xfffffe0093d4c990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093d4cab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093d4cab0
      --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---
      
      Tracing command getty pid 92995 tid 100122 td 0xfffff80005db8a00
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093d3d660
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093d3d690
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093d3d710
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093d3d740
      _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe0093d3d790
      tty_wait() at tty_wait+0x1c/frame 0xfffffe0093d3d7b0
      ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe0093d3d810
      ttydev_read() at ttydev_read+0x34/frame 0xfffffe0093d3d840
      devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe0093d3d8a0
      dofileread() at dofileread+0x98/frame 0xfffffe0093d3d8f0
      kern_readv() at kern_readv+0x68/frame 0xfffffe0093d3d940
      sys_read() at sys_read+0x84/frame 0xfffffe0093d3d990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093d3dab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093d3dab0
      --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---
      
      Tracing command getty pid 92927 tid 100105 td 0xfffff80005dce500
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093ce8660
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093ce8690
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093ce8710
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093ce8740
      _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe0093ce8790
      tty_wait() at tty_wait+0x1c/frame 0xfffffe0093ce87b0
      ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe0093ce8810
      ttydev_read() at ttydev_read+0x34/frame 0xfffffe0093ce8840
      devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe0093ce88a0
      dofileread() at dofileread+0x98/frame 0xfffffe0093ce88f0
      kern_readv() at kern_readv+0x68/frame 0xfffffe0093ce8940
      sys_read() at sys_read+0x84/frame 0xfffffe0093ce8990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093ce8ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093ce8ab0
      --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---
      
      Tracing command login pid 92794 tid 100103 td 0xfffff80005df7000
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093cde560
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093cde590
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093cde610
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093cde640
      _sleep() at _sleep+0x294/frame 0xfffffe0093cde6d0
      kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe0093cde7a0
      sys_wait4() at sys_wait4+0x73/frame 0xfffffe0093cde990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093cdeab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093cdeab0
      --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d2475a, rsp = 0x7fffffffebe8, rbp = 0x7fffffffed40 ---
      
      Tracing command minicron pid 79583 tid 100129 td 0xfffff80009365500
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093d60780
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093d607b0
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093d60830
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0093d60860
      _sleep() at _sleep+0x26f/frame 0xfffffe0093d608f0
      kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe0093d60950
      sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe0093d60990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093d60ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093d60ab0
      --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8009008fa, rsp = 0x7fffffffeb88, rbp = 0x7fffffffebf0 ---
      
      Tracing command minicron pid 79293 tid 100163 td 0xfffff800598d4a00
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093de2560
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093de2590
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093de2610
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093de2640
      _sleep() at _sleep+0x294/frame 0xfffffe0093de26d0
      kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe0093de27a0
      sys_wait4() at sys_wait4+0x73/frame 0xfffffe0093de2990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093de2ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093de2ab0
      --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80090075a, rsp = 0x7fffffffebf8, rbp = 0x7fffffffec30 ---
      
      Tracing command minicron pid 79085 tid 100172 td 0xfffff800649d8500
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093e13780
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093e137b0
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093e13830
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0093e13860
      _sleep() at _sleep+0x26f/frame 0xfffffe0093e138f0
      kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe0093e13950
      sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe0093e13990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093e13ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093e13ab0
      --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8009008fa, rsp = 0x7fffffffeb98, rbp = 0x7fffffffec00 ---
      
      Tracing command minicron pid 78636 tid 100164 td 0xfffff80059813a00
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093de7560
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093de7590
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093de7610
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093de7640
      _sleep() at _sleep+0x294/frame 0xfffffe0093de76d0
      kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe0093de77a0
      sys_wait4() at sys_wait4+0x73/frame 0xfffffe0093de7990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093de7ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093de7ab0
      --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80090075a, rsp = 0x7fffffffec08, rbp = 0x7fffffffec40 ---
      
      Tracing command minicron pid 78429 tid 100117 td 0xfffff80005dcda00
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093d24780
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093d247b0
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093d24830
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0093d24860
      _sleep() at _sleep+0x26f/frame 0xfffffe0093d248f0
      kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe0093d24950
      sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe0093d24990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093d24ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093d24ab0
      --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8009008fa, rsp = 0x7fffffffebb8, rbp = 0x7fffffffec20 ---
      
      Tracing command minicron pid 77978 tid 100160 td 0xfffff800098dd500
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093dd3560
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093dd3590
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093dd3610
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093dd3640
      _sleep() at _sleep+0x294/frame 0xfffffe0093dd36d0
      kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe0093dd37a0
      sys_wait4() at sys_wait4+0x73/frame 0xfffffe0093dd3990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093dd3ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093dd3ab0
      --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80090075a, rsp = 0x7fffffffec28, rbp = 0x7fffffffec60 ---
      
      Tracing command sshlockout_pf pid 77255 tid 100118 td 0xfffff80005df4a00
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093d296d0
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093d29700
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093d29780
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093d297b0
      _sleep() at _sleep+0x294/frame 0xfffffe0093d29840
      pipe_read() at pipe_read+0x36d/frame 0xfffffe0093d298a0
      dofileread() at dofileread+0x98/frame 0xfffffe0093d298f0
      kern_readv() at kern_readv+0x68/frame 0xfffffe0093d29940
      sys_read() at sys_read+0x84/frame 0xfffffe0093d29990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093d29ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093d29ab0
      --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b911ea, rsp = 0x7fffffffe7d8, rbp = 0x7fffffffe7f0 ---
      
      Tracing command sshlockout_pf pid 77255 tid 100174 td 0xfffff800598d4500
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093e1d780
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093e1d7b0
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093e1d830
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0093e1d860
      _sleep() at _sleep+0x26f/frame 0xfffffe0093e1d8f0
      kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe0093e1d950
      sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe0093e1d990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093e1dab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093e1dab0
      --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b288fa, rsp = 0x7fffdfffdf48, rbp = 0x7fffdfffdf70 ---
      
      Tracing command syslogd pid 76709 tid 100165 td 0xfffff80059813500
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093dec5e0
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093dec610
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093dec690
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093dec6c0
      _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe0093dec710
      seltdwait() at seltdwait+0x10f/frame 0xfffffe0093dec760
      kern_select() at kern_select+0x979/frame 0xfffffe0093dec950
      sys_select() at sys_select+0x54/frame 0xfffffe0093dec990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093decab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093decab0
      --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800b851ca, rsp = 0x7fffffffdd78, rbp = 0x7fffffffec20 ---
      
      Tracing command sleep pid 74485 tid 100168 td 0xfffff800090ef500
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093dfb780
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093dfb7b0
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093dfb830
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0093dfb860
      _sleep() at _sleep+0x26f/frame 0xfffffe0093dfb8f0
      kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe0093dfb950
      sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe0093dfb990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093dfbab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093dfbab0
      --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8009008fa, rsp = 0x7fffffffec58, rbp = 0x7fffffffeca0 ---
      
      Tracing command sh pid 43738 tid 100161 td 0xfffff800598ca000
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093dd8560
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093dd8590
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093dd8610
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093dd8640
      _sleep() at _sleep+0x294/frame 0xfffffe0093dd86d0
      kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe0093dd87a0
      sys_wait4() at sys_wait4+0x73/frame 0xfffffe0093dd8990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093dd8ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093dd8ab0
      --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800b5d75a, rsp = 0x7fffffffe858, rbp = 0x7fffffffe910 ---
      
      Tracing command dhcpd pid 36608 tid 100159 td 0xfffff800598ca500
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093dce5c0
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093dce5f0
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093dce670
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0093dce6a0
      _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe0093dce710
      seltdwait() at seltdwait+0xc7/frame 0xfffffe0093dce760
      kern_select() at kern_select+0x979/frame 0xfffffe0093dce950
      sys_select() at sys_select+0x54/frame 0xfffffe0093dce990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093dceab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093dceab0
      --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800d641ca, rsp = 0x7fffffffea28, rbp = 0x7fffffffea50 ---
      
      Tracing command ntpd pid 30448 tid 100156 td 0xfffff800597a0a00
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093dbf5e0
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093dbf610
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093dbf690
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093dbf6c0
      _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe0093dbf710
      seltdwait() at seltdwait+0x10f/frame 0xfffffe0093dbf760
      kern_select() at kern_select+0x979/frame 0xfffffe0093dbf950
      sys_select() at sys_select+0x54/frame 0xfffffe0093dbf990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093dbfab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093dbfab0
      --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8017e51ca, rsp = 0x7fffffffdc08, rbp = 0x7fffffffdc40 ---
      
      Tracing command ntpd pid 30448 tid 100162 td 0xfffff800597a0000
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093ddd700
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093ddd730
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093ddd7b0
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093ddd7e0
      _sleep() at _sleep+0x294/frame 0xfffffe0093ddd870
      umtxq_sleep() at umtxq_sleep+0x122/frame 0xfffffe0093ddd8d0
      do_sem2_wait() at do_sem2_wait+0x6da/frame 0xfffffe0093ddd950
      __umtx_op_sem2_wait() at __umtx_op_sem2_wait+0x6d/frame 0xfffffe0093ddd990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093dddab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093dddab0
      --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x801716a9a, rsp = 0x7fffdfffdee8, rbp = 0x7fffdfffdf40 ---
      
      Tracing command cron pid 30006 tid 100140 td 0xfffff80009364500
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093d6f780
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093d6f7b0
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093d6f830
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0093d6f860
      _sleep() at _sleep+0x26f/frame 0xfffffe0093d6f8f0
      kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe0093d6f950
      sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe0093d6f990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093d6fab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093d6fab0
      --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800d288fa, rsp = 0x7fffffffea88, rbp = 0x7fffffffecb0 ---
      
      Tracing command nginx pid 29449 tid 100155 td 0xfffff800597a1000
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093dba520
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093dba550
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093dba5d0
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0093dba600
      _sleep() at _sleep+0x26f/frame 0xfffffe0093dba690
      kqueue_kevent() at kqueue_kevent+0x336/frame 0xfffffe0093dba870
      kern_kevent() at kern_kevent+0xb5/frame 0xfffffe0093dba8e0
      sys_kevent() at sys_kevent+0x11c/frame 0xfffffe0093dba990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093dbaab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093dbaab0
      --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x801b6293a, rsp = 0x7fffffffe718, rbp = 0x7fffffffe760 ---
      
      Tracing command nginx pid 29441 tid 100143 td 0xfffff8000916e000
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093d7e520
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093d7e550
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093d7e5d0
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093d7e600
      _sleep() at _sleep+0x294/frame 0xfffffe0093d7e690
      kqueue_kevent() at kqueue_kevent+0x336/frame 0xfffffe0093d7e870
      kern_kevent() at kern_kevent+0xb5/frame 0xfffffe0093d7e8e0
      sys_kevent() at sys_kevent+0x11c/frame 0xfffffe0093d7e990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093d7eab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093d7eab0
      --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x801b6293a, rsp = 0x7fffffffe718, rbp = 0x7fffffffe760 ---
      
      Tracing command nginx pid 29124 tid 100154 td 0xfffff80009364000
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093db5780
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093db57b0
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093db5830
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093db5860
      _sleep() at _sleep+0x294/frame 0xfffffe0093db58f0
      kern_sigsuspend() at kern_sigsuspend+0x134/frame 0xfffffe0093db5960
      sys_sigsuspend() at sys_sigsuspend+0x31/frame 0xfffffe0093db5990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093db5ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093db5ab0
      --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x801b627da, rsp = 0x7fffffffe8b8, rbp = 0x7fffffffe8f0 ---
      
      Tracing command unbound pid 22719 tid 100126 td 0xfffff8000935c500
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093d51520
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093d51550
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093d515d0
      sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0093d51600
      _sleep() at _sleep+0x26f/frame 0xfffffe0093d51690
      kqueue_kevent() at kqueue_kevent+0x336/frame 0xfffffe0093d51870
      kern_kevent() at kern_kevent+0xb5/frame 0xfffffe0093d518e0
      sys_kevent() at sys_kevent+0x11c/frame 0xfffffe0093d51990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093d51ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093d51ab0
      --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x801b7293a, rsp = 0x7fffffffea08, rbp = 0x7fffffffea50 ---
      
      Tracing command unbound pid 22719 tid 100151 td 0xfffff8000913a000
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093da6520
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093da6550
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093da65d0
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093da6600
      _sleep() at _sleep+0x294/frame 0xfffffe0093da6690
      kqueue_kevent() at kqueue_kevent+0x336/frame 0xfffffe0093da6870
      kern_kevent() at kern_kevent+0xb5/frame 0xfffffe0093da68e0
      sys_kevent() at sys_kevent+0x11c/frame 0xfffffe0093da6990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093da6ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093da6ab0
      --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x801b7293a, rsp = 0x7fffdfffde18, rbp = 0x7fffdfffde60 ---
      
      Tracing command unbound pid 22719 tid 100152 td 0xfffff800597a1a00
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093dab520
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093dab550
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093dab5d0
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093dab600
      _sleep() at _sleep+0x294/frame 0xfffffe0093dab690
      kqueue_kevent() at kqueue_kevent+0x336/frame 0xfffffe0093dab870
      kern_kevent() at kern_kevent+0xb5/frame 0xfffffe0093dab8e0
      sys_kevent() at sys_kevent+0x11c/frame 0xfffffe0093dab990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093dabab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093dabab0
      --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x801b7293a, rsp = 0x7fffdfdfce18, rbp = 0x7fffdfdfce60 ---
      
      Tracing command unbound pid 22719 tid 100153 td 0xfffff800597a1500
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093db0520
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093db0550
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093db05d0
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093db0600
      _sleep() at _sleep+0x294/frame 0xfffffe0093db0690
      kqueue_kevent() at kqueue_kevent+0x336/frame 0xfffffe0093db0870
      kern_kevent() at kern_kevent+0xb5/frame 0xfffffe0093db08e0
      sys_kevent() at sys_kevent+0x11c/frame 0xfffffe0093db0990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093db0ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093db0ab0
      --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x801b7293a, rsp = 0x7fffdfbfbe18, rbp = 0x7fffdfbfbe60 ---
      
      Tracing command dpinger pid 15927 tid 100145 td 0xfffff8000913aa00
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093d886f0
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093d88720
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093d887a0
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093d887d0
      _sleep() at _sleep+0x294/frame 0xfffffe0093d88860
      umtxq_sleep() at umtxq_sleep+0x122/frame 0xfffffe0093d888c0
      do_wait() at do_wait+0x419/frame 0xfffffe0093d88950
      __umtx_op_wait() at __umtx_op_wait+0x77/frame 0xfffffe0093d88990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093d88ab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093d88ab0
      --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083c67c, rsp = 0x7fffffffe8f8, rbp = 0x7fffffffe980 ---
      
      Tracing command dpinger pid 15927 tid 100146 td 0xfffff8000997ea00
      sched_switch() at sched_switch+0x6cb/frame 0xfffffe0093d8d5f0
      mi_switch() at mi_switch+0xd2/frame 0xfffffe0093d8d620
      sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe0093d8d6a0
      sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0093d8d6d0
      _sleep() at _sleep+0x294/frame 0xfffffe0093d8d760
      sbwait() at sbwait+0x4f/frame 0xfffffe0093d8d780
      soreceive_generic() at soreceive_generic+0x1ae/frame 0xfffffe0093d8d850
      kern_recvit() at kern_recvit+0x1d6/frame 0xfffffe0093d8d920
      sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe0093d8d990
      amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe0093d8dab0
      Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0093d8dab0
      --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b3cf5a, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 ---
      
      (truncated because the forum doesn't like it, ends with:)
      
      [code]
      --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
      db:0:kdb.enter.default>  capture off
      db:0:kdb.enter.default>  textdump dump
      Textdump complete.
      db:0:kdb.enter.default>  reset
      cpu_reset: Restarting BSP
      cpu_reset_proxy: Stopped CPU 2
      [/code][/thread]
      
      1 Reply Last reply Reply Quote 0
      • jimpJ
        jimp Rebel Alliance Developer Netgate
        last edited by

        Looks like a filesystem panic. Could be anything from a corrupted fs that needs fsck run to an actual hardware issue in the hypervisor, or anything in between.

        Those are hard to nail down in a virtualized environment.

        Safest bet is to wipe+reload+restore config

        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

        Need help fast? Netgate Global Support!

        Do not Chat/PM for help!

        1 Reply Last reply Reply Quote 0
        • ?
          Guest
          last edited by

          @jimp:

          Looks like a filesystem panic. Could be anything from a corrupted fs that needs fsck run to an actual hardware issue in the hypervisor, or anything in between.

          Those are hard to nail down in a virtualized environment.

          Safest bet is to wipe+reload+restore config

          Yeah, did that. The report should have been sent to your crash report recording service. Some info:

          
          					Crash report begins.  Anonymous machine information:
          
          amd64
          11.0-RELEASE-p11
          FreeBSD 11.0-RELEASE-p11 #191 de7122ed242(RELENG_2_4): Fri Jul 28 12:05:08 CDT 2017     root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense
          
          Crash report details:
          
          No PHP errors found.
          
          Filename: /var/crash/bounds
          1
          
          Filename: /var/crash/info.0
          Dump header from device: /dev/gptid/e609f1ad-7471-11e7-a3b1-00163e4568fd
            Architecture: amd64
            Architecture Version: 1
            Dump Length: 69632
            Blocksize: 512
            Dumptime: Tue Aug  1 13:14:27 2017
            Hostname: firewall-1-prod.adivare.net
            Magic: FreeBSD Text Dump
            Version String: FreeBSD 11.0-RELEASE-p11 #191 de7122ed242(RELENG_2_4): Fri Jul 28 12:05:08 CDT 2017
              root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense
            Panic String: HYPERVISOR_memory_op failed to map gnttab
            Dump Parity: 2668466774
            Bounds: 0
            Dump Status: good
          
          

          So it's 2.4.0-BETA (amd64) built on Fri Jul 28 11:57:59 CDT 2017, running on Xen PVHVM 4.8.1, backed by a LVM disk which is stored on a mirrored SSD (MD Array). I suspect it must have been somewhere in the virtual block device layer, either the backing end or the communication between the backing disk and the block frontend in BSD, but most likely not a BSD fault anyway. Recording it here might help someone in the future. It runs fine now, and I couldn't reproduce this.

          1 Reply Last reply Reply Quote 0
          • ?
            Guest
            last edited by

            More information; 2.4 / FreeBSD 11.0 reliably crashes when more than 3 virtual network interfaces are added.

            Configuration used:

            
            name = 'firewall'
            bios = "ovmf"
            uuid="11d1366c-40e2-43b2-83be-52cfe0c6542d"
            builder = 'hvm'
            memory = '2024'
            vcpus = 4
            disk = ['file:/dev/hypervisor-local/firewall-system,xvda,w']
            vif = [
            'mac=00:16:3e:e5:68:fd, bridge=wan0, script=vif-openvswitch-no-offload, type=vif',
            'mac=00:16:3e:e8:20:db, bridge=lan10, script=vif-openvswitch-no-offload, type=vif',
            #'mac=00:16:3e:ee:02:44, bridge=lan20, script=vif-openvswitch-no-offload, type=vif',
            #'mac=00:16:3e:e1:03:f1, bridge=lan30, script=vif-openvswitch-no-offload, type=vif',
            'mac=00:16:3e:e2:30:44, bridge=lan40, script=vif-openvswitch-no-offload, type=vif'
            ]
            boot = 'c'
            serial='pty'
            
            

            The VIF lines define network interfaces, when using the 3 as described in this configuration pfSense works, when adding one or more beyond that, it always crashes. With 4 it generates a panic and dump, with 5 it just panics and hangs in kdb with db>.

            1 Reply Last reply Reply Quote 0
            • First post
              Last post
            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.