Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    SG-4860 "Configuring firewall…." on boot error

    Scheduled Pinned Locked Moved Problems Installing or Upgrading pfSense Software
    7 Posts 2 Posters 1.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      dbennett
      last edited by

      Just rebuilt my pfsense SG-4860, had to reload the OS which went smoothly, manually modified/updated the interface types (em# -> igb#), and rebooted it at least 4 times.

      I went to deploy it this morning and some of the servers on vLAN2 and 3 couldn't be reached.  Logged into the firewall GUI and noticed that the vlan assignments were incorrect.  I updated the interface assignments again, saved, and pretty much saw my entire network go down.  When i tried to access the console via the serial connection it connected but would load anything.  Failed back over to the old equipment then power cycled the SG.  This time the console came alive as it booted up.  That's when this error showed up and the boot up stalled.

      SeaBIOS (version SageBIOS-20150314_140038-localhost.localdomain)
      Found mainboard Intel Mohon Peak
      Relocating init from 0x000ef1b0 to 0x7fb995f0 (size 26960)
      Found CBFS header at 0xfffffbe0
      CPU Mhz=2401
      Found 20 PCI devices (max PCI bus is 04)
      Copying MPTABLE from 0x7fbe3e00/7fbe3e10 to 0x000f5950 with length 1ac
      Copying ACPI RSDP from 0x7fbe4e00 to 0x000f5920
      Copying SMBIOS entry point from 0x7fbf0400 to 0x000f5900
      Using pmtimer, ioport 0x408
      Scan for VGA option rom
      EHCI init on dev 00:16.0 (regs=0xdff35420)
      Found 2 serial ports
      ebda moved from 9f000 to 9e800
      AHCI controller at 17.0, iobase dff34000, irq 15
      AHCI controller at 18.0, iobase dff34800, irq 10
      Searching bootorder for: /pci@i0cf8/usb@16/hub@1/storage@4/*@0/*@0,0
      Searching bootorder for: /pci@i0cf8/usb@16/hub@1/usb-*@4
      USB MSC vendor='Generic' product='Ultra HS-COMBO' rev='1.98' type=0 removable=1
      USB MSC blksize=512 sectors=7733248
      Initialized USB HUB (1 ports used)
      All threads complete.
      Scan for option roms
      Running option rom at c000:0003
      pmm call arg1=1
      pmm call arg1=0
      pmm call arg1=1
      pmm call arg1=0
      Running option rom at c100:0003
      pmm call arg1=1
      pmm call arg1=1
      Running option rom at c200:0003
      pmm call arg1=1
      pmm call arg1=1
      Running option rom at c300:0003
      pmm call arg1=1
      pmm call arg1=1
      Searching bootorder for: /pci@i0cf8/*@14
      Searching bootorder for: /pci@i0cf8/*@14,1
      Searching bootorder for: /pci@i0cf8/*@14,2
      Searching bootorder for: /pci@i0cf8/*@14,3
      
      Press F12 for boot menu.
      
      Searching bootorder for: HALT
      drive 0x000f5890: PCHS=0/0/0 translation=lba LCHS=958/128/63 s=7733248
      Space available for UMB: c4000-f0000, f0000-f5890
      Returned 253952 bytes of ZoneHigh
      e820 map has 8 items:
        0: 0000000000000000 - 000000000009e800 = 1 RAM
        1: 000000000009e800 - 00000000000a0000 = 2 RESERVED
        2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
        3: 0000000000100000 - 000000007fbde000 = 1 RAM
        4: 000000007fbde000 - 000000007fe00000 = 2 RESERVED
        5: 00000000e0000000 - 00000000f0000000 = 2 RESERVED
        6: 00000000fee00000 - 00000000fee01000 = 2 RESERVED
        7: 0000000100000000 - 0000000280000000 = 1 RAM
      enter handle_19:
        NULL
      Booting from Hard Disk...
      Booting from 0000:7c00
      -
                __ ____
         _ __  / _/ ___|  ___ _ __  ___  ___
        | '_ \| |_\___ \ / _ \ '_ \/ __|/ _ \
        | |_) |  _|___) |  __/ | | \__ \  __/
        | .__/|_| |____/ \___|_| |_|___/\___|
        |_|
      
       +============Welcome to pfSense===========+
       |                                         |                 ______
       |  1\. Boot Multi User [Enter]             |                /      \
       |  2\. Boot [s]ingle User                  |          _____/    f   \
       |  3\. [Esc]ape to loader prompt           |         /     \        /
       |  4\. Reboot                              |        /   p   \______/  Sense
       |                                         |        \       /      \
       |  Options:                               |         \_____/        \
       |  5\. [K]ernel: kernel (1 of 2)           |               \        /
       |  6\. Configure Boot [O]ptions...         |                \______/
       |                                         |
       |                                         |
       |                                         |
       +=========================================+
      
      /boot/kernel/kernel text=0x1292d98 data=0x8a1bb0+0x23ec10 syms=[0x8+0x177720+0x8+0x173eef]
      Booting...
      KDB: debugger backends: ddb
      KDB: current backend: ddb
      Copyright (c) 1992-2016 The FreeBSD Project.
      Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
              The Regents of the University of California. All rights reserved.
      FreeBSD is a registered trademark of The FreeBSD Foundation.
      FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016
          root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense amd64
      FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
      CPU: Intel(R) Atom(TM) CPU  C2558  @ 2.40GHz (2400.06-MHz K8-class CPU)
        Origin="GenuineIntel"  Id=0x406d8  Family=0x6  Model=0x4d  Stepping=8
        Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x43d8e3bf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand>AMD Features=0x28100800 <syscall,nx,rdtscp,lm>AMD Features2=0x101 <lahf,prefetch>Structured Extended Features=0x2282 <tscadj,smep,erms,nfpusg>VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
        TSC: P-state invariant, performance statistics
      real memory  = 10737418240 (10240 MB)
      avail memory = 8220663808 (7839 MB)
      Event timer "LAPIC" quality 600
      ACPI APIC Table: <core  coreboot="">
      FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
      FreeBSD/SMP: 1 package(s) x 4 core(s)
       cpu0 (BSP): APIC ID:  0
       cpu1 (AP): APIC ID:  2
       cpu2 (AP): APIC ID:  4
       cpu3 (AP): APIC ID:  6
      random: <software, yarrow=""> initialized
      ioapic0 <version 2.0=""> irqs 0-23 on motherboard
      wlan: mac acl policy registered
      ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
      ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
      module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff806208f0, 0) error 1
      ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
      ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
      module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff806209a0, 0) error 1
      ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
      ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in/boot/loader.conf.
      module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80620a50, 0) error 1
      iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
      iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
      module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80647bf0, 0) error 1
      iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
      iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
      module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80647ca0, 0) error 1
      iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
      iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in/boot/loader.conf.
      module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80647d50, 0) error 1
      netmap: loaded module
      kbd0 at kbdmux0
      module_register_init: MOD_LOAD (vesa, 0xffffffff81016b90, 0) error 19
      cryptosoft0: <software crypto=""> on motherboard
      padlock0: No ACE support.
      acpi0: <core coreboot=""> on motherboard
      acpi0: Power Button (fixed)
      hpet0: invalid period
      device_attach: hpet0 attach returned 6
      cpu0: <acpi cpu=""> on acpi0
      cpu1: <acpi cpu=""> on acpi0
      cpu2: <acpi cpu=""> on acpi0
      cpu3: <acpi cpu=""> on acpi0
      hpet0: invalid period
      device_attach: hpet0 attach returned 6
      atrtc0: <at realtime="" clock=""> port 0x70-0x77 on acpi0
      Event timer "RTC" frequency 32768 Hz quality 0
      attimer0: <at timer=""> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
      Timecounter "i8254" frequency 1193182 Hz quality 0
      Event timer "i8254" frequency 1193182 Hz quality 100
      Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
      acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
      pcib0: <acpi host-pci="" bridge=""> port 0xcf8-0xcff on acpi0
      pci0: <acpi pci="" bus=""> on pcib0
      pcib1: <acpi pci-pci="" bridge=""> mem 0xdfe00000-0xdfe1ffff irq 16 at device 1.0 on pci0
      pci1: <acpi pci="" bus=""> on pcib1
      pcib2: <acpi pci-pci="" bridge=""> mem 0xdfe20000-0xdfe3ffff irq 19 at device 2.0 on pci0
      pci2: <acpi pci="" bus=""> on pcib2
      pcib3: <acpi pci-pci="" bridge=""> mem 0xdfe40000-0xdfe5ffff irq 20 at device 3.0 on pci0
      pci3: <acpi pci="" bus=""> on pcib3
      igb0: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x1000-0x101f mem 0xdfc00000-0xdfc1ffff,0xdfc20000-0xdfc23fff irq 18 at device 0.0 on pci3
      igb0: Using MSIX interrupts with 3 vectors
      igb0: Ethernet address: 00:08:a2:09:36:f0
      igb0: Bound queue 0 to cpu 0
      igb0: Bound queue 1 to cpu 1
      igb0: netmap queues/slots: TX 2/1024, RX 2/1024
      pcib4: <acpi pci-pci="" bridge=""> mem 0xdfe60000-0xdfe7ffff irq 23 at device 4.0 on pci0
      pci4: <acpi pci="" bus=""> on pcib4
      igb1: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x2000-0x201f mem 0xdfd00000-0xdfd1ffff,0xdfd20000-0xdfd23fff irq 19 at device 0.0 on pci4
      igb1: Using MSIX interrupts with 3 vectors
      igb1: Ethernet address: 00:08:a2:09:36:f1
      igb1: Bound queue 0 to cpu 2
      igb1: Bound queue 1 to cpu 3
      igb1: netmap queues/slots: TX 2/1024, RX 2/1024
      pci0: <processor> at device 11.0 (no driver attached)
      pci0: <base peripheral,="" iommu=""> at device 15.0 (no driver attached)
      igb2: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x3000-0x301f mem 0xdfea0000-0xdfebffff,0xdff24000-0xdff27fff irq 18 at device 20.0 on pci0
      igb2: Using MSIX interrupts with 5 vectors
      igb2: Ethernet address: 00:08:a2:09:36:ec
      igb2: Bound queue 0 to cpu 0
      igb2: Bound queue 1 to cpu 1
      igb2: Bound queue 2 to cpu 2
      igb2: Bound queue 3 to cpu 3
      igb2: netmap queues/slots: TX 4/1024, RX 4/1024
      igb3: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x3020-0x303f mem 0xdfec0000-0xdfedffff,0xdff28000-0xdff2bfff irq 19 at device 20.1 on pci0
      igb3: Using MSIX interrupts with 5 vectors
      igb3: Ethernet address: 00:08:a2:09:36:ed
      igb3: Bound queue 0 to cpu 0
      igb3: Bound queue 1 to cpu 1
      igb3: Bound queue 2 to cpu 2
      igb3: Bound queue 3 to cpu 3
      igb3: netmap queues/slots: TX 4/1024, RX 4/1024
      igb4: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x3040-0x305f mem 0xdfee0000-0xdfefffff,0xdff2c000-0xdff2ffff irq 20 at device 20.2 on pci0
      igb4: Using MSIX interrupts with 5 vectors
      igb4: Ethernet address: 00:08:a2:09:36:ee
      igb4: Bound queue 0 to cpu 0
      igb4: Bound queue 1 to cpu 1
      igb4: Bound queue 2 to cpu 2
      igb4: Bound queue 3 to cpu 3
      igb4: netmap queues/slots: TX 4/1024, RX 4/1024
      igb5: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x3060-0x307f mem 0xdff00000-0xdff1ffff,0xdff30000-0xdff33fff irq 21 at device 20.3 on pci0
      igb5: Using MSIX interrupts with 5 vectors
      igb5: Ethernet address: 00:08:a2:09:36:ef
      igb5: Bound queue 0 to cpu 0
      igb5: Bound queue 1 to cpu 1
      igb5: Bound queue 2 to cpu 2
      igb5: Bound queue 3 to cpu 3
      igb5: netmap queues/slots: TX 4/1024, RX 4/1024
      ehci0: <intel avoton="" usb="" 2.0="" controller=""> mem 0xdff35400-0xdff357ff irq 22 at device 22.0 on pci0
      usbus0: EHCI version 1.0
      usbus0 on ehci0
      ahci0: <intel avoton="" ahci="" sata="" controller=""> port 0x30c0-0x30c7,0x30e0-0x30e3,0x30c8-0x30cf,0x30e4-0x30e7,0x3080-0x309f mem 0xdff34000-0xdff347ff irq 23 at device                          23.0 on pci0
      ahci0: AHCI v1.30 with 4 3Gbps ports, Port Multiplier not supported
      ahci1: <intel avoton="" ahci="" sata="" controller=""> port 0x30d0-0x30d7,0x30e8-0x30eb,0x30d8-0x30df,0x30ec-0x30ef,0x30a0-0x30bf mem 0xdff34800-0xdff34fff irq 16 at device                          24.0 on pci0
      ahci1: AHCI v1.30 with 2 6Gbps ports, Port Multiplier not supported
      isab0: <pci-isa bridge=""> at device 31.0 on pci0
      isa0: <isa bus=""> on isab0
      ppc0: cannot reserve I/O port range
      uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 4 on isa0
      uart1: <16550 or compatible> at port 0x2f8-0x2ff irq 3 flags 0x10 on isa0
      uart1: console (115200,n,8,1)
      est0: <enhanced speedstep="" frequency="" control=""> on cpu0
      est1: <enhanced speedstep="" frequency="" control=""> on cpu1
      est2: <enhanced speedstep="" frequency="" control=""> on cpu2
      est3: <enhanced speedstep="" frequency="" control=""> on cpu3
      Timecounters tick every 1.000 msec
      random: unblocking device.
      usbus0: 480Mbps High Speed USB v2.0
      ugen0.1: <intel> at usbus0
      uhub0: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr=""> on usbus0
      uhub0: 8 ports with 8 removable, self powered
      ugen0.2: <vendor 0x8087=""> at usbus0
      uhub1: <vendor 2="" 9="" 0x8087="" product="" 0x07db,="" class="" 0,="" rev="" 2.00="" 0.02,="" addr=""> on usbus0
      uhub1: 4 ports with 4 removable, self powered
      ugen0.3: <generic> at usbus0
      umass0: <generic 0="" 3="" ultra="" fast="" media,="" class="" 0,="" rev="" 2.00="" 1.98,="" addr=""> on usbus0
      da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
      da0: <generic ultra="" hs-combo="" 1.98=""> Removable Direct Access SCSI device
      da0: Serial Number 000000225001
      da0: 40.000MB/s transfers
      da0: 3776MB (7733248 512 byte sectors)
      da0: quirks=0x2 <no_6_byte>SMP: AP CPU #1 Launched!
      SMP: AP CPU #3 Launched!
      SMP: AP CPU #2 Launched!
      Timecounter "TSC-low" frequency 1200029904 Hz quality 1000
      Trying to mount root from ufs:/dev/ufsid/58236354d4349f25 [rw]...
      WARNING: / was not properly dismounted
      WARNING: /: mount pending error: blocks 1176 files 32
      Configuring crash dumps...
      Using /dev/label/swap0 for dump device.
      ** SU+J Recovering /dev/ufsid/58236354d4349f25
      ** Reading 23199744 byte journal from inode 4.
      ** Building recovery table.
      ** Resolving unreferenced inode list.
      ** Processing journal entries.
      ** 665 journal records in 82432 bytes for 25.82% utilization
      ** Freed 69 inodes (2 dirs) 23 blocks, and 88 frags.
      
      ***** FILE SYSTEM MARKED CLEAN *****
      Filesystems are clean, continuing...
      Mounting filesystems...
      
           ___
       ___/ f \
      / p \___/ Sense
      \___/   \
          \___/
      
      Welcome to pfSense 2.3.2-RELEASE on the 'pfSense' platform...
      
      No core dumps found.
      Creating symlinks......ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec/usr/local/lib/perl5/5.20/mach/CORE
      32-bit compatibility ldconfig path: /usr/lib32
      done.
      External config loader 1.0 is now starting... da0s1 da0s1a da0s1b
      Launching the init system....... done.
      Initializing.................. done.
      Starting device manager (devd)...done.
      Loading configuration......done.
      Updating configuration...done.
      Cleaning backup cache...............done.
      Setting up extended sysctls...done.
      Setting timezone...done.
      Configuring loopback interface...done.
      Starting syslog...done.
      Starting Secure Shell Services...done.
      Setting up polling defaults...done.
      Setting up interfaces microcode...done.
      Configuring loopback interface...done.
      Creating wireless clone interfaces...done.
      Configuring LAGG interfaces...done.
      Configuring VLAN interfaces...done.
      Configuring QinQ interfaces...done.
      Configuring DMZ interface...done.
      Configuring LAN interface...done.
      Configuring OPT1VLAN2VOIP interface...done.
      Configuring OPT2VLAN3CAMERAS interface...done.
      Configuring SYNC interface...done.
      Configuring OPT5VLAN5MANAGMENTVLAN interface...done.
      Configuring OPT4VLAN4ESXISAN interface...done.
      Configuring CONTROL_RM_VLAN10 interface...done.
      Configuring CARP settings...done.
      Configuring CARP settings...done.
      Syncing OpenVPN settings...done.
      Configuring firewall.....0 addresses deleted.
      .done.
      Starting PFLOG...done.
      Setting up gateway monitors...done.
      Synchronizing user settings...done.
      Starting webConfigurator...done.
      Configuring CRON...done.
      Starting DNS forwarder...done.
      Starting NTP time client...done.
      Starting DHCP service...done.
      Configuring firewall.....g_vfs_done():md0[WRITE(offset=270336, length=4096)]error = 28
      
      Any idea what this means?[/s]</no_6_byte></generic></generic></generic></vendor></vendor></intel></intel></enhanced></enhanced></enhanced></enhanced></isa></pci-isa></intel></intel></intel></intel(r)></intel(r)></intel(r)></intel(r)></processor></intel(r)></acpi></acpi></intel(r)></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></at></at></acpi></acpi></acpi></acpi></core></software></version></software,></core ></tscadj,smep,erms,nfpusg></lahf,prefetch></syscall,nx,rdtscp,lm></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>
      
      1 Reply Last reply Reply Quote 0
      • jimpJ
        jimp Rebel Alliance Developer Netgate
        last edited by

        g_vfs_done():md0[WRITE(offset=270336, length=4096)]error = 28
        

        That is saying it can't write to a RAM disk (md0). That is a fairly certain indicator that system has somehow run out of RAM, or at least kernel memory.

        I don't think I've ever seen that happen on one of those units before, what sorts of things were in the config you restored? Which packages?

        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

        Need help fast? Netgate Global Support!

        Do not Chat/PM for help!

        1 Reply Last reply Reply Quote 0
        • D
          dbennett
          last edited by

          thanks for responding!

          This is from the current production device (2.2.6) that will be disco'd after the 2.3.2 unit is built.

          Package Name Category Package Version
          File Manager Utility 0.2.2
          FTP Client Proxy Services 0.3
          NRPE v2 Network Management 2.2.7
          ntopng Network Management 0.8.2
          OpenVPN Client Export Utility Security 1.3.0
          Service Watchdog Services 1.7.1
          syslog-ng System 1.1.1

          There were a number of crash dumps prior to this message which I was unable to retrieve.  Could crash dumps cause this?

          1 Reply Last reply Reply Quote 0
          • jimpJ
            jimp Rebel Alliance Developer Netgate
            last edited by

            The crash dumps wouldn't cause that, but it's possible that whatever caused the crash dumps also contributed to this. Without knowing what was in the other crashes it's difficult to say what might have happened.

            None of those packages should be too memory hungry except for ntopng, and it wouldn't have launched before that error came up.

            At this point I'd reinstall it fresh again and restore, and be sure to capture any crash report output you can from the serial console.

            Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

            Need help fast? Netgate Global Support!

            Do not Chat/PM for help!

            1 Reply Last reply Reply Quote 0
            • D
              dbennett
              last edited by

              Been building it from scratch (for the most part).  Reached a point that it crashed again.  Reinstalled the OS and a crash was indicated at the top of the screen.

              					Crash report begins.  Anonymous machine information:
              
              amd64
              10.3-RELEASE-p5
              FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016     root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense
              
              Crash report details:
              
              Filename: /var/crash/bounds
              1
              
              Filename: /var/crash/info.0
              Dump header from device /dev/label/swap0
                Architecture: amd64
                Architecture Version: 1
                Dump Length: 71680B (0 MB)
                Blocksize: 512
                Dumptime: Thu Nov 17 20:57:27 2016
                Hostname: hq-int-1.hq.informaticsinc.com
                Magic: FreeBSD Text Dump
                Version String: FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016
                  root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense
                Panic String: 
                Dump Parity: 2893441565
                Bounds: 0
                Dump Status: good
              
              Filename: /var/crash/info.last
              Dump header from device /dev/label/swap0
                Architecture: amd64
                Architecture Version: 1
                Dump Length: 71680B (0 MB)
                Blocksize: 512
                Dumptime: Thu Nov 17 20:57:27 2016
                Hostname: hq-int-1.hq.informaticsinc.com
                Magic: FreeBSD Text Dump
                Version String: FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016
                  root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense
                Panic String: 
                Dump Parity: 2893441565
                Bounds: 0
                Dump Status: good
              
              Filename: /var/crash/minfree
              2048
              
              Filename: /var/crash/textdump.tar.0
              ddb.txt06000014000013013414667  7077 ustarrootwheeldb:0:kdb.enter.default>  run lockinfo
              db:1:lockinfo> show locks
              No such command
              db:1:locks>  show alllocks
              No such command
              db:1:alllocks>  show lockedvnods
              Locked vnodes
              db:0:kdb.enter.default>  show pcpu
              cpuid        = 3
              dynamic pcpu = 0xfffffe026b8c8500
              curthread    = 0xfffff8000d25a960: pid 46850 "pfctl"
              curpcb       = 0xfffffe0235a1dcc0
              fpcurthread  = none
              idlethread   = 0xfffff80005240000: tid 100006 "idle: cpu3"
              curpmap      = 0xfffff80115c12d78
              tssp         = 0xffffffff821135c8
              commontssp   = 0xffffffff821135c8
              rsp0         = 0xfffffe0235a1dcc0
              gs32p        = 0xffffffff82115020
              ldt          = 0xffffffff82115060
              tss          = 0xffffffff82115050
              db:0:kdb.enter.default>  bt
              Tracing pid 46850 tid 100166 td 0xfffff8000d25a960
              m_tag_delete_chain() at m_tag_delete_chain+0x38/frame 0xfffffe0235a1ca30
              uma_zfree_arg() at uma_zfree_arg+0x3e/frame 0xfffffe0235a1caa0
              m_freem() at m_freem+0x63/frame 0xfffffe0235a1cac0
              hfsc_purgeq() at hfsc_purgeq+0x37/frame 0xfffffe0235a1cae0
              hfsc_class_destroy() at hfsc_class_destroy+0x75/frame 0xfffffe0235a1cb00
              hfsc_remove_altq() at hfsc_remove_altq+0x66/frame 0xfffffe0235a1cb30
              pf_commit_altq() at pf_commit_altq+0x221/frame 0xfffffe0235a1cb80
              pfioctl() at pfioctl+0x4e63/frame 0xfffffe0235a1d920
              devfs_ioctl_f() at devfs_ioctl_f+0x139/frame 0xfffffe0235a1d980
              kern_ioctl() at kern_ioctl+0x255/frame 0xfffffe0235a1d9f0
              sys_ioctl() at sys_ioctl+0x15c/frame 0xfffffe0235a1dad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235a1dbf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235a1dbf0
              --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800dce6ea, rsp = 0x7fffffffd9b8, rbp = 0x7fffffffd9d0 ---
              db:0:kdb.enter.default>  ps
                pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
              47065   280   280     0  R       CPU 1                       php-fpm
              46850   322    21     0  R+      CPU 3                       pfctl
              45872   280   280     0  S       lockf    0xfffff80115a3bc80 php-fpm
              39690     1 39690     0  Ls      (threaded)                  ntpd
              100164                   S       kqread   0xfffff801153b4700 ntpd
              100160                   L      *pf rules 0xfffff800054b4000 ntpd
              38583     1 38449 65534  S       select   0xfffff80115950640 dnsmasq
              34765     1 34765     0  Ss      nanslp   0xffffffff82001170 cron
              33967 33473 33473     0  S       kqread   0xfffff8011530f900 nginx
              33716 33473 33473     0  S       kqread   0xfffff8011530f400 nginx
              33473     1 33473     0  Ss      pause    0xfffff8011594ba88 nginx
              23430   280   280     0  S       lockf    0xfffff80115410e00 php-fpm
              18600     1 18600     0  Rs      (threaded)                  dpinger
              100153                   S       accept   0xfffff8000d1f488e dpinger
              100152                   S       nanslp   0xffffffff82001172 dpinger
              100151                   Run     CPU 0                       dpinger
              100150                   S       sbwait   0xfffff8000d1f4c24 dpinger
              100128                   S       uwait    0xfffff8000d14f800 dpinger
              18225     1 18225     0  Rs      (threaded)                  dpinger
              100149                   S       accept   0xfffff8000d1f531e dpinger
              100148                   RunQ                                dpinger
              100147                   L      *pf rules 0xfffff800054b4000 dpinger
              100146                   S       sbwait   0xfffff8000d1f596c dpinger
              100135                   S       uwait    0xfffff8000d150000 dpinger
              17658     1 17658     0  Rs      (threaded)                  dpinger
              100145                   S       accept   0xfffff801158f031e dpinger
              100144                   RunQ                                dpinger
              100143                   L      *pf rules 0xfffff800054b4000 dpinger
              100142                   S       sbwait   0xfffff801158f06b4 dpinger
              100137                   S       uwait    0xfffff8000d14dc00 dpinger
              15517     1 15517     0  Ss      select   0xfffff8000db19cc0 xinetd
              15124     1 15124     0  Ss      bpf      0xfffff8011530ae00 filterlog
               9174  3699  9174     0  Ss      (threaded)                  sshlockout_pf
              100141                   S       nanslp   0xffffffff82001173 sshlockout_pf
              100108                   S       piperd   0xfffff8000d0d4000 sshlockout_pf
               9167     1  9167     0  Ss      select   0xfffff8000db1a7c0 sshd
               3699     1  3699     0  Ss      select   0xfffff8000d14fdc0 syslogd
                332     1   332     0  Ss      select   0xfffff80005fa4c40 devd
                322    21    21     0  S+      piperd   0xfffff8000d1285d0 php-cgi
                320   318   318     0  S       kqread   0xfffff80005fe8200 check_reload_status
                318     1   318     0  Ss      kqread   0xfffff80005fe8400 check_reload_status
                281   280   280     0  S       nanslp   0xffffffff82001172 php-fpm
                280     1   280     0  Ss      kqread   0xfffff80005f7a500 php-fpm
                 51     0     0     0  DL      mdwait   0xfffff8000d012000 [md0]
                 21     1    21     0  Ss+     pause    0xfffff8000d039a88 sh
                 20     0     0     0  DL      syncer   0xffffffff82052108 [syncer]
                 19     0     0     0  DL      vlruwt   0xfffff80005f869e0 [vnlru]
                 18     0     0     0  DL      (threaded)                  [bufdaemon]
              100111                   D       sdflush  0xfffff8000d1030e8 [/ worker]
              100100                   D       psleep   0xffffffff82051304 [bufdaemon]
                 17     0     0     0  DL      pgzero   0xffffffff8206243c [pagezero]
                  9     0     0     0  DL      pollid   0xffffffff81fffa90 [idlepoll]
                  8     0     0     0  DL      psleep   0xffffffff820617c0 [vmdaemon]
                  7     0     0     0  DL      (threaded)                  [pagedaemon]
              100106                   D       umarcl   0xffffffff82061140 [uma]
              100096                   D       psleep   0xffffffff82112804 [pagedaemon]
                  6     0     0     0  DL      waiting_ 0xffffffff821032c0 [sctp_iterator]
                  5     0     0     0  LL     *pf rules 0xfffff800054b4000 [pf purge]
                 16     0     0     0  DL      (threaded)                  [usb]
              100085                   D       -        0xfffffe0000c56e70 [usbus0]
              100084                   D       -        0xfffffe0000c56e18 [usbus0]
              100083                   D       -        0xfffffe0000c56dc0 [usbus0]
              100082                   D       -        0xfffffe0000c56d68 [usbus0]
              100081                   D       -        0xfffffe0000c56d10 [usbus0]
                  4     0     0     0  DL      (threaded)                  [cam]
              100095                   D       -        0xffffffff81f35cc8 [scanner]
              100025                   D       -        0xffffffff81f35e80 [doneq0]
                 15     0     0     0  DL      -        0xffffffff81f575c0 [rand_harvestq]
                  3     0     0     0  DL      crypto_r 0xffffffff8205fc98 [crypto returns]
                  2     0     0     0  DL      crypto_w 0xffffffff8205fb40 [crypto]
                 14     0     0     0  DL      (threaded)                  [geom]
              100019                   D       -        0xffffffff820f79f8 [g_down]
              100018                   D       -        0xffffffff820f79f0 [g_up]
              100017                   D       -        0xffffffff820f79e8 [g_event]
                 13     0     0     0  DL      (threaded)                  [ng_queue]
              100016                   D       sleep    0xffffffff81ef4318 [ng_queue3]
              100015                   D       sleep    0xffffffff81ef4318 [ng_queue2]
              100014                   D       sleep    0xffffffff81ef4318 [ng_queue1]
              100013                   D       sleep    0xffffffff81ef4318 [ng_queue0]
                 12     0     0     0  RL      (threaded)                  [intr]
              100105                   I                                   [swi1: netisr 3]
              100104                   I                                   [swi1: netisr 2]
              100103                   I                                   [swi1: netisr 1]
              100093                   I                                   [swi1: pfsync]
              100091                   I                                   [swi1: pf send]
              100088                   I                                   [swi0: uart uart]
              100087                   I                                   [irq283: ahci1]
              100086                   I                                   [irq282: ahci0]
              100080                   I                                   [irq22: ehci0]
              100079                   I                                   [irq281: igb5:link]
              100077                   I                                   [irq280: igb5:que 3]
              100075                   I                                   [irq279: igb5:que 2]
              100073                   I                                   [irq278: igb5:que 1]
              100071                   I                                   [irq277: igb5:que 0]
              100070                   I                                   [irq276: igb4:link]
              100068                   I                                   [irq275: igb4:que 3]
              100066                   I                                   [irq274: igb4:que 2]
              100064                   I                                   [irq273: igb4:que 1]
              100062                   I                                   [irq272: igb4:que 0]
              100061                   I                                   [irq271: igb3:link]
              100059                   I                                   [irq270: igb3:que 3]
              100057                   I                                   [irq269: igb3:que 2]
              100055                   I                                   [irq268: igb3:que 1]
              100053                   I                                   [irq267: igb3:que 0]
              100052                   I                                   [irq266: igb2:link]
              100050                   I                                   [irq265: igb2:que 3]
              100048                   Run     CPU 2                       [irq264: igb2:que 2]
              100046                   L      *pf rules 0xfffff800054b4000 [irq263: igb2:que 1]
              100044                   I                                   [irq262: igb2:que 0]
              100043                   I                                   [irq261: igb1:link]
              100041                   I                                   [irq260: igb1:que 1]
              100039                   I                                   [irq259: igb1:que 0]
              100038                   I                                   [irq258: igb0:link]
              100036                   I                                   [irq257: igb0:que 1]
              100034                   I                                   [irq256: igb0:que 0]
              100033                   I                                   [swi5: fast taskq]
              100031                   I                                   [swi6: Giant taskq]
              100029                   I                                   [swi6: task queue]
              100012                   I                                   [swi4: clock]
              100011                   I                                   [swi4: clock]
              100010                   I                                   [swi4: clock]
              100009                   I                                   [swi4: clock]
              100008                   I                                   [swi1: netisr 0]
              100007                   I                                   [swi3: vm]
                 11     0     0     0  RL      (threaded)                  [idle]
              100006                   CanRun                              [idle: cpu3]
              100005                   CanRun                              [idle: cpu2]
              100004                   CanRun                              [idle: cpu1]
              100003                   CanRun                              [idle: cpu0]
                  1     0     1     0  SLs     wait     0xfffff8000523e4f0 [init]
                 10     0     0     0  DL      audit_wo 0xffffffff82109d60 [audit]
                  0     0     0     0  DLs     (threaded)                  [kernel]
              100094                   D       -        0xfffff80005260900 [CAM taskq]
              100089                   D       -        0xfffff800054f2700 [mca taskq]
              100078                   D       -        0xfffff800054c8200 [igb5 que]
              100076                   D       -        0xfffff800054c8500 [igb5 que]
              100074                   D       -        0xfffff800054c8800 [igb5 que]
              100072                   D       -        0xfffff800054c8b00 [igb5 que]
              100069                   D       -        0xfffff800054caa00 [igb4 que]
              100067                   D       -        0xfffff800054cad00 [igb4 que]
              100065                   D       -        0xfffff800054cb100 [igb4 que]
              100063                   D       -        0xfffff800054cb400 [igb4 que]
              100060                   D       -        0xfffff800054a6100 [igb3 que]
              100058                   D       -        0xfffff800054a6400 [igb3 que]
              100056                   D       -        0xfffff800054a6700 [igb3 que]
              100054                   D       -        0xfffff800054a6a00 [igb3 que]
              100051                   D       -        0xfffff8000543d700 [igb2 que]
              100049                   D       -        0xfffff8000543da00 [igb2 que]
              100047                   D       -        0xfffff8000543dd00 [igb2 que]
              100045                   D       -        0xfffff80005480100 [igb2 que]
              100042                   D       -        0xfffff80005482000 [igb1 que]
              100040                   D       -        0xfffff80005482300 [igb1 que]
              100037                   D       -        0xfffff80005262600 [igb0 que]
              100035                   D       -        0xfffff80005262900 [igb0 que]
              100032                   D       -        0xfffff8000525f800 [thread]
              100030                   D       -        0xfffff8000525fd00 [ffs_trim taskq]
              100028                   D       -        0xfffff80005260500 [acpi_task_2]
              100027                   D       -        0xfffff80005260500 [acpi_task_1]
              100026                   D       -        0xfffff80005260500 [acpi_task_0]
              100024                   D       -        0xfffff80005260b00 [kqueue taskq]
              100020                   D       -        0xfffff80005230100 [firmware taskq]
              100000                   D       swapin   0xffffffff820f7af8 [swapper]
              db:0:kdb.enter.default>  alltrace
              
              Tracing command php-fpm pid 47065 tid 100167 td 0xfffff8011594d4b0
              cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe01ed7ebcf0
              ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xfffffe01ed7ebd10
              trap() at trap+0x3a/frame 0xfffffe01ed7ebf20
              nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe01ed7ebf20
              --- trap 0x13, rip = 0x55f269, rsp = 0x7fffffff8220, rbp = 0x7fffffff82f0 ---
              
              Tracing command pfctl pid 46850 tid 100166 td 0xfffff8000d25a960
              m_tag_delete_chain() at m_tag_delete_chain+0x38/frame 0xfffffe0235a1ca30
              uma_zfree_arg() at uma_zfree_arg+0x3e/frame 0xfffffe0235a1caa0
              m_freem() at m_freem+0x63/frame 0xfffffe0235a1cac0
              hfsc_purgeq() at hfsc_purgeq+0x37/frame 0xfffffe0235a1cae0
              hfsc_class_destroy() at hfsc_class_destroy+0x75/frame 0xfffffe0235a1cb00
              hfsc_remove_altq() at hfsc_remove_altq+0x66/frame 0xfffffe0235a1cb30
              pf_commit_altq() at pf_commit_altq+0x221/frame 0xfffffe0235a1cb80
              pfioctl() at pfioctl+0x4e63/frame 0xfffffe0235a1d920
              devfs_ioctl_f() at devfs_ioctl_f+0x139/frame 0xfffffe0235a1d980
              kern_ioctl() at kern_ioctl+0x255/frame 0xfffffe0235a1d9f0
              sys_ioctl() at sys_ioctl+0x15c/frame 0xfffffe0235a1dad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235a1dbf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235a1dbf0
              --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800dce6ea, rsp = 0x7fffffffd9b8, rbp = 0x7fffffffd9d0 ---
              
              Tracing command php-fpm pid 45872 tid 100165 td 0xfffff8000d25b000
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235a18690
              mi_switch() at mi_switch+0xe1/frame 0xfffffe0235a186d0
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235a18750
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235a18780
              _sleep() at _sleep+0x27d/frame 0xfffffe0235a18800
              lf_advlockasync() at lf_advlockasync+0xb4a/frame 0xfffffe0235a188c0
              lf_advlock() at lf_advlock+0x45/frame 0xfffffe0235a18910
              vop_stdadvlock() at vop_stdadvlock+0xc7/frame 0xfffffe0235a18a00
              VOP_ADVLOCK_APV() at VOP_ADVLOCK_APV+0xa7/frame 0xfffffe0235a18a30
              sys_flock() at sys_flock+0x137/frame 0xfffffe0235a18ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235a18bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235a18bf0
              --- syscall (131, FreeBSD ELF64, sys_flock), rip = 0x801ae69ba, rsp = 0x7fffffffab08, rbp = 0x7fffffffab40 ---
              
              Tracing command ntpd pid 39690 tid 100164 td 0xfffff8011594d960
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235a13670
              mi_switch() at mi_switch+0xe1/frame 0xfffffe0235a136b0
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235a13730
              sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0235a13760
              _sleep() at _sleep+0x238/frame 0xfffffe0235a137e0
              kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe0235a139c0
              kern_kevent() at kern_kevent+0x9f/frame 0xfffffe0235a13a20
              sys_kevent() at sys_kevent+0x12a/frame 0xfffffe0235a13ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235a13bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235a13bf0
              --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x8015f88fa, rsp = 0x7fffdfffc5c8, rbp = 0x7fffdfffcef0 ---
              
              Tracing command ntpd pid 39690 tid 100160 td 0xfffff80115ac24b0
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359fd230
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02359fd270
              turnstile_wait() at turnstile_wait+0x42a/frame 0xfffffe02359fd2c0
              __rw_rlock() at __rw_rlock+0x319/frame 0xfffffe02359fd360
              pf_test() at pf_test+0x191/frame 0xfffffe02359fd680
              pf_check_out() at pf_check_out+0x1d/frame 0xfffffe02359fd6a0
              pfil_run_hooks() at pfil_run_hooks+0x8d/frame 0xfffffe02359fd730
              ip_output() at ip_output+0x9c0/frame 0xfffffe02359fd830
              udp_send() at udp_send+0x972/frame 0xfffffe02359fd900
              sosend_dgram() at sosend_dgram+0x406/frame 0xfffffe02359fd980
              kern_sendit() at kern_sendit+0x205/frame 0xfffffe02359fda30
              sendit() at sendit+0x129/frame 0xfffffe02359fda80
              sys_sendto() at sys_sendto+0x4d/frame 0xfffffe02359fdad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359fdbf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359fdbf0
              --- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x8015f88ba, rsp = 0x7fffffffddf8, rbp = 0x7fffffffde40 ---
              
              Tracing command dnsmasq pid 38583 tid 100162 td 0xfffff80115ac2000
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235a09790
              mi_switch() at mi_switch+0xe1/frame 0xfffffe0235a097d0
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235a09850
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235a09880
              _cv_wait_sig() at _cv_wait_sig+0x17a/frame 0xfffffe0235a098d0
              seltdwait() at seltdwait+0xae/frame 0xfffffe0235a09910
              kern_poll() at kern_poll+0x464/frame 0xfffffe0235a09ab0
              sys_poll() at sys_poll+0x61/frame 0xfffffe0235a09ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235a09bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235a09bf0
              --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x801463efa, rsp = 0x7fffffffeb08, rbp = 0x7fffffffec10 ---
              
              Tracing command cron pid 34765 tid 100154 td 0xfffff8011548b000
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359df8c0
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02359df900
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359df980
              sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe02359df9b0
              _sleep() at _sleep+0x238/frame 0xfffffe02359dfa30
              kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe02359dfa90
              sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe02359dfad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359dfbf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359dfbf0
              --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800d19f3a, rsp = 0x7fffffffea88, rbp = 0x7fffffffecb0 ---
              
              Tracing command nginx pid 33967 tid 100110 td 0xfffff80005f804b0
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235903670
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02359036b0
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235903730
              sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0235903760
              _sleep() at _sleep+0x238/frame 0xfffffe02359037e0
              kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe02359039c0
              kern_kevent() at kern_kevent+0x9f/frame 0xfffffe0235903a20
              sys_kevent() at sys_kevent+0x12a/frame 0xfffffe0235903ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235903bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235903bf0
              --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x8018fd8fa, rsp = 0x7fffffffe768, rbp = 0x7fffffffe7b0 ---
              
              Tracing command nginx pid 33716 tid 100156 td 0xfffff80115a9e960
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359e9670
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02359e96b0
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359e9730
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359e9760
              _sleep() at _sleep+0x27d/frame 0xfffffe02359e97e0
              kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe02359e99c0
              kern_kevent() at kern_kevent+0x9f/frame 0xfffffe02359e9a20
              sys_kevent() at sys_kevent+0x12a/frame 0xfffffe02359e9ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359e9bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359e9bf0
              --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x8018fd8fa, rsp = 0x7fffffffe768, rbp = 0x7fffffffe7b0 ---
              
              Tracing command nginx pid 33473 tid 100155 td 0xfffff8011594e960
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359e48d0
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02359e4910
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359e4990
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359e49c0
              _sleep() at _sleep+0x27d/frame 0xfffffe02359e4a40
              kern_sigsuspend() at kern_sigsuspend+0xf4/frame 0xfffffe02359e4aa0
              sys_sigsuspend() at sys_sigsuspend+0x31/frame 0xfffffe02359e4ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359e4bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359e4bf0
              --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x8018f2dfa, rsp = 0x7fffffffe908, rbp = 0x7fffffffe940 ---
              
              Tracing command php-fpm pid 23430 tid 100114 td 0xfffff8000d042960
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235917690
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02359176d0
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235917750
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235917780
              _sleep() at _sleep+0x27d/frame 0xfffffe0235917800
              lf_advlockasync() at lf_advlockasync+0xb4a/frame 0xfffffe02359178c0
              lf_advlock() at lf_advlock+0x45/frame 0xfffffe0235917910
              vop_stdadvlock() at vop_stdadvlock+0xc7/frame 0xfffffe0235917a00
              VOP_ADVLOCK_APV() at VOP_ADVLOCK_APV+0xa7/frame 0xfffffe0235917a30
              sys_flock() at sys_flock+0x137/frame 0xfffffe0235917ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235917bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235917bf0
              --- syscall (131, FreeBSD ELF64, sys_flock), rip = 0x801ae69ba, rsp = 0x7fffffffab08, rbp = 0x7fffffffab40 ---
              
              Tracing command dpinger pid 18600 tid 100153 td 0xfffff8011548b4b0
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359da850
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02359da890
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359da910
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359da940
              _sleep() at _sleep+0x27d/frame 0xfffffe02359da9c0
              kern_accept4() at kern_accept4+0x1b1/frame 0xfffffe02359daa70
              accept1() at accept1+0x92/frame 0xfffffe02359daad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359dabf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359dabf0
              --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x800b1bfda, rsp = 0x7fffdf9fae78, rbp = 0x7fffdf9faeb0 ---
              
              Tracing command dpinger pid 18600 tid 100152 td 0xfffff8011548b960
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359d58c0
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02359d5900
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359d5980
              sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe02359d59b0
              _sleep() at _sleep+0x238/frame 0xfffffe02359d5a30
              kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe02359d5a90
              sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe02359d5ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359d5bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359d5bf0
              --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b1bf3a, rsp = 0x7fffdfbfbef8, rbp = 0x7fffdfbfbf20 ---
              
              Tracing command dpinger pid 18600 tid 100151 td 0xfffff80115904000
              cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff820669e0
              ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xffffffff82066a00
              trap() at trap+0x3a/frame 0xffffffff82066c10
              nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff82066c10
              --- trap 0x13, rip = 0xffffffff80b090b2, rsp = 0xfffffe02359d0300, rbp = 0xfffffe02359d0390 ---
              __rw_rlock() at __rw_rlock+0x1f2/frame 0xfffffe02359d0390
              pf_test() at pf_test+0x191/frame 0xfffffe02359d06b0
              pf_check_out() at pf_check_out+0x1d/frame 0xfffffe02359d06d0
              pfil_run_hooks() at pfil_run_hooks+0x8d/frame 0xfffffe02359d0760
              ip_output() at ip_output+0x9c0/frame 0xfffffe02359d0860
              rip_output() at rip_output+0x2cd/frame 0xfffffe02359d08b0
              sosend_generic() at sosend_generic+0x476/frame 0xfffffe02359d0980
              kern_sendit() at kern_sendit+0x205/frame 0xfffffe02359d0a30
              sendit() at sendit+0x129/frame 0xfffffe02359d0a80
              sys_sendto() at sys_sendto+0x4d/frame 0xfffffe02359d0ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359d0bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359d0bf0
              --- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x800b268ba, rsp = 0x7fffdfdfcf38, rbp = 0x7fffdfdfcf80 ---
              
              Tracing command dpinger pid 18600 tid 100150 td 0xfffff801159044b0
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359cb740
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02359cb780
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359cb800
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359cb830
              _sleep() at _sleep+0x27d/frame 0xfffffe02359cb8b0
              sbwait() at sbwait+0x4f/frame 0xfffffe02359cb8d0
              soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe02359cb9a0
              kern_recvit() at kern_recvit+0x1f2/frame 0xfffffe02359cba60
              sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe02359cbad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359cbbf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359cbbf0
              --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b268da, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 ---
              
              Tracing command dpinger pid 18600 tid 100128 td 0xfffff8000d1e5000
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe023595d840
              mi_switch() at mi_switch+0xe1/frame 0xfffffe023595d880
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023595d900
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe023595d930
              _sleep() at _sleep+0x27d/frame 0xfffffe023595d9b0
              umtxq_sleep() at umtxq_sleep+0x125/frame 0xfffffe023595da10
              do_wait() at do_wait+0x387/frame 0xfffffe023595da90
              __umtx_op_wait() at __umtx_op_wait+0x7d/frame 0xfffffe023595dad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023595dbf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe023595dbf0
              --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083813c, rsp = 0x7fffffffe798, rbp = 0x7fffffffe810 ---
              
              Tracing command dpinger pid 18225 tid 100149 td 0xfffff80115904960
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359c6850
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02359c6890
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359c6910
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359c6940
              _sleep() at _sleep+0x27d/frame 0xfffffe02359c69c0
              kern_accept4() at kern_accept4+0x1b1/frame 0xfffffe02359c6a70
              accept1() at accept1+0x92/frame 0xfffffe02359c6ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359c6bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359c6bf0
              --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x800b1bfda, rsp = 0x7fffdf9fae78, rbp = 0x7fffdf9faeb0 ---
              
              Tracing command dpinger pid 18225 tid 100148 td 0xfffff80115905000
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359c18c0
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02359c1900
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359c1980
              sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe02359c19b0
              _sleep() at _sleep+0x238/frame 0xfffffe02359c1a30
              kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe02359c1a90
              sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe02359c1ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359c1bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359c1bf0
              --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b1bf3a, rsp = 0x7fffdfbfbef8, rbp = 0x7fffdfbfbf20 ---
              
              Tracing command dpinger pid 18225 tid 100147 td 0xfffff801159054b0
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359bc260
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02359bc2a0
              turnstile_wait() at turnstile_wait+0x42a/frame 0xfffffe02359bc2f0
              __rw_rlock() at __rw_rlock+0x319/frame 0xfffffe02359bc390
              pf_test() at pf_test+0x191/frame 0xfffffe02359bc6b0
              pf_check_out() at pf_check_out+0x1d/frame 0xfffffe02359bc6d0
              pfil_run_hooks() at pfil_run_hooks+0x8d/frame 0xfffffe02359bc760
              ip_output() at ip_output+0x9c0/frame 0xfffffe02359bc860
              rip_output() at rip_output+0x2cd/frame 0xfffffe02359bc8b0
              sosend_generic() at sosend_generic+0x476/frame 0xfffffe02359bc980
              kern_sendit() at kern_sendit+0x205/frame 0xfffffe02359bca30
              sendit() at sendit+0x129/frame 0xfffffe02359bca80
              sys_sendto() at sys_sendto+0x4d/frame 0xfffffe02359bcad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359bcbf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359bcbf0
              --- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x800b268ba, rsp = 0x7fffdfdfcf38, rbp = 0x7fffdfdfcf80 ---
              
              Tracing command dpinger pid 18225 tid 100146 td 0xfffff80115905960
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359b7740
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02359b7780
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359b7800
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359b7830
              _sleep() at _sleep+0x27d/frame 0xfffffe02359b78b0
              sbwait() at sbwait+0x4f/frame 0xfffffe02359b78d0
              soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe02359b79a0
              kern_recvit() at kern_recvit+0x1f2/frame 0xfffffe02359b7a60
              sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe02359b7ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359b7bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359b7bf0
              --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b268da, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 ---
              
              Tracing command dpinger pid 18225 tid 100135 td 0xfffff8000d1f8000
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235980840
              mi_switch() at mi_switch+0xe1/frame 0xfffffe0235980880
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235980900
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235980930
              _sleep() at _sleep+0x27d/frame 0xfffffe02359809b0
              umtxq_sleep() at umtxq_sleep+0x125/frame 0xfffffe0235980a10
              do_wait() at do_wait+0x387/frame 0xfffffe0235980a90
              __umtx_op_wait() at __umtx_op_wait+0x7d/frame 0xfffffe0235980ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235980bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235980bf0
              --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083813c, rsp = 0x7fffffffe798, rbp = 0x7fffffffe810 ---
              
              Tracing command dpinger pid 17658 tid 100145 td 0xfffff8000d48a4b0
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359b2850
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02359b2890
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359b2910
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359b2940
              _sleep() at _sleep+0x27d/frame 0xfffffe02359b29c0
              kern_accept4() at kern_accept4+0x1b1/frame 0xfffffe02359b2a70
              accept1() at accept1+0x92/frame 0xfffffe02359b2ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359b2bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359b2bf0
              --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x800b1bfda, rsp = 0x7fffdf9fae78, rbp = 0x7fffdf9faeb0 ---
              
              Tracing command dpinger pid 17658 tid 100144 td 0xfffff8000d48a960
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359ad8c0
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02359ad900
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359ad980
              sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe02359ad9b0
              _sleep() at _sleep+0x238/frame 0xfffffe02359ada30
              kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe02359ada90
              sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe02359adad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359adbf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359adbf0
              --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b1bf3a, rsp = 0x7fffdfbfbef8, rbp = 0x7fffdfbfbf20 ---
              
              Tracing command dpinger pid 17658 tid 100143 td 0xfffff8000d48b000
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359a8260
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02359a82a0
              turnstile_wait() at turnstile_wait+0x42a/frame 0xfffffe02359a82f0
              __rw_rlock() at __rw_rlock+0x319/frame 0xfffffe02359a8390
              pf_test() at pf_test+0x191/frame 0xfffffe02359a86b0
              pf_check_out() at pf_check_out+0x1d/frame 0xfffffe02359a86d0
              pfil_run_hooks() at pfil_run_hooks+0x8d/frame 0xfffffe02359a8760
              ip_output() at ip_output+0x9c0/frame 0xfffffe02359a8860
              rip_output() at rip_output+0x2cd/frame 0xfffffe02359a88b0
              sosend_generic() at sosend_generic+0x476/frame 0xfffffe02359a8980
              kern_sendit() at kern_sendit+0x205/frame 0xfffffe02359a8a30
              sendit() at sendit+0x129/frame 0xfffffe02359a8a80
              sys_sendto() at sys_sendto+0x4d/frame 0xfffffe02359a8ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359a8bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359a8bf0
              --- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x800b268ba, rsp = 0x7fffdfdfcf38, rbp = 0x7fffdfdfcf80 ---
              
              Tracing command dpinger pid 17658 tid 100142 td 0xfffff8000d48b4b0
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359a3740
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02359a3780
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359a3800
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359a3830
              _sleep() at _sleep+0x27d/frame 0xfffffe02359a38b0
              sbwait() at sbwait+0x4f/frame 0xfffffe02359a38d0
              soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe02359a39a0
              kern_recvit() at kern_recvit+0x1f2/frame 0xfffffe02359a3a60
              sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe02359a3ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359a3bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359a3bf0
              --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b268da, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 ---
              
              Tracing command dpinger pid 17658 tid 100137 td 0xfffff8000d4aa4b0
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe023598a840
              mi_switch() at mi_switch+0xe1/frame 0xfffffe023598a880
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023598a900
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe023598a930
              _sleep() at _sleep+0x27d/frame 0xfffffe023598a9b0
              umtxq_sleep() at umtxq_sleep+0x125/frame 0xfffffe023598aa10
              do_wait() at do_wait+0x387/frame 0xfffffe023598aa90
              __umtx_op_wait() at __umtx_op_wait+0x7d/frame 0xfffffe023598aad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023598abf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe023598abf0
              --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083813c, rsp = 0x7fffffffe7a8, rbp = 0x7fffffffe820 ---
              
              Tracing command xinetd pid 15517 tid 100134 td 0xfffff8000d1f84b0
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe023597b710
              mi_switch() at mi_switch+0xe1/frame 0xfffffe023597b750
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023597b7d0
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe023597b800
              _cv_wait_sig() at _cv_wait_sig+0x17a/frame 0xfffffe023597b850
              seltdwait() at seltdwait+0xae/frame 0xfffffe023597b890
              kern_select() at kern_select+0x8fa/frame 0xfffffe023597ba90
              sys_select() at sys_select+0x54/frame 0xfffffe023597bad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023597bbf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe023597bbf0
              --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800fca66a, rsp = 0x7fffffffeb08, rbp = 0x7fffffffec40 ---
              
              Tracing command filterlog pid 15124 tid 100140 td 0xfffff8000d03a4b0
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359997c0
              mi_switch() at mi_switch+0xe1/frame 0xfffffe0235999800
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235999880
              sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe02359998b0
              _sleep() at _sleep+0x238/frame 0xfffffe0235999930
              bpfread() at bpfread+0x1c7/frame 0xfffffe0235999980
              devfs_read_f() at devfs_read_f+0xd8/frame 0xfffffe02359999e0
              dofileread() at dofileread+0x95/frame 0xfffffe0235999a30
              kern_readv() at kern_readv+0x68/frame 0xfffffe0235999a80
              sys_read() at sys_read+0x63/frame 0xfffffe0235999ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235999bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235999bf0
              --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800da168a, rsp = 0x7fffffffeac8, rbp = 0x7fffffffec00 ---
              
              Tracing command sshlockout_pf pid 9174 tid 100141 td 0xfffff8000d03a000
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe023599e8c0
              mi_switch() at mi_switch+0xe1/frame 0xfffffe023599e900
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023599e980
              sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe023599e9b0
              _sleep() at _sleep+0x238/frame 0xfffffe023599ea30
              kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe023599ea90
              sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe023599ead0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023599ebf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe023599ebf0
              --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b18f3a, rsp = 0x7fffdfffdf48, rbp = 0x7fffdfffdf70 ---
              
              Tracing command sshlockout_pf pid 9174 tid 100108 td 0xfffff80005fa94b0
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe02358f9810
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02358f9850
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02358f98d0
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02358f9900
              _sleep() at _sleep+0x27d/frame 0xfffffe02358f9980
              pipe_read() at pipe_read+0x2aa/frame 0xfffffe02358f99e0
              dofileread() at dofileread+0x95/frame 0xfffffe02358f9a30
              kern_readv() at kern_readv+0x68/frame 0xfffffe02358f9a80
              sys_read() at sys_read+0x63/frame 0xfffffe02358f9ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02358f9bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02358f9bf0
              --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b7b68a, rsp = 0x7fffffffe7d8, rbp = 0x7fffffffe7f0 ---
              
              Tracing command sshd pid 9167 tid 100132 td 0xfffff8000d1e4000
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235971710
              mi_switch() at mi_switch+0xe1/frame 0xfffffe0235971750
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359717d0
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235971800
              _cv_wait_sig() at _cv_wait_sig+0x17a/frame 0xfffffe0235971850
              seltdwait() at seltdwait+0xae/frame 0xfffffe0235971890
              kern_select() at kern_select+0x8fa/frame 0xfffffe0235971a90
              sys_select() at sys_select+0x54/frame 0xfffffe0235971ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235971bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235971bf0
              --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80327966a, rsp = 0x7fffffffe228, rbp = 0x7fffffffe260 ---
              
              Tracing command syslogd pid 3699 tid 100139 td 0xfffff8000d48b960
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235994710
              mi_switch() at mi_switch+0xe1/frame 0xfffffe0235994750
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359947d0
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235994800
              _cv_wait_sig() at _cv_wait_sig+0x17a/frame 0xfffffe0235994850
              seltdwait() at seltdwait+0xae/frame 0xfffffe0235994890
              kern_select() at kern_select+0x8fa/frame 0xfffffe0235994a90
              sys_select() at sys_select+0x54/frame 0xfffffe0235994ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235994bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235994bf0
              --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800b7166a, rsp = 0x7fffffffdc08, rbp = 0x7fffffffec20 ---
              
              Tracing command devd pid 332 tid 100123 td 0xfffff80005f97000
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235944700
              mi_switch() at mi_switch+0xe1/frame 0xfffffe0235944740
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359447c0
              sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe02359447f0
              _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x19e/frame 0xfffffe0235944850
              seltdwait() at seltdwait+0xa4/frame 0xfffffe0235944890
              kern_select() at kern_select+0x8fa/frame 0xfffffe0235944a90
              sys_select() at sys_select+0x54/frame 0xfffffe0235944ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235944bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235944bf0
              --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x4a8caa, rsp = 0x7fffffffcb48, rbp = 0x7fffffffecd0 ---
              
              Tracing command php-cgi pid 322 tid 100124 td 0xfffff8000d1e54b0
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235949810
              mi_switch() at mi_switch+0xe1/frame 0xfffffe0235949850
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359498d0
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235949900
              _sleep() at _sleep+0x27d/frame 0xfffffe0235949980
              pipe_read() at pipe_read+0x2aa/frame 0xfffffe02359499e0
              dofileread() at dofileread+0x95/frame 0xfffffe0235949a30
              kern_readv() at kern_readv+0x68/frame 0xfffffe0235949a80
              sys_read() at sys_read+0x63/frame 0xfffffe0235949ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235949bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235949bf0
              --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x801b9968a, rsp = 0x7fffffffcb48, rbp = 0x7fffffffcb80 ---
              
              Tracing command check_reload_status pid 320 tid 100125 td 0xfffff8000d25b960
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe023594e670
              mi_switch() at mi_switch+0xe1/frame 0xfffffe023594e6b0
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023594e730
              sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe023594e760
              _sleep() at _sleep+0x27d/frame 0xfffffe023594e7e0
              kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe023594e9c0
              kern_kevent() at kern_kevent+0x9f/frame 0xfffffe023594ea20
              sys_kevent() at sys_kevent+0x12a/frame 0xfffffe023594ead0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023594ebf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe023594ebf0
              --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x800d478fa, rsp = 0x7fffffffe9e8, rbp = 0x7fffffffea30 ---
              
              Tracing command check_reload_status pid 318 tid 100129 td 0xfffff8000d03a960
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235962670
              mi_switch() at mi_switch+0xe1/frame 0xfffffe02359626b0
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235962730
              sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0235962760
              _sleep() at _sleep+0x238/frame 0xfffffe02359627e0
              kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe02359629c0
              kern_kevent() at kern_kevent+0x9f/frame 0xfffffe0235962a20
              sys_kevent() at sys_kevent+0x12a/frame 0xfffffe0235962ad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235962bf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235962bf0
              --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x800d478fa, rsp = 0x7fffffffe8a8, rbp = 0x7fffffffe8f0 ---
              
              Tracing command php-fpm pid 281 tid 100122 td 0xfffff80005f974b0
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe023593f8c0
              mi_switch() at mi_switch+0xe1/frame 0xfffffe023593f900
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023593f980
              sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe023593f9b0
              _sleep() at _sleep+0x238/frame 0xfffffe023593fa30
              kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe023593fa90
              sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe023593fad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023593fbf0
              Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe023593fbf0
              --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x801b4af3a, rsp = 0x7fffffffae78, rbp = 0x7fffffffaea0 ---
              
              Tracing command php-fpm pid 280 tid 100118 td 0xfffff80005fa8960
              sched_switch() at sched_switch+0x2b3/frame 0xfffffe023592b670
              mi_switch() at mi_switch+0xe1/frame 0xfffffe023592b6b0
              sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023592b730
              sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe023592b760
              _sleep() at _sleep+0x238/frame 0xfffffe023592b7e0
              kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe023592b9c0
              kern_kevent() at kern_kevent+0x9f/frame 0xfffffe023592ba20
              sys_kevent() at sys_kevent+0x12a/frame 0xfffffe023592bad0
              amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023592b[/thread]
              
              1 Reply Last reply Reply Quote 0
              • jimpJ
                jimp Rebel Alliance Developer Netgate
                last edited by

                Seems like it might be https://redmine.pfsense.org/issues/6257 but it's not quite an exact match for the ones we saw hitting that previously.

                Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                Need help fast? Netgate Global Support!

                Do not Chat/PM for help!

                1 Reply Last reply Reply Quote 0
                • D
                  dbennett
                  last edited by

                  Ironic, I manually rebuilt the config and when I added the Shaper it crashed.  the 2.2.6 Shaper must not be compatible in 2.3.

                  Thanks!!

                  1 Reply Last reply Reply Quote 0
                  • First post
                    Last post
                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.