SG-4860 "Configuring firewall…." on boot error



  • Just rebuilt my pfsense SG-4860, had to reload the OS which went smoothly, manually modified/updated the interface types (em# -> igb#), and rebooted it at least 4 times.

    I went to deploy it this morning and some of the servers on vLAN2 and 3 couldn't be reached.  Logged into the firewall GUI and noticed that the vlan assignments were incorrect.  I updated the interface assignments again, saved, and pretty much saw my entire network go down.  When i tried to access the console via the serial connection it connected but would load anything.  Failed back over to the old equipment then power cycled the SG.  This time the console came alive as it booted up.  That's when this error showed up and the boot up stalled.

    SeaBIOS (version SageBIOS-20150314_140038-localhost.localdomain)
    Found mainboard Intel Mohon Peak
    Relocating init from 0x000ef1b0 to 0x7fb995f0 (size 26960)
    Found CBFS header at 0xfffffbe0
    CPU Mhz=2401
    Found 20 PCI devices (max PCI bus is 04)
    Copying MPTABLE from 0x7fbe3e00/7fbe3e10 to 0x000f5950 with length 1ac
    Copying ACPI RSDP from 0x7fbe4e00 to 0x000f5920
    Copying SMBIOS entry point from 0x7fbf0400 to 0x000f5900
    Using pmtimer, ioport 0x408
    Scan for VGA option rom
    EHCI init on dev 00:16.0 (regs=0xdff35420)
    Found 2 serial ports
    ebda moved from 9f000 to 9e800
    AHCI controller at 17.0, iobase dff34000, irq 15
    AHCI controller at 18.0, iobase dff34800, irq 10
    Searching bootorder for: /pci@i0cf8/usb@16/hub@1/storage@4/*@0/*@0,0
    Searching bootorder for: /pci@i0cf8/usb@16/hub@1/usb-*@4
    USB MSC vendor='Generic' product='Ultra HS-COMBO' rev='1.98' type=0 removable=1
    USB MSC blksize=512 sectors=7733248
    Initialized USB HUB (1 ports used)
    All threads complete.
    Scan for option roms
    Running option rom at c000:0003
    pmm call arg1=1
    pmm call arg1=0
    pmm call arg1=1
    pmm call arg1=0
    Running option rom at c100:0003
    pmm call arg1=1
    pmm call arg1=1
    Running option rom at c200:0003
    pmm call arg1=1
    pmm call arg1=1
    Running option rom at c300:0003
    pmm call arg1=1
    pmm call arg1=1
    Searching bootorder for: /pci@i0cf8/*@14
    Searching bootorder for: /pci@i0cf8/*@14,1
    Searching bootorder for: /pci@i0cf8/*@14,2
    Searching bootorder for: /pci@i0cf8/*@14,3
    
    Press F12 for boot menu.
    
    Searching bootorder for: HALT
    drive 0x000f5890: PCHS=0/0/0 translation=lba LCHS=958/128/63 s=7733248
    Space available for UMB: c4000-f0000, f0000-f5890
    Returned 253952 bytes of ZoneHigh
    e820 map has 8 items:
      0: 0000000000000000 - 000000000009e800 = 1 RAM
      1: 000000000009e800 - 00000000000a0000 = 2 RESERVED
      2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
      3: 0000000000100000 - 000000007fbde000 = 1 RAM
      4: 000000007fbde000 - 000000007fe00000 = 2 RESERVED
      5: 00000000e0000000 - 00000000f0000000 = 2 RESERVED
      6: 00000000fee00000 - 00000000fee01000 = 2 RESERVED
      7: 0000000100000000 - 0000000280000000 = 1 RAM
    enter handle_19:
      NULL
    Booting from Hard Disk...
    Booting from 0000:7c00
    -
              __ ____
       _ __  / _/ ___|  ___ _ __  ___  ___
      | '_ \| |_\___ \ / _ \ '_ \/ __|/ _ \
      | |_) |  _|___) |  __/ | | \__ \  __/
      | .__/|_| |____/ \___|_| |_|___/\___|
      |_|
    
     +============Welcome to pfSense===========+
     |                                         |                 ______
     |  1\. Boot Multi User [Enter]             |                /      \
     |  2\. Boot [s]ingle User                  |          _____/    f   \
     |  3\. [Esc]ape to loader prompt           |         /     \        /
     |  4\. Reboot                              |        /   p   \______/  Sense
     |                                         |        \       /      \
     |  Options:                               |         \_____/        \
     |  5\. [K]ernel: kernel (1 of 2)           |               \        /
     |  6\. Configure Boot [O]ptions...         |                \______/
     |                                         |
     |                                         |
     |                                         |
     +=========================================+
    
    /boot/kernel/kernel text=0x1292d98 data=0x8a1bb0+0x23ec10 syms=[0x8+0x177720+0x8+0x173eef]
    Booting...
    KDB: debugger backends: ddb
    KDB: current backend: ddb
    Copyright (c) 1992-2016 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
            The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016
        root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense amd64
    FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
    CPU: Intel(R) Atom(TM) CPU  C2558  @ 2.40GHz (2400.06-MHz K8-class CPU)
      Origin="GenuineIntel"  Id=0x406d8  Family=0x6  Model=0x4d  Stepping=8
      Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x43d8e3bf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand>AMD Features=0x28100800 <syscall,nx,rdtscp,lm>AMD Features2=0x101 <lahf,prefetch>Structured Extended Features=0x2282 <tscadj,smep,erms,nfpusg>VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
      TSC: P-state invariant, performance statistics
    real memory  = 10737418240 (10240 MB)
    avail memory = 8220663808 (7839 MB)
    Event timer "LAPIC" quality 600
    ACPI APIC Table: <core  coreboot="">
    FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
    FreeBSD/SMP: 1 package(s) x 4 core(s)
     cpu0 (BSP): APIC ID:  0
     cpu1 (AP): APIC ID:  2
     cpu2 (AP): APIC ID:  4
     cpu3 (AP): APIC ID:  6
    random: <software, yarrow=""> initialized
    ioapic0 <version 2.0=""> irqs 0-23 on motherboard
    wlan: mac acl policy registered
    ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff806208f0, 0) error 1
    ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff806209a0, 0) error 1
    ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in/boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80620a50, 0) error 1
    iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
    iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80647bf0, 0) error 1
    iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
    iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80647ca0, 0) error 1
    iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
    iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in/boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80647d50, 0) error 1
    netmap: loaded module
    kbd0 at kbdmux0
    module_register_init: MOD_LOAD (vesa, 0xffffffff81016b90, 0) error 19
    cryptosoft0: <software crypto=""> on motherboard
    padlock0: No ACE support.
    acpi0: <core coreboot=""> on motherboard
    acpi0: Power Button (fixed)
    hpet0: invalid period
    device_attach: hpet0 attach returned 6
    cpu0: <acpi cpu=""> on acpi0
    cpu1: <acpi cpu=""> on acpi0
    cpu2: <acpi cpu=""> on acpi0
    cpu3: <acpi cpu=""> on acpi0
    hpet0: invalid period
    device_attach: hpet0 attach returned 6
    atrtc0: <at realtime="" clock=""> port 0x70-0x77 on acpi0
    Event timer "RTC" frequency 32768 Hz quality 0
    attimer0: <at timer=""> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
    Timecounter "i8254" frequency 1193182 Hz quality 0
    Event timer "i8254" frequency 1193182 Hz quality 100
    Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
    acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
    pcib0: <acpi host-pci="" bridge=""> port 0xcf8-0xcff on acpi0
    pci0: <acpi pci="" bus=""> on pcib0
    pcib1: <acpi pci-pci="" bridge=""> mem 0xdfe00000-0xdfe1ffff irq 16 at device 1.0 on pci0
    pci1: <acpi pci="" bus=""> on pcib1
    pcib2: <acpi pci-pci="" bridge=""> mem 0xdfe20000-0xdfe3ffff irq 19 at device 2.0 on pci0
    pci2: <acpi pci="" bus=""> on pcib2
    pcib3: <acpi pci-pci="" bridge=""> mem 0xdfe40000-0xdfe5ffff irq 20 at device 3.0 on pci0
    pci3: <acpi pci="" bus=""> on pcib3
    igb0: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x1000-0x101f mem 0xdfc00000-0xdfc1ffff,0xdfc20000-0xdfc23fff irq 18 at device 0.0 on pci3
    igb0: Using MSIX interrupts with 3 vectors
    igb0: Ethernet address: 00:08:a2:09:36:f0
    igb0: Bound queue 0 to cpu 0
    igb0: Bound queue 1 to cpu 1
    igb0: netmap queues/slots: TX 2/1024, RX 2/1024
    pcib4: <acpi pci-pci="" bridge=""> mem 0xdfe60000-0xdfe7ffff irq 23 at device 4.0 on pci0
    pci4: <acpi pci="" bus=""> on pcib4
    igb1: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x2000-0x201f mem 0xdfd00000-0xdfd1ffff,0xdfd20000-0xdfd23fff irq 19 at device 0.0 on pci4
    igb1: Using MSIX interrupts with 3 vectors
    igb1: Ethernet address: 00:08:a2:09:36:f1
    igb1: Bound queue 0 to cpu 2
    igb1: Bound queue 1 to cpu 3
    igb1: netmap queues/slots: TX 2/1024, RX 2/1024
    pci0: <processor> at device 11.0 (no driver attached)
    pci0: <base peripheral,="" iommu=""> at device 15.0 (no driver attached)
    igb2: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x3000-0x301f mem 0xdfea0000-0xdfebffff,0xdff24000-0xdff27fff irq 18 at device 20.0 on pci0
    igb2: Using MSIX interrupts with 5 vectors
    igb2: Ethernet address: 00:08:a2:09:36:ec
    igb2: Bound queue 0 to cpu 0
    igb2: Bound queue 1 to cpu 1
    igb2: Bound queue 2 to cpu 2
    igb2: Bound queue 3 to cpu 3
    igb2: netmap queues/slots: TX 4/1024, RX 4/1024
    igb3: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x3020-0x303f mem 0xdfec0000-0xdfedffff,0xdff28000-0xdff2bfff irq 19 at device 20.1 on pci0
    igb3: Using MSIX interrupts with 5 vectors
    igb3: Ethernet address: 00:08:a2:09:36:ed
    igb3: Bound queue 0 to cpu 0
    igb3: Bound queue 1 to cpu 1
    igb3: Bound queue 2 to cpu 2
    igb3: Bound queue 3 to cpu 3
    igb3: netmap queues/slots: TX 4/1024, RX 4/1024
    igb4: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x3040-0x305f mem 0xdfee0000-0xdfefffff,0xdff2c000-0xdff2ffff irq 20 at device 20.2 on pci0
    igb4: Using MSIX interrupts with 5 vectors
    igb4: Ethernet address: 00:08:a2:09:36:ee
    igb4: Bound queue 0 to cpu 0
    igb4: Bound queue 1 to cpu 1
    igb4: Bound queue 2 to cpu 2
    igb4: Bound queue 3 to cpu 3
    igb4: netmap queues/slots: TX 4/1024, RX 4/1024
    igb5: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x3060-0x307f mem 0xdff00000-0xdff1ffff,0xdff30000-0xdff33fff irq 21 at device 20.3 on pci0
    igb5: Using MSIX interrupts with 5 vectors
    igb5: Ethernet address: 00:08:a2:09:36:ef
    igb5: Bound queue 0 to cpu 0
    igb5: Bound queue 1 to cpu 1
    igb5: Bound queue 2 to cpu 2
    igb5: Bound queue 3 to cpu 3
    igb5: netmap queues/slots: TX 4/1024, RX 4/1024
    ehci0: <intel avoton="" usb="" 2.0="" controller=""> mem 0xdff35400-0xdff357ff irq 22 at device 22.0 on pci0
    usbus0: EHCI version 1.0
    usbus0 on ehci0
    ahci0: <intel avoton="" ahci="" sata="" controller=""> port 0x30c0-0x30c7,0x30e0-0x30e3,0x30c8-0x30cf,0x30e4-0x30e7,0x3080-0x309f mem 0xdff34000-0xdff347ff irq 23 at device                          23.0 on pci0
    ahci0: AHCI v1.30 with 4 3Gbps ports, Port Multiplier not supported
    ahci1: <intel avoton="" ahci="" sata="" controller=""> port 0x30d0-0x30d7,0x30e8-0x30eb,0x30d8-0x30df,0x30ec-0x30ef,0x30a0-0x30bf mem 0xdff34800-0xdff34fff irq 16 at device                          24.0 on pci0
    ahci1: AHCI v1.30 with 2 6Gbps ports, Port Multiplier not supported
    isab0: <pci-isa bridge=""> at device 31.0 on pci0
    isa0: <isa bus=""> on isab0
    ppc0: cannot reserve I/O port range
    uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 4 on isa0
    uart1: <16550 or compatible> at port 0x2f8-0x2ff irq 3 flags 0x10 on isa0
    uart1: console (115200,n,8,1)
    est0: <enhanced speedstep="" frequency="" control=""> on cpu0
    est1: <enhanced speedstep="" frequency="" control=""> on cpu1
    est2: <enhanced speedstep="" frequency="" control=""> on cpu2
    est3: <enhanced speedstep="" frequency="" control=""> on cpu3
    Timecounters tick every 1.000 msec
    random: unblocking device.
    usbus0: 480Mbps High Speed USB v2.0
    ugen0.1: <intel> at usbus0
    uhub0: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr=""> on usbus0
    uhub0: 8 ports with 8 removable, self powered
    ugen0.2: <vendor 0x8087=""> at usbus0
    uhub1: <vendor 2="" 9="" 0x8087="" product="" 0x07db,="" class="" 0,="" rev="" 2.00="" 0.02,="" addr=""> on usbus0
    uhub1: 4 ports with 4 removable, self powered
    ugen0.3: <generic> at usbus0
    umass0: <generic 0="" 3="" ultra="" fast="" media,="" class="" 0,="" rev="" 2.00="" 1.98,="" addr=""> on usbus0
    da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
    da0: <generic ultra="" hs-combo="" 1.98=""> Removable Direct Access SCSI device
    da0: Serial Number 000000225001
    da0: 40.000MB/s transfers
    da0: 3776MB (7733248 512 byte sectors)
    da0: quirks=0x2 <no_6_byte>SMP: AP CPU #1 Launched!
    SMP: AP CPU #3 Launched!
    SMP: AP CPU #2 Launched!
    Timecounter "TSC-low" frequency 1200029904 Hz quality 1000
    Trying to mount root from ufs:/dev/ufsid/58236354d4349f25 [rw]...
    WARNING: / was not properly dismounted
    WARNING: /: mount pending error: blocks 1176 files 32
    Configuring crash dumps...
    Using /dev/label/swap0 for dump device.
    ** SU+J Recovering /dev/ufsid/58236354d4349f25
    ** Reading 23199744 byte journal from inode 4.
    ** Building recovery table.
    ** Resolving unreferenced inode list.
    ** Processing journal entries.
    ** 665 journal records in 82432 bytes for 25.82% utilization
    ** Freed 69 inodes (2 dirs) 23 blocks, and 88 frags.
    
    ***** FILE SYSTEM MARKED CLEAN *****
    Filesystems are clean, continuing...
    Mounting filesystems...
    
         ___
     ___/ f \
    / p \___/ Sense
    \___/   \
        \___/
    
    Welcome to pfSense 2.3.2-RELEASE on the 'pfSense' platform...
    
    No core dumps found.
    Creating symlinks......ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec/usr/local/lib/perl5/5.20/mach/CORE
    32-bit compatibility ldconfig path: /usr/lib32
    done.
    External config loader 1.0 is now starting... da0s1 da0s1a da0s1b
    Launching the init system....... done.
    Initializing.................. done.
    Starting device manager (devd)...done.
    Loading configuration......done.
    Updating configuration...done.
    Cleaning backup cache...............done.
    Setting up extended sysctls...done.
    Setting timezone...done.
    Configuring loopback interface...done.
    Starting syslog...done.
    Starting Secure Shell Services...done.
    Setting up polling defaults...done.
    Setting up interfaces microcode...done.
    Configuring loopback interface...done.
    Creating wireless clone interfaces...done.
    Configuring LAGG interfaces...done.
    Configuring VLAN interfaces...done.
    Configuring QinQ interfaces...done.
    Configuring DMZ interface...done.
    Configuring LAN interface...done.
    Configuring OPT1VLAN2VOIP interface...done.
    Configuring OPT2VLAN3CAMERAS interface...done.
    Configuring SYNC interface...done.
    Configuring OPT5VLAN5MANAGMENTVLAN interface...done.
    Configuring OPT4VLAN4ESXISAN interface...done.
    Configuring CONTROL_RM_VLAN10 interface...done.
    Configuring CARP settings...done.
    Configuring CARP settings...done.
    Syncing OpenVPN settings...done.
    Configuring firewall.....0 addresses deleted.
    .done.
    Starting PFLOG...done.
    Setting up gateway monitors...done.
    Synchronizing user settings...done.
    Starting webConfigurator...done.
    Configuring CRON...done.
    Starting DNS forwarder...done.
    Starting NTP time client...done.
    Starting DHCP service...done.
    Configuring firewall.....g_vfs_done():md0[WRITE(offset=270336, length=4096)]error = 28
    
    Any idea what this means?[/s]</no_6_byte></generic></generic></generic></vendor></vendor></intel></intel></enhanced></enhanced></enhanced></enhanced></isa></pci-isa></intel></intel></intel></intel(r)></intel(r)></intel(r)></intel(r)></processor></intel(r)></acpi></acpi></intel(r)></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></at></at></acpi></acpi></acpi></acpi></core></software></version></software,></core ></tscadj,smep,erms,nfpusg></lahf,prefetch></syscall,nx,rdtscp,lm></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>
    

  • Rebel Alliance Developer Netgate

    g_vfs_done():md0[WRITE(offset=270336, length=4096)]error = 28
    

    That is saying it can't write to a RAM disk (md0). That is a fairly certain indicator that system has somehow run out of RAM, or at least kernel memory.

    I don't think I've ever seen that happen on one of those units before, what sorts of things were in the config you restored? Which packages?



  • thanks for responding!

    This is from the current production device (2.2.6) that will be disco'd after the 2.3.2 unit is built.

    Package Name Category Package Version
    File Manager Utility 0.2.2
    FTP Client Proxy Services 0.3
    NRPE v2 Network Management 2.2.7
    ntopng Network Management 0.8.2
    OpenVPN Client Export Utility Security 1.3.0
    Service Watchdog Services 1.7.1
    syslog-ng System 1.1.1

    There were a number of crash dumps prior to this message which I was unable to retrieve.  Could crash dumps cause this?


  • Rebel Alliance Developer Netgate

    The crash dumps wouldn't cause that, but it's possible that whatever caused the crash dumps also contributed to this. Without knowing what was in the other crashes it's difficult to say what might have happened.

    None of those packages should be too memory hungry except for ntopng, and it wouldn't have launched before that error came up.

    At this point I'd reinstall it fresh again and restore, and be sure to capture any crash report output you can from the serial console.



  • Been building it from scratch (for the most part).  Reached a point that it crashed again.  Reinstalled the OS and a crash was indicated at the top of the screen.

    					Crash report begins.  Anonymous machine information:
    
    amd64
    10.3-RELEASE-p5
    FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016     root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense
    
    Crash report details:
    
    Filename: /var/crash/bounds
    1
    
    Filename: /var/crash/info.0
    Dump header from device /dev/label/swap0
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 71680B (0 MB)
      Blocksize: 512
      Dumptime: Thu Nov 17 20:57:27 2016
      Hostname: hq-int-1.hq.informaticsinc.com
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016
        root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense
      Panic String: 
      Dump Parity: 2893441565
      Bounds: 0
      Dump Status: good
    
    Filename: /var/crash/info.last
    Dump header from device /dev/label/swap0
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 71680B (0 MB)
      Blocksize: 512
      Dumptime: Thu Nov 17 20:57:27 2016
      Hostname: hq-int-1.hq.informaticsinc.com
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016
        root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense
      Panic String: 
      Dump Parity: 2893441565
      Bounds: 0
      Dump Status: good
    
    Filename: /var/crash/minfree
    2048
    
    Filename: /var/crash/textdump.tar.0
    ddb.txt06000014000013013414667  7077 ustarrootwheeldb:0:kdb.enter.default>  run lockinfo
    db:1:lockinfo> show locks
    No such command
    db:1:locks>  show alllocks
    No such command
    db:1:alllocks>  show lockedvnods
    Locked vnodes
    db:0:kdb.enter.default>  show pcpu
    cpuid        = 3
    dynamic pcpu = 0xfffffe026b8c8500
    curthread    = 0xfffff8000d25a960: pid 46850 "pfctl"
    curpcb       = 0xfffffe0235a1dcc0
    fpcurthread  = none
    idlethread   = 0xfffff80005240000: tid 100006 "idle: cpu3"
    curpmap      = 0xfffff80115c12d78
    tssp         = 0xffffffff821135c8
    commontssp   = 0xffffffff821135c8
    rsp0         = 0xfffffe0235a1dcc0
    gs32p        = 0xffffffff82115020
    ldt          = 0xffffffff82115060
    tss          = 0xffffffff82115050
    db:0:kdb.enter.default>  bt
    Tracing pid 46850 tid 100166 td 0xfffff8000d25a960
    m_tag_delete_chain() at m_tag_delete_chain+0x38/frame 0xfffffe0235a1ca30
    uma_zfree_arg() at uma_zfree_arg+0x3e/frame 0xfffffe0235a1caa0
    m_freem() at m_freem+0x63/frame 0xfffffe0235a1cac0
    hfsc_purgeq() at hfsc_purgeq+0x37/frame 0xfffffe0235a1cae0
    hfsc_class_destroy() at hfsc_class_destroy+0x75/frame 0xfffffe0235a1cb00
    hfsc_remove_altq() at hfsc_remove_altq+0x66/frame 0xfffffe0235a1cb30
    pf_commit_altq() at pf_commit_altq+0x221/frame 0xfffffe0235a1cb80
    pfioctl() at pfioctl+0x4e63/frame 0xfffffe0235a1d920
    devfs_ioctl_f() at devfs_ioctl_f+0x139/frame 0xfffffe0235a1d980
    kern_ioctl() at kern_ioctl+0x255/frame 0xfffffe0235a1d9f0
    sys_ioctl() at sys_ioctl+0x15c/frame 0xfffffe0235a1dad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235a1dbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235a1dbf0
    --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800dce6ea, rsp = 0x7fffffffd9b8, rbp = 0x7fffffffd9d0 ---
    db:0:kdb.enter.default>  ps
      pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
    47065   280   280     0  R       CPU 1                       php-fpm
    46850   322    21     0  R+      CPU 3                       pfctl
    45872   280   280     0  S       lockf    0xfffff80115a3bc80 php-fpm
    39690     1 39690     0  Ls      (threaded)                  ntpd
    100164                   S       kqread   0xfffff801153b4700 ntpd
    100160                   L      *pf rules 0xfffff800054b4000 ntpd
    38583     1 38449 65534  S       select   0xfffff80115950640 dnsmasq
    34765     1 34765     0  Ss      nanslp   0xffffffff82001170 cron
    33967 33473 33473     0  S       kqread   0xfffff8011530f900 nginx
    33716 33473 33473     0  S       kqread   0xfffff8011530f400 nginx
    33473     1 33473     0  Ss      pause    0xfffff8011594ba88 nginx
    23430   280   280     0  S       lockf    0xfffff80115410e00 php-fpm
    18600     1 18600     0  Rs      (threaded)                  dpinger
    100153                   S       accept   0xfffff8000d1f488e dpinger
    100152                   S       nanslp   0xffffffff82001172 dpinger
    100151                   Run     CPU 0                       dpinger
    100150                   S       sbwait   0xfffff8000d1f4c24 dpinger
    100128                   S       uwait    0xfffff8000d14f800 dpinger
    18225     1 18225     0  Rs      (threaded)                  dpinger
    100149                   S       accept   0xfffff8000d1f531e dpinger
    100148                   RunQ                                dpinger
    100147                   L      *pf rules 0xfffff800054b4000 dpinger
    100146                   S       sbwait   0xfffff8000d1f596c dpinger
    100135                   S       uwait    0xfffff8000d150000 dpinger
    17658     1 17658     0  Rs      (threaded)                  dpinger
    100145                   S       accept   0xfffff801158f031e dpinger
    100144                   RunQ                                dpinger
    100143                   L      *pf rules 0xfffff800054b4000 dpinger
    100142                   S       sbwait   0xfffff801158f06b4 dpinger
    100137                   S       uwait    0xfffff8000d14dc00 dpinger
    15517     1 15517     0  Ss      select   0xfffff8000db19cc0 xinetd
    15124     1 15124     0  Ss      bpf      0xfffff8011530ae00 filterlog
     9174  3699  9174     0  Ss      (threaded)                  sshlockout_pf
    100141                   S       nanslp   0xffffffff82001173 sshlockout_pf
    100108                   S       piperd   0xfffff8000d0d4000 sshlockout_pf
     9167     1  9167     0  Ss      select   0xfffff8000db1a7c0 sshd
     3699     1  3699     0  Ss      select   0xfffff8000d14fdc0 syslogd
      332     1   332     0  Ss      select   0xfffff80005fa4c40 devd
      322    21    21     0  S+      piperd   0xfffff8000d1285d0 php-cgi
      320   318   318     0  S       kqread   0xfffff80005fe8200 check_reload_status
      318     1   318     0  Ss      kqread   0xfffff80005fe8400 check_reload_status
      281   280   280     0  S       nanslp   0xffffffff82001172 php-fpm
      280     1   280     0  Ss      kqread   0xfffff80005f7a500 php-fpm
       51     0     0     0  DL      mdwait   0xfffff8000d012000 [md0]
       21     1    21     0  Ss+     pause    0xfffff8000d039a88 sh
       20     0     0     0  DL      syncer   0xffffffff82052108 [syncer]
       19     0     0     0  DL      vlruwt   0xfffff80005f869e0 [vnlru]
       18     0     0     0  DL      (threaded)                  [bufdaemon]
    100111                   D       sdflush  0xfffff8000d1030e8 [/ worker]
    100100                   D       psleep   0xffffffff82051304 [bufdaemon]
       17     0     0     0  DL      pgzero   0xffffffff8206243c [pagezero]
        9     0     0     0  DL      pollid   0xffffffff81fffa90 [idlepoll]
        8     0     0     0  DL      psleep   0xffffffff820617c0 [vmdaemon]
        7     0     0     0  DL      (threaded)                  [pagedaemon]
    100106                   D       umarcl   0xffffffff82061140 [uma]
    100096                   D       psleep   0xffffffff82112804 [pagedaemon]
        6     0     0     0  DL      waiting_ 0xffffffff821032c0 [sctp_iterator]
        5     0     0     0  LL     *pf rules 0xfffff800054b4000 [pf purge]
       16     0     0     0  DL      (threaded)                  [usb]
    100085                   D       -        0xfffffe0000c56e70 [usbus0]
    100084                   D       -        0xfffffe0000c56e18 [usbus0]
    100083                   D       -        0xfffffe0000c56dc0 [usbus0]
    100082                   D       -        0xfffffe0000c56d68 [usbus0]
    100081                   D       -        0xfffffe0000c56d10 [usbus0]
        4     0     0     0  DL      (threaded)                  [cam]
    100095                   D       -        0xffffffff81f35cc8 [scanner]
    100025                   D       -        0xffffffff81f35e80 [doneq0]
       15     0     0     0  DL      -        0xffffffff81f575c0 [rand_harvestq]
        3     0     0     0  DL      crypto_r 0xffffffff8205fc98 [crypto returns]
        2     0     0     0  DL      crypto_w 0xffffffff8205fb40 [crypto]
       14     0     0     0  DL      (threaded)                  [geom]
    100019                   D       -        0xffffffff820f79f8 [g_down]
    100018                   D       -        0xffffffff820f79f0 [g_up]
    100017                   D       -        0xffffffff820f79e8 [g_event]
       13     0     0     0  DL      (threaded)                  [ng_queue]
    100016                   D       sleep    0xffffffff81ef4318 [ng_queue3]
    100015                   D       sleep    0xffffffff81ef4318 [ng_queue2]
    100014                   D       sleep    0xffffffff81ef4318 [ng_queue1]
    100013                   D       sleep    0xffffffff81ef4318 [ng_queue0]
       12     0     0     0  RL      (threaded)                  [intr]
    100105                   I                                   [swi1: netisr 3]
    100104                   I                                   [swi1: netisr 2]
    100103                   I                                   [swi1: netisr 1]
    100093                   I                                   [swi1: pfsync]
    100091                   I                                   [swi1: pf send]
    100088                   I                                   [swi0: uart uart]
    100087                   I                                   [irq283: ahci1]
    100086                   I                                   [irq282: ahci0]
    100080                   I                                   [irq22: ehci0]
    100079                   I                                   [irq281: igb5:link]
    100077                   I                                   [irq280: igb5:que 3]
    100075                   I                                   [irq279: igb5:que 2]
    100073                   I                                   [irq278: igb5:que 1]
    100071                   I                                   [irq277: igb5:que 0]
    100070                   I                                   [irq276: igb4:link]
    100068                   I                                   [irq275: igb4:que 3]
    100066                   I                                   [irq274: igb4:que 2]
    100064                   I                                   [irq273: igb4:que 1]
    100062                   I                                   [irq272: igb4:que 0]
    100061                   I                                   [irq271: igb3:link]
    100059                   I                                   [irq270: igb3:que 3]
    100057                   I                                   [irq269: igb3:que 2]
    100055                   I                                   [irq268: igb3:que 1]
    100053                   I                                   [irq267: igb3:que 0]
    100052                   I                                   [irq266: igb2:link]
    100050                   I                                   [irq265: igb2:que 3]
    100048                   Run     CPU 2                       [irq264: igb2:que 2]
    100046                   L      *pf rules 0xfffff800054b4000 [irq263: igb2:que 1]
    100044                   I                                   [irq262: igb2:que 0]
    100043                   I                                   [irq261: igb1:link]
    100041                   I                                   [irq260: igb1:que 1]
    100039                   I                                   [irq259: igb1:que 0]
    100038                   I                                   [irq258: igb0:link]
    100036                   I                                   [irq257: igb0:que 1]
    100034                   I                                   [irq256: igb0:que 0]
    100033                   I                                   [swi5: fast taskq]
    100031                   I                                   [swi6: Giant taskq]
    100029                   I                                   [swi6: task queue]
    100012                   I                                   [swi4: clock]
    100011                   I                                   [swi4: clock]
    100010                   I                                   [swi4: clock]
    100009                   I                                   [swi4: clock]
    100008                   I                                   [swi1: netisr 0]
    100007                   I                                   [swi3: vm]
       11     0     0     0  RL      (threaded)                  [idle]
    100006                   CanRun                              [idle: cpu3]
    100005                   CanRun                              [idle: cpu2]
    100004                   CanRun                              [idle: cpu1]
    100003                   CanRun                              [idle: cpu0]
        1     0     1     0  SLs     wait     0xfffff8000523e4f0 [init]
       10     0     0     0  DL      audit_wo 0xffffffff82109d60 [audit]
        0     0     0     0  DLs     (threaded)                  [kernel]
    100094                   D       -        0xfffff80005260900 [CAM taskq]
    100089                   D       -        0xfffff800054f2700 [mca taskq]
    100078                   D       -        0xfffff800054c8200 [igb5 que]
    100076                   D       -        0xfffff800054c8500 [igb5 que]
    100074                   D       -        0xfffff800054c8800 [igb5 que]
    100072                   D       -        0xfffff800054c8b00 [igb5 que]
    100069                   D       -        0xfffff800054caa00 [igb4 que]
    100067                   D       -        0xfffff800054cad00 [igb4 que]
    100065                   D       -        0xfffff800054cb100 [igb4 que]
    100063                   D       -        0xfffff800054cb400 [igb4 que]
    100060                   D       -        0xfffff800054a6100 [igb3 que]
    100058                   D       -        0xfffff800054a6400 [igb3 que]
    100056                   D       -        0xfffff800054a6700 [igb3 que]
    100054                   D       -        0xfffff800054a6a00 [igb3 que]
    100051                   D       -        0xfffff8000543d700 [igb2 que]
    100049                   D       -        0xfffff8000543da00 [igb2 que]
    100047                   D       -        0xfffff8000543dd00 [igb2 que]
    100045                   D       -        0xfffff80005480100 [igb2 que]
    100042                   D       -        0xfffff80005482000 [igb1 que]
    100040                   D       -        0xfffff80005482300 [igb1 que]
    100037                   D       -        0xfffff80005262600 [igb0 que]
    100035                   D       -        0xfffff80005262900 [igb0 que]
    100032                   D       -        0xfffff8000525f800 [thread]
    100030                   D       -        0xfffff8000525fd00 [ffs_trim taskq]
    100028                   D       -        0xfffff80005260500 [acpi_task_2]
    100027                   D       -        0xfffff80005260500 [acpi_task_1]
    100026                   D       -        0xfffff80005260500 [acpi_task_0]
    100024                   D       -        0xfffff80005260b00 [kqueue taskq]
    100020                   D       -        0xfffff80005230100 [firmware taskq]
    100000                   D       swapin   0xffffffff820f7af8 [swapper]
    db:0:kdb.enter.default>  alltrace
    
    Tracing command php-fpm pid 47065 tid 100167 td 0xfffff8011594d4b0
    cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe01ed7ebcf0
    ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xfffffe01ed7ebd10
    trap() at trap+0x3a/frame 0xfffffe01ed7ebf20
    nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe01ed7ebf20
    --- trap 0x13, rip = 0x55f269, rsp = 0x7fffffff8220, rbp = 0x7fffffff82f0 ---
    
    Tracing command pfctl pid 46850 tid 100166 td 0xfffff8000d25a960
    m_tag_delete_chain() at m_tag_delete_chain+0x38/frame 0xfffffe0235a1ca30
    uma_zfree_arg() at uma_zfree_arg+0x3e/frame 0xfffffe0235a1caa0
    m_freem() at m_freem+0x63/frame 0xfffffe0235a1cac0
    hfsc_purgeq() at hfsc_purgeq+0x37/frame 0xfffffe0235a1cae0
    hfsc_class_destroy() at hfsc_class_destroy+0x75/frame 0xfffffe0235a1cb00
    hfsc_remove_altq() at hfsc_remove_altq+0x66/frame 0xfffffe0235a1cb30
    pf_commit_altq() at pf_commit_altq+0x221/frame 0xfffffe0235a1cb80
    pfioctl() at pfioctl+0x4e63/frame 0xfffffe0235a1d920
    devfs_ioctl_f() at devfs_ioctl_f+0x139/frame 0xfffffe0235a1d980
    kern_ioctl() at kern_ioctl+0x255/frame 0xfffffe0235a1d9f0
    sys_ioctl() at sys_ioctl+0x15c/frame 0xfffffe0235a1dad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235a1dbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235a1dbf0
    --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800dce6ea, rsp = 0x7fffffffd9b8, rbp = 0x7fffffffd9d0 ---
    
    Tracing command php-fpm pid 45872 tid 100165 td 0xfffff8000d25b000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235a18690
    mi_switch() at mi_switch+0xe1/frame 0xfffffe0235a186d0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235a18750
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235a18780
    _sleep() at _sleep+0x27d/frame 0xfffffe0235a18800
    lf_advlockasync() at lf_advlockasync+0xb4a/frame 0xfffffe0235a188c0
    lf_advlock() at lf_advlock+0x45/frame 0xfffffe0235a18910
    vop_stdadvlock() at vop_stdadvlock+0xc7/frame 0xfffffe0235a18a00
    VOP_ADVLOCK_APV() at VOP_ADVLOCK_APV+0xa7/frame 0xfffffe0235a18a30
    sys_flock() at sys_flock+0x137/frame 0xfffffe0235a18ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235a18bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235a18bf0
    --- syscall (131, FreeBSD ELF64, sys_flock), rip = 0x801ae69ba, rsp = 0x7fffffffab08, rbp = 0x7fffffffab40 ---
    
    Tracing command ntpd pid 39690 tid 100164 td 0xfffff8011594d960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235a13670
    mi_switch() at mi_switch+0xe1/frame 0xfffffe0235a136b0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235a13730
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0235a13760
    _sleep() at _sleep+0x238/frame 0xfffffe0235a137e0
    kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe0235a139c0
    kern_kevent() at kern_kevent+0x9f/frame 0xfffffe0235a13a20
    sys_kevent() at sys_kevent+0x12a/frame 0xfffffe0235a13ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235a13bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235a13bf0
    --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x8015f88fa, rsp = 0x7fffdfffc5c8, rbp = 0x7fffdfffcef0 ---
    
    Tracing command ntpd pid 39690 tid 100160 td 0xfffff80115ac24b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359fd230
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02359fd270
    turnstile_wait() at turnstile_wait+0x42a/frame 0xfffffe02359fd2c0
    __rw_rlock() at __rw_rlock+0x319/frame 0xfffffe02359fd360
    pf_test() at pf_test+0x191/frame 0xfffffe02359fd680
    pf_check_out() at pf_check_out+0x1d/frame 0xfffffe02359fd6a0
    pfil_run_hooks() at pfil_run_hooks+0x8d/frame 0xfffffe02359fd730
    ip_output() at ip_output+0x9c0/frame 0xfffffe02359fd830
    udp_send() at udp_send+0x972/frame 0xfffffe02359fd900
    sosend_dgram() at sosend_dgram+0x406/frame 0xfffffe02359fd980
    kern_sendit() at kern_sendit+0x205/frame 0xfffffe02359fda30
    sendit() at sendit+0x129/frame 0xfffffe02359fda80
    sys_sendto() at sys_sendto+0x4d/frame 0xfffffe02359fdad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359fdbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359fdbf0
    --- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x8015f88ba, rsp = 0x7fffffffddf8, rbp = 0x7fffffffde40 ---
    
    Tracing command dnsmasq pid 38583 tid 100162 td 0xfffff80115ac2000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235a09790
    mi_switch() at mi_switch+0xe1/frame 0xfffffe0235a097d0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235a09850
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235a09880
    _cv_wait_sig() at _cv_wait_sig+0x17a/frame 0xfffffe0235a098d0
    seltdwait() at seltdwait+0xae/frame 0xfffffe0235a09910
    kern_poll() at kern_poll+0x464/frame 0xfffffe0235a09ab0
    sys_poll() at sys_poll+0x61/frame 0xfffffe0235a09ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235a09bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235a09bf0
    --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x801463efa, rsp = 0x7fffffffeb08, rbp = 0x7fffffffec10 ---
    
    Tracing command cron pid 34765 tid 100154 td 0xfffff8011548b000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359df8c0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02359df900
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359df980
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe02359df9b0
    _sleep() at _sleep+0x238/frame 0xfffffe02359dfa30
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe02359dfa90
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe02359dfad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359dfbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359dfbf0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800d19f3a, rsp = 0x7fffffffea88, rbp = 0x7fffffffecb0 ---
    
    Tracing command nginx pid 33967 tid 100110 td 0xfffff80005f804b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235903670
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02359036b0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235903730
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0235903760
    _sleep() at _sleep+0x238/frame 0xfffffe02359037e0
    kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe02359039c0
    kern_kevent() at kern_kevent+0x9f/frame 0xfffffe0235903a20
    sys_kevent() at sys_kevent+0x12a/frame 0xfffffe0235903ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235903bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235903bf0
    --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x8018fd8fa, rsp = 0x7fffffffe768, rbp = 0x7fffffffe7b0 ---
    
    Tracing command nginx pid 33716 tid 100156 td 0xfffff80115a9e960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359e9670
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02359e96b0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359e9730
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359e9760
    _sleep() at _sleep+0x27d/frame 0xfffffe02359e97e0
    kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe02359e99c0
    kern_kevent() at kern_kevent+0x9f/frame 0xfffffe02359e9a20
    sys_kevent() at sys_kevent+0x12a/frame 0xfffffe02359e9ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359e9bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359e9bf0
    --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x8018fd8fa, rsp = 0x7fffffffe768, rbp = 0x7fffffffe7b0 ---
    
    Tracing command nginx pid 33473 tid 100155 td 0xfffff8011594e960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359e48d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02359e4910
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359e4990
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359e49c0
    _sleep() at _sleep+0x27d/frame 0xfffffe02359e4a40
    kern_sigsuspend() at kern_sigsuspend+0xf4/frame 0xfffffe02359e4aa0
    sys_sigsuspend() at sys_sigsuspend+0x31/frame 0xfffffe02359e4ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359e4bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359e4bf0
    --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x8018f2dfa, rsp = 0x7fffffffe908, rbp = 0x7fffffffe940 ---
    
    Tracing command php-fpm pid 23430 tid 100114 td 0xfffff8000d042960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235917690
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02359176d0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235917750
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235917780
    _sleep() at _sleep+0x27d/frame 0xfffffe0235917800
    lf_advlockasync() at lf_advlockasync+0xb4a/frame 0xfffffe02359178c0
    lf_advlock() at lf_advlock+0x45/frame 0xfffffe0235917910
    vop_stdadvlock() at vop_stdadvlock+0xc7/frame 0xfffffe0235917a00
    VOP_ADVLOCK_APV() at VOP_ADVLOCK_APV+0xa7/frame 0xfffffe0235917a30
    sys_flock() at sys_flock+0x137/frame 0xfffffe0235917ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235917bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235917bf0
    --- syscall (131, FreeBSD ELF64, sys_flock), rip = 0x801ae69ba, rsp = 0x7fffffffab08, rbp = 0x7fffffffab40 ---
    
    Tracing command dpinger pid 18600 tid 100153 td 0xfffff8011548b4b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359da850
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02359da890
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359da910
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359da940
    _sleep() at _sleep+0x27d/frame 0xfffffe02359da9c0
    kern_accept4() at kern_accept4+0x1b1/frame 0xfffffe02359daa70
    accept1() at accept1+0x92/frame 0xfffffe02359daad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359dabf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359dabf0
    --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x800b1bfda, rsp = 0x7fffdf9fae78, rbp = 0x7fffdf9faeb0 ---
    
    Tracing command dpinger pid 18600 tid 100152 td 0xfffff8011548b960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359d58c0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02359d5900
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359d5980
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe02359d59b0
    _sleep() at _sleep+0x238/frame 0xfffffe02359d5a30
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe02359d5a90
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe02359d5ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359d5bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359d5bf0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b1bf3a, rsp = 0x7fffdfbfbef8, rbp = 0x7fffdfbfbf20 ---
    
    Tracing command dpinger pid 18600 tid 100151 td 0xfffff80115904000
    cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff820669e0
    ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xffffffff82066a00
    trap() at trap+0x3a/frame 0xffffffff82066c10
    nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff82066c10
    --- trap 0x13, rip = 0xffffffff80b090b2, rsp = 0xfffffe02359d0300, rbp = 0xfffffe02359d0390 ---
    __rw_rlock() at __rw_rlock+0x1f2/frame 0xfffffe02359d0390
    pf_test() at pf_test+0x191/frame 0xfffffe02359d06b0
    pf_check_out() at pf_check_out+0x1d/frame 0xfffffe02359d06d0
    pfil_run_hooks() at pfil_run_hooks+0x8d/frame 0xfffffe02359d0760
    ip_output() at ip_output+0x9c0/frame 0xfffffe02359d0860
    rip_output() at rip_output+0x2cd/frame 0xfffffe02359d08b0
    sosend_generic() at sosend_generic+0x476/frame 0xfffffe02359d0980
    kern_sendit() at kern_sendit+0x205/frame 0xfffffe02359d0a30
    sendit() at sendit+0x129/frame 0xfffffe02359d0a80
    sys_sendto() at sys_sendto+0x4d/frame 0xfffffe02359d0ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359d0bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359d0bf0
    --- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x800b268ba, rsp = 0x7fffdfdfcf38, rbp = 0x7fffdfdfcf80 ---
    
    Tracing command dpinger pid 18600 tid 100150 td 0xfffff801159044b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359cb740
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02359cb780
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359cb800
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359cb830
    _sleep() at _sleep+0x27d/frame 0xfffffe02359cb8b0
    sbwait() at sbwait+0x4f/frame 0xfffffe02359cb8d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe02359cb9a0
    kern_recvit() at kern_recvit+0x1f2/frame 0xfffffe02359cba60
    sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe02359cbad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359cbbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359cbbf0
    --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b268da, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 ---
    
    Tracing command dpinger pid 18600 tid 100128 td 0xfffff8000d1e5000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe023595d840
    mi_switch() at mi_switch+0xe1/frame 0xfffffe023595d880
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023595d900
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe023595d930
    _sleep() at _sleep+0x27d/frame 0xfffffe023595d9b0
    umtxq_sleep() at umtxq_sleep+0x125/frame 0xfffffe023595da10
    do_wait() at do_wait+0x387/frame 0xfffffe023595da90
    __umtx_op_wait() at __umtx_op_wait+0x7d/frame 0xfffffe023595dad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023595dbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe023595dbf0
    --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083813c, rsp = 0x7fffffffe798, rbp = 0x7fffffffe810 ---
    
    Tracing command dpinger pid 18225 tid 100149 td 0xfffff80115904960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359c6850
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02359c6890
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359c6910
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359c6940
    _sleep() at _sleep+0x27d/frame 0xfffffe02359c69c0
    kern_accept4() at kern_accept4+0x1b1/frame 0xfffffe02359c6a70
    accept1() at accept1+0x92/frame 0xfffffe02359c6ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359c6bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359c6bf0
    --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x800b1bfda, rsp = 0x7fffdf9fae78, rbp = 0x7fffdf9faeb0 ---
    
    Tracing command dpinger pid 18225 tid 100148 td 0xfffff80115905000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359c18c0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02359c1900
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359c1980
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe02359c19b0
    _sleep() at _sleep+0x238/frame 0xfffffe02359c1a30
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe02359c1a90
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe02359c1ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359c1bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359c1bf0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b1bf3a, rsp = 0x7fffdfbfbef8, rbp = 0x7fffdfbfbf20 ---
    
    Tracing command dpinger pid 18225 tid 100147 td 0xfffff801159054b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359bc260
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02359bc2a0
    turnstile_wait() at turnstile_wait+0x42a/frame 0xfffffe02359bc2f0
    __rw_rlock() at __rw_rlock+0x319/frame 0xfffffe02359bc390
    pf_test() at pf_test+0x191/frame 0xfffffe02359bc6b0
    pf_check_out() at pf_check_out+0x1d/frame 0xfffffe02359bc6d0
    pfil_run_hooks() at pfil_run_hooks+0x8d/frame 0xfffffe02359bc760
    ip_output() at ip_output+0x9c0/frame 0xfffffe02359bc860
    rip_output() at rip_output+0x2cd/frame 0xfffffe02359bc8b0
    sosend_generic() at sosend_generic+0x476/frame 0xfffffe02359bc980
    kern_sendit() at kern_sendit+0x205/frame 0xfffffe02359bca30
    sendit() at sendit+0x129/frame 0xfffffe02359bca80
    sys_sendto() at sys_sendto+0x4d/frame 0xfffffe02359bcad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359bcbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359bcbf0
    --- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x800b268ba, rsp = 0x7fffdfdfcf38, rbp = 0x7fffdfdfcf80 ---
    
    Tracing command dpinger pid 18225 tid 100146 td 0xfffff80115905960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359b7740
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02359b7780
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359b7800
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359b7830
    _sleep() at _sleep+0x27d/frame 0xfffffe02359b78b0
    sbwait() at sbwait+0x4f/frame 0xfffffe02359b78d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe02359b79a0
    kern_recvit() at kern_recvit+0x1f2/frame 0xfffffe02359b7a60
    sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe02359b7ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359b7bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359b7bf0
    --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b268da, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 ---
    
    Tracing command dpinger pid 18225 tid 100135 td 0xfffff8000d1f8000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235980840
    mi_switch() at mi_switch+0xe1/frame 0xfffffe0235980880
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235980900
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235980930
    _sleep() at _sleep+0x27d/frame 0xfffffe02359809b0
    umtxq_sleep() at umtxq_sleep+0x125/frame 0xfffffe0235980a10
    do_wait() at do_wait+0x387/frame 0xfffffe0235980a90
    __umtx_op_wait() at __umtx_op_wait+0x7d/frame 0xfffffe0235980ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235980bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235980bf0
    --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083813c, rsp = 0x7fffffffe798, rbp = 0x7fffffffe810 ---
    
    Tracing command dpinger pid 17658 tid 100145 td 0xfffff8000d48a4b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359b2850
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02359b2890
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359b2910
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359b2940
    _sleep() at _sleep+0x27d/frame 0xfffffe02359b29c0
    kern_accept4() at kern_accept4+0x1b1/frame 0xfffffe02359b2a70
    accept1() at accept1+0x92/frame 0xfffffe02359b2ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359b2bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359b2bf0
    --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x800b1bfda, rsp = 0x7fffdf9fae78, rbp = 0x7fffdf9faeb0 ---
    
    Tracing command dpinger pid 17658 tid 100144 td 0xfffff8000d48a960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359ad8c0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02359ad900
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359ad980
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe02359ad9b0
    _sleep() at _sleep+0x238/frame 0xfffffe02359ada30
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe02359ada90
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe02359adad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359adbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359adbf0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b1bf3a, rsp = 0x7fffdfbfbef8, rbp = 0x7fffdfbfbf20 ---
    
    Tracing command dpinger pid 17658 tid 100143 td 0xfffff8000d48b000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359a8260
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02359a82a0
    turnstile_wait() at turnstile_wait+0x42a/frame 0xfffffe02359a82f0
    __rw_rlock() at __rw_rlock+0x319/frame 0xfffffe02359a8390
    pf_test() at pf_test+0x191/frame 0xfffffe02359a86b0
    pf_check_out() at pf_check_out+0x1d/frame 0xfffffe02359a86d0
    pfil_run_hooks() at pfil_run_hooks+0x8d/frame 0xfffffe02359a8760
    ip_output() at ip_output+0x9c0/frame 0xfffffe02359a8860
    rip_output() at rip_output+0x2cd/frame 0xfffffe02359a88b0
    sosend_generic() at sosend_generic+0x476/frame 0xfffffe02359a8980
    kern_sendit() at kern_sendit+0x205/frame 0xfffffe02359a8a30
    sendit() at sendit+0x129/frame 0xfffffe02359a8a80
    sys_sendto() at sys_sendto+0x4d/frame 0xfffffe02359a8ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359a8bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359a8bf0
    --- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x800b268ba, rsp = 0x7fffdfdfcf38, rbp = 0x7fffdfdfcf80 ---
    
    Tracing command dpinger pid 17658 tid 100142 td 0xfffff8000d48b4b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359a3740
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02359a3780
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359a3800
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359a3830
    _sleep() at _sleep+0x27d/frame 0xfffffe02359a38b0
    sbwait() at sbwait+0x4f/frame 0xfffffe02359a38d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe02359a39a0
    kern_recvit() at kern_recvit+0x1f2/frame 0xfffffe02359a3a60
    sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe02359a3ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359a3bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359a3bf0
    --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b268da, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 ---
    
    Tracing command dpinger pid 17658 tid 100137 td 0xfffff8000d4aa4b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe023598a840
    mi_switch() at mi_switch+0xe1/frame 0xfffffe023598a880
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023598a900
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe023598a930
    _sleep() at _sleep+0x27d/frame 0xfffffe023598a9b0
    umtxq_sleep() at umtxq_sleep+0x125/frame 0xfffffe023598aa10
    do_wait() at do_wait+0x387/frame 0xfffffe023598aa90
    __umtx_op_wait() at __umtx_op_wait+0x7d/frame 0xfffffe023598aad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023598abf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe023598abf0
    --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083813c, rsp = 0x7fffffffe7a8, rbp = 0x7fffffffe820 ---
    
    Tracing command xinetd pid 15517 tid 100134 td 0xfffff8000d1f84b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe023597b710
    mi_switch() at mi_switch+0xe1/frame 0xfffffe023597b750
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023597b7d0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe023597b800
    _cv_wait_sig() at _cv_wait_sig+0x17a/frame 0xfffffe023597b850
    seltdwait() at seltdwait+0xae/frame 0xfffffe023597b890
    kern_select() at kern_select+0x8fa/frame 0xfffffe023597ba90
    sys_select() at sys_select+0x54/frame 0xfffffe023597bad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023597bbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe023597bbf0
    --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800fca66a, rsp = 0x7fffffffeb08, rbp = 0x7fffffffec40 ---
    
    Tracing command filterlog pid 15124 tid 100140 td 0xfffff8000d03a4b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359997c0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe0235999800
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235999880
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe02359998b0
    _sleep() at _sleep+0x238/frame 0xfffffe0235999930
    bpfread() at bpfread+0x1c7/frame 0xfffffe0235999980
    devfs_read_f() at devfs_read_f+0xd8/frame 0xfffffe02359999e0
    dofileread() at dofileread+0x95/frame 0xfffffe0235999a30
    kern_readv() at kern_readv+0x68/frame 0xfffffe0235999a80
    sys_read() at sys_read+0x63/frame 0xfffffe0235999ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235999bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235999bf0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800da168a, rsp = 0x7fffffffeac8, rbp = 0x7fffffffec00 ---
    
    Tracing command sshlockout_pf pid 9174 tid 100141 td 0xfffff8000d03a000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe023599e8c0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe023599e900
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023599e980
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe023599e9b0
    _sleep() at _sleep+0x238/frame 0xfffffe023599ea30
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe023599ea90
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe023599ead0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023599ebf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe023599ebf0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b18f3a, rsp = 0x7fffdfffdf48, rbp = 0x7fffdfffdf70 ---
    
    Tracing command sshlockout_pf pid 9174 tid 100108 td 0xfffff80005fa94b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe02358f9810
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02358f9850
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02358f98d0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02358f9900
    _sleep() at _sleep+0x27d/frame 0xfffffe02358f9980
    pipe_read() at pipe_read+0x2aa/frame 0xfffffe02358f99e0
    dofileread() at dofileread+0x95/frame 0xfffffe02358f9a30
    kern_readv() at kern_readv+0x68/frame 0xfffffe02358f9a80
    sys_read() at sys_read+0x63/frame 0xfffffe02358f9ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02358f9bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02358f9bf0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b7b68a, rsp = 0x7fffffffe7d8, rbp = 0x7fffffffe7f0 ---
    
    Tracing command sshd pid 9167 tid 100132 td 0xfffff8000d1e4000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235971710
    mi_switch() at mi_switch+0xe1/frame 0xfffffe0235971750
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359717d0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235971800
    _cv_wait_sig() at _cv_wait_sig+0x17a/frame 0xfffffe0235971850
    seltdwait() at seltdwait+0xae/frame 0xfffffe0235971890
    kern_select() at kern_select+0x8fa/frame 0xfffffe0235971a90
    sys_select() at sys_select+0x54/frame 0xfffffe0235971ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235971bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235971bf0
    --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80327966a, rsp = 0x7fffffffe228, rbp = 0x7fffffffe260 ---
    
    Tracing command syslogd pid 3699 tid 100139 td 0xfffff8000d48b960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235994710
    mi_switch() at mi_switch+0xe1/frame 0xfffffe0235994750
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359947d0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235994800
    _cv_wait_sig() at _cv_wait_sig+0x17a/frame 0xfffffe0235994850
    seltdwait() at seltdwait+0xae/frame 0xfffffe0235994890
    kern_select() at kern_select+0x8fa/frame 0xfffffe0235994a90
    sys_select() at sys_select+0x54/frame 0xfffffe0235994ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235994bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235994bf0
    --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800b7166a, rsp = 0x7fffffffdc08, rbp = 0x7fffffffec20 ---
    
    Tracing command devd pid 332 tid 100123 td 0xfffff80005f97000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235944700
    mi_switch() at mi_switch+0xe1/frame 0xfffffe0235944740
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359447c0
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe02359447f0
    _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x19e/frame 0xfffffe0235944850
    seltdwait() at seltdwait+0xa4/frame 0xfffffe0235944890
    kern_select() at kern_select+0x8fa/frame 0xfffffe0235944a90
    sys_select() at sys_select+0x54/frame 0xfffffe0235944ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235944bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235944bf0
    --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x4a8caa, rsp = 0x7fffffffcb48, rbp = 0x7fffffffecd0 ---
    
    Tracing command php-cgi pid 322 tid 100124 td 0xfffff8000d1e54b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235949810
    mi_switch() at mi_switch+0xe1/frame 0xfffffe0235949850
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359498d0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235949900
    _sleep() at _sleep+0x27d/frame 0xfffffe0235949980
    pipe_read() at pipe_read+0x2aa/frame 0xfffffe02359499e0
    dofileread() at dofileread+0x95/frame 0xfffffe0235949a30
    kern_readv() at kern_readv+0x68/frame 0xfffffe0235949a80
    sys_read() at sys_read+0x63/frame 0xfffffe0235949ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235949bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235949bf0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x801b9968a, rsp = 0x7fffffffcb48, rbp = 0x7fffffffcb80 ---
    
    Tracing command check_reload_status pid 320 tid 100125 td 0xfffff8000d25b960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe023594e670
    mi_switch() at mi_switch+0xe1/frame 0xfffffe023594e6b0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023594e730
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe023594e760
    _sleep() at _sleep+0x27d/frame 0xfffffe023594e7e0
    kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe023594e9c0
    kern_kevent() at kern_kevent+0x9f/frame 0xfffffe023594ea20
    sys_kevent() at sys_kevent+0x12a/frame 0xfffffe023594ead0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023594ebf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe023594ebf0
    --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x800d478fa, rsp = 0x7fffffffe9e8, rbp = 0x7fffffffea30 ---
    
    Tracing command check_reload_status pid 318 tid 100129 td 0xfffff8000d03a960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235962670
    mi_switch() at mi_switch+0xe1/frame 0xfffffe02359626b0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235962730
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0235962760
    _sleep() at _sleep+0x238/frame 0xfffffe02359627e0
    kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe02359629c0
    kern_kevent() at kern_kevent+0x9f/frame 0xfffffe0235962a20
    sys_kevent() at sys_kevent+0x12a/frame 0xfffffe0235962ad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235962bf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235962bf0
    --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x800d478fa, rsp = 0x7fffffffe8a8, rbp = 0x7fffffffe8f0 ---
    
    Tracing command php-fpm pid 281 tid 100122 td 0xfffff80005f974b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe023593f8c0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe023593f900
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023593f980
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe023593f9b0
    _sleep() at _sleep+0x238/frame 0xfffffe023593fa30
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe023593fa90
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe023593fad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023593fbf0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe023593fbf0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x801b4af3a, rsp = 0x7fffffffae78, rbp = 0x7fffffffaea0 ---
    
    Tracing command php-fpm pid 280 tid 100118 td 0xfffff80005fa8960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe023592b670
    mi_switch() at mi_switch+0xe1/frame 0xfffffe023592b6b0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023592b730
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe023592b760
    _sleep() at _sleep+0x238/frame 0xfffffe023592b7e0
    kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe023592b9c0
    kern_kevent() at kern_kevent+0x9f/frame 0xfffffe023592ba20
    sys_kevent() at sys_kevent+0x12a/frame 0xfffffe023592bad0
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023592b[/thread]
    

  • Rebel Alliance Developer Netgate

    Seems like it might be https://redmine.pfsense.org/issues/6257 but it's not quite an exact match for the ones we saw hitting that previously.



  • Ironic, I manually rebuilt the config and when I added the Shaper it crashed.  the 2.2.6 Shaper must not be compatible in 2.3.

    Thanks!!