SG-4860 "Configuring firewall…." on boot error
-
Just rebuilt my pfsense SG-4860, had to reload the OS which went smoothly, manually modified/updated the interface types (em# -> igb#), and rebooted it at least 4 times.
I went to deploy it this morning and some of the servers on vLAN2 and 3 couldn't be reached. Logged into the firewall GUI and noticed that the vlan assignments were incorrect. I updated the interface assignments again, saved, and pretty much saw my entire network go down. When i tried to access the console via the serial connection it connected but would load anything. Failed back over to the old equipment then power cycled the SG. This time the console came alive as it booted up. That's when this error showed up and the boot up stalled.
SeaBIOS (version SageBIOS-20150314_140038-localhost.localdomain) Found mainboard Intel Mohon Peak Relocating init from 0x000ef1b0 to 0x7fb995f0 (size 26960) Found CBFS header at 0xfffffbe0 CPU Mhz=2401 Found 20 PCI devices (max PCI bus is 04) Copying MPTABLE from 0x7fbe3e00/7fbe3e10 to 0x000f5950 with length 1ac Copying ACPI RSDP from 0x7fbe4e00 to 0x000f5920 Copying SMBIOS entry point from 0x7fbf0400 to 0x000f5900 Using pmtimer, ioport 0x408 Scan for VGA option rom EHCI init on dev 00:16.0 (regs=0xdff35420) Found 2 serial ports ebda moved from 9f000 to 9e800 AHCI controller at 17.0, iobase dff34000, irq 15 AHCI controller at 18.0, iobase dff34800, irq 10 Searching bootorder for: /pci@i0cf8/usb@16/hub@1/storage@4/*@0/*@0,0 Searching bootorder for: /pci@i0cf8/usb@16/hub@1/usb-*@4 USB MSC vendor='Generic' product='Ultra HS-COMBO' rev='1.98' type=0 removable=1 USB MSC blksize=512 sectors=7733248 Initialized USB HUB (1 ports used) All threads complete. Scan for option roms Running option rom at c000:0003 pmm call arg1=1 pmm call arg1=0 pmm call arg1=1 pmm call arg1=0 Running option rom at c100:0003 pmm call arg1=1 pmm call arg1=1 Running option rom at c200:0003 pmm call arg1=1 pmm call arg1=1 Running option rom at c300:0003 pmm call arg1=1 pmm call arg1=1 Searching bootorder for: /pci@i0cf8/*@14 Searching bootorder for: /pci@i0cf8/*@14,1 Searching bootorder for: /pci@i0cf8/*@14,2 Searching bootorder for: /pci@i0cf8/*@14,3 Press F12 for boot menu. Searching bootorder for: HALT drive 0x000f5890: PCHS=0/0/0 translation=lba LCHS=958/128/63 s=7733248 Space available for UMB: c4000-f0000, f0000-f5890 Returned 253952 bytes of ZoneHigh e820 map has 8 items: 0: 0000000000000000 - 000000000009e800 = 1 RAM 1: 000000000009e800 - 00000000000a0000 = 2 RESERVED 2: 00000000000f0000 - 0000000000100000 = 2 RESERVED 3: 0000000000100000 - 000000007fbde000 = 1 RAM 4: 000000007fbde000 - 000000007fe00000 = 2 RESERVED 5: 00000000e0000000 - 00000000f0000000 = 2 RESERVED 6: 00000000fee00000 - 00000000fee01000 = 2 RESERVED 7: 0000000100000000 - 0000000280000000 = 1 RAM enter handle_19: NULL Booting from Hard Disk... Booting from 0000:7c00 - __ ____ _ __ / _/ ___| ___ _ __ ___ ___ | '_ \| |_\___ \ / _ \ '_ \/ __|/ _ \ | |_) | _|___) | __/ | | \__ \ __/ | .__/|_| |____/ \___|_| |_|___/\___| |_| +============Welcome to pfSense===========+ | | ______ | 1\. Boot Multi User [Enter] | / \ | 2\. Boot [s]ingle User | _____/ f \ | 3\. [Esc]ape to loader prompt | / \ / | 4\. Reboot | / p \______/ Sense | | \ / \ | Options: | \_____/ \ | 5\. [K]ernel: kernel (1 of 2) | \ / | 6\. Configure Boot [O]ptions... | \______/ | | | | | | +=========================================+ /boot/kernel/kernel text=0x1292d98 data=0x8a1bb0+0x23ec10 syms=[0x8+0x177720+0x8+0x173eef] Booting... KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2016 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense amd64 FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 CPU: Intel(R) Atom(TM) CPU C2558 @ 2.40GHz (2400.06-MHz K8-class CPU) Origin="GenuineIntel" Id=0x406d8 Family=0x6 Model=0x4d Stepping=8 Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x43d8e3bf <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand>AMD Features=0x28100800 <syscall,nx,rdtscp,lm>AMD Features2=0x101 <lahf,prefetch>Structured Extended Features=0x2282 <tscadj,smep,erms,nfpusg>VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 10737418240 (10240 MB) avail memory = 8220663808 (7839 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <core coreboot=""> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 2 cpu2 (AP): APIC ID: 4 cpu3 (AP): APIC ID: 6 random: <software, yarrow=""> initialized ioapic0 <version 2.0=""> irqs 0-23 on motherboard wlan: mac acl policy registered ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/. ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff806208f0, 0) error 1 ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/. ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff806209a0, 0) error 1 ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/. ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in/boot/loader.conf. module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80620a50, 0) error 1 iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/. iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80647bf0, 0) error 1 iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/. iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80647ca0, 0) error 1 iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/. iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in/boot/loader.conf. module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80647d50, 0) error 1 netmap: loaded module kbd0 at kbdmux0 module_register_init: MOD_LOAD (vesa, 0xffffffff81016b90, 0) error 19 cryptosoft0: <software crypto=""> on motherboard padlock0: No ACE support. acpi0: <core coreboot=""> on motherboard acpi0: Power Button (fixed) hpet0: invalid period device_attach: hpet0 attach returned 6 cpu0: <acpi cpu=""> on acpi0 cpu1: <acpi cpu=""> on acpi0 cpu2: <acpi cpu=""> on acpi0 cpu3: <acpi cpu=""> on acpi0 hpet0: invalid period device_attach: hpet0 attach returned 6 atrtc0: <at realtime="" clock=""> port 0x70-0x77 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: <at timer=""> port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: <acpi host-pci="" bridge=""> port 0xcf8-0xcff on acpi0 pci0: <acpi pci="" bus=""> on pcib0 pcib1: <acpi pci-pci="" bridge=""> mem 0xdfe00000-0xdfe1ffff irq 16 at device 1.0 on pci0 pci1: <acpi pci="" bus=""> on pcib1 pcib2: <acpi pci-pci="" bridge=""> mem 0xdfe20000-0xdfe3ffff irq 19 at device 2.0 on pci0 pci2: <acpi pci="" bus=""> on pcib2 pcib3: <acpi pci-pci="" bridge=""> mem 0xdfe40000-0xdfe5ffff irq 20 at device 3.0 on pci0 pci3: <acpi pci="" bus=""> on pcib3 igb0: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x1000-0x101f mem 0xdfc00000-0xdfc1ffff,0xdfc20000-0xdfc23fff irq 18 at device 0.0 on pci3 igb0: Using MSIX interrupts with 3 vectors igb0: Ethernet address: 00:08:a2:09:36:f0 igb0: Bound queue 0 to cpu 0 igb0: Bound queue 1 to cpu 1 igb0: netmap queues/slots: TX 2/1024, RX 2/1024 pcib4: <acpi pci-pci="" bridge=""> mem 0xdfe60000-0xdfe7ffff irq 23 at device 4.0 on pci0 pci4: <acpi pci="" bus=""> on pcib4 igb1: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x2000-0x201f mem 0xdfd00000-0xdfd1ffff,0xdfd20000-0xdfd23fff irq 19 at device 0.0 on pci4 igb1: Using MSIX interrupts with 3 vectors igb1: Ethernet address: 00:08:a2:09:36:f1 igb1: Bound queue 0 to cpu 2 igb1: Bound queue 1 to cpu 3 igb1: netmap queues/slots: TX 2/1024, RX 2/1024 pci0: <processor> at device 11.0 (no driver attached) pci0: <base peripheral,="" iommu=""> at device 15.0 (no driver attached) igb2: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x3000-0x301f mem 0xdfea0000-0xdfebffff,0xdff24000-0xdff27fff irq 18 at device 20.0 on pci0 igb2: Using MSIX interrupts with 5 vectors igb2: Ethernet address: 00:08:a2:09:36:ec igb2: Bound queue 0 to cpu 0 igb2: Bound queue 1 to cpu 1 igb2: Bound queue 2 to cpu 2 igb2: Bound queue 3 to cpu 3 igb2: netmap queues/slots: TX 4/1024, RX 4/1024 igb3: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x3020-0x303f mem 0xdfec0000-0xdfedffff,0xdff28000-0xdff2bfff irq 19 at device 20.1 on pci0 igb3: Using MSIX interrupts with 5 vectors igb3: Ethernet address: 00:08:a2:09:36:ed igb3: Bound queue 0 to cpu 0 igb3: Bound queue 1 to cpu 1 igb3: Bound queue 2 to cpu 2 igb3: Bound queue 3 to cpu 3 igb3: netmap queues/slots: TX 4/1024, RX 4/1024 igb4: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x3040-0x305f mem 0xdfee0000-0xdfefffff,0xdff2c000-0xdff2ffff irq 20 at device 20.2 on pci0 igb4: Using MSIX interrupts with 5 vectors igb4: Ethernet address: 00:08:a2:09:36:ee igb4: Bound queue 0 to cpu 0 igb4: Bound queue 1 to cpu 1 igb4: Bound queue 2 to cpu 2 igb4: Bound queue 3 to cpu 3 igb4: netmap queues/slots: TX 4/1024, RX 4/1024 igb5: <intel(r) 1000="" pro="" network="" connection,="" version="" -="" 2.5.3-k=""> port 0x3060-0x307f mem 0xdff00000-0xdff1ffff,0xdff30000-0xdff33fff irq 21 at device 20.3 on pci0 igb5: Using MSIX interrupts with 5 vectors igb5: Ethernet address: 00:08:a2:09:36:ef igb5: Bound queue 0 to cpu 0 igb5: Bound queue 1 to cpu 1 igb5: Bound queue 2 to cpu 2 igb5: Bound queue 3 to cpu 3 igb5: netmap queues/slots: TX 4/1024, RX 4/1024 ehci0: <intel avoton="" usb="" 2.0="" controller=""> mem 0xdff35400-0xdff357ff irq 22 at device 22.0 on pci0 usbus0: EHCI version 1.0 usbus0 on ehci0 ahci0: <intel avoton="" ahci="" sata="" controller=""> port 0x30c0-0x30c7,0x30e0-0x30e3,0x30c8-0x30cf,0x30e4-0x30e7,0x3080-0x309f mem 0xdff34000-0xdff347ff irq 23 at device 23.0 on pci0 ahci0: AHCI v1.30 with 4 3Gbps ports, Port Multiplier not supported ahci1: <intel avoton="" ahci="" sata="" controller=""> port 0x30d0-0x30d7,0x30e8-0x30eb,0x30d8-0x30df,0x30ec-0x30ef,0x30a0-0x30bf mem 0xdff34800-0xdff34fff irq 16 at device 24.0 on pci0 ahci1: AHCI v1.30 with 2 6Gbps ports, Port Multiplier not supported isab0: <pci-isa bridge=""> at device 31.0 on pci0 isa0: <isa bus=""> on isab0 ppc0: cannot reserve I/O port range uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 4 on isa0 uart1: <16550 or compatible> at port 0x2f8-0x2ff irq 3 flags 0x10 on isa0 uart1: console (115200,n,8,1) est0: <enhanced speedstep="" frequency="" control=""> on cpu0 est1: <enhanced speedstep="" frequency="" control=""> on cpu1 est2: <enhanced speedstep="" frequency="" control=""> on cpu2 est3: <enhanced speedstep="" frequency="" control=""> on cpu3 Timecounters tick every 1.000 msec random: unblocking device. usbus0: 480Mbps High Speed USB v2.0 ugen0.1: <intel> at usbus0 uhub0: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr=""> on usbus0 uhub0: 8 ports with 8 removable, self powered ugen0.2: <vendor 0x8087=""> at usbus0 uhub1: <vendor 2="" 9="" 0x8087="" product="" 0x07db,="" class="" 0,="" rev="" 2.00="" 0.02,="" addr=""> on usbus0 uhub1: 4 ports with 4 removable, self powered ugen0.3: <generic> at usbus0 umass0: <generic 0="" 3="" ultra="" fast="" media,="" class="" 0,="" rev="" 2.00="" 1.98,="" addr=""> on usbus0 da0 at umass-sim0 bus 0 scbus0 target 0 lun 0 da0: <generic ultra="" hs-combo="" 1.98=""> Removable Direct Access SCSI device da0: Serial Number 000000225001 da0: 40.000MB/s transfers da0: 3776MB (7733248 512 byte sectors) da0: quirks=0x2 <no_6_byte>SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! Timecounter "TSC-low" frequency 1200029904 Hz quality 1000 Trying to mount root from ufs:/dev/ufsid/58236354d4349f25 [rw]... WARNING: / was not properly dismounted WARNING: /: mount pending error: blocks 1176 files 32 Configuring crash dumps... Using /dev/label/swap0 for dump device. ** SU+J Recovering /dev/ufsid/58236354d4349f25 ** Reading 23199744 byte journal from inode 4. ** Building recovery table. ** Resolving unreferenced inode list. ** Processing journal entries. ** 665 journal records in 82432 bytes for 25.82% utilization ** Freed 69 inodes (2 dirs) 23 blocks, and 88 frags. ***** FILE SYSTEM MARKED CLEAN ***** Filesystems are clean, continuing... Mounting filesystems... ___ ___/ f \ / p \___/ Sense \___/ \ \___/ Welcome to pfSense 2.3.2-RELEASE on the 'pfSense' platform... No core dumps found. Creating symlinks......ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec/usr/local/lib/perl5/5.20/mach/CORE 32-bit compatibility ldconfig path: /usr/lib32 done. External config loader 1.0 is now starting... da0s1 da0s1a da0s1b Launching the init system....... done. Initializing.................. done. Starting device manager (devd)...done. Loading configuration......done. Updating configuration...done. Cleaning backup cache...............done. Setting up extended sysctls...done. Setting timezone...done. Configuring loopback interface...done. Starting syslog...done. Starting Secure Shell Services...done. Setting up polling defaults...done. Setting up interfaces microcode...done. Configuring loopback interface...done. Creating wireless clone interfaces...done. Configuring LAGG interfaces...done. Configuring VLAN interfaces...done. Configuring QinQ interfaces...done. Configuring DMZ interface...done. Configuring LAN interface...done. Configuring OPT1VLAN2VOIP interface...done. Configuring OPT2VLAN3CAMERAS interface...done. Configuring SYNC interface...done. Configuring OPT5VLAN5MANAGMENTVLAN interface...done. Configuring OPT4VLAN4ESXISAN interface...done. Configuring CONTROL_RM_VLAN10 interface...done. Configuring CARP settings...done. Configuring CARP settings...done. Syncing OpenVPN settings...done. Configuring firewall.....0 addresses deleted. .done. Starting PFLOG...done. Setting up gateway monitors...done. Synchronizing user settings...done. Starting webConfigurator...done. Configuring CRON...done. Starting DNS forwarder...done. Starting NTP time client...done. Starting DHCP service...done. Configuring firewall.....g_vfs_done():md0[WRITE(offset=270336, length=4096)]error = 28 Any idea what this means?[/s]</no_6_byte></generic></generic></generic></vendor></vendor></intel></intel></enhanced></enhanced></enhanced></enhanced></isa></pci-isa></intel></intel></intel></intel(r)></intel(r)></intel(r)></intel(r)></processor></intel(r)></acpi></acpi></intel(r)></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></at></at></acpi></acpi></acpi></acpi></core></software></version></software,></core ></tscadj,smep,erms,nfpusg></lahf,prefetch></syscall,nx,rdtscp,lm></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,sse4.1,sse4.2,movbe,popcnt,tscdlt,aesni,rdrand></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe> -
g_vfs_done():md0[WRITE(offset=270336, length=4096)]error = 28That is saying it can't write to a RAM disk (md0). That is a fairly certain indicator that system has somehow run out of RAM, or at least kernel memory.
I don't think I've ever seen that happen on one of those units before, what sorts of things were in the config you restored? Which packages?
-
thanks for responding!
This is from the current production device (2.2.6) that will be disco'd after the 2.3.2 unit is built.
Package Name Category Package Version
File Manager Utility 0.2.2
FTP Client Proxy Services 0.3
NRPE v2 Network Management 2.2.7
ntopng Network Management 0.8.2
OpenVPN Client Export Utility Security 1.3.0
Service Watchdog Services 1.7.1
syslog-ng System 1.1.1There were a number of crash dumps prior to this message which I was unable to retrieve. Could crash dumps cause this?
-
The crash dumps wouldn't cause that, but it's possible that whatever caused the crash dumps also contributed to this. Without knowing what was in the other crashes it's difficult to say what might have happened.
None of those packages should be too memory hungry except for ntopng, and it wouldn't have launched before that error came up.
At this point I'd reinstall it fresh again and restore, and be sure to capture any crash report output you can from the serial console.
-
Been building it from scratch (for the most part). Reached a point that it crashed again. Reinstalled the OS and a crash was indicated at the top of the screen.
Crash report begins. Anonymous machine information: amd64 10.3-RELEASE-p5 FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense Crash report details: Filename: /var/crash/bounds 1 Filename: /var/crash/info.0 Dump header from device /dev/label/swap0 Architecture: amd64 Architecture Version: 1 Dump Length: 71680B (0 MB) Blocksize: 512 Dumptime: Thu Nov 17 20:57:27 2016 Hostname: hq-int-1.hq.informaticsinc.com Magic: FreeBSD Text Dump Version String: FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense Panic String: Dump Parity: 2893441565 Bounds: 0 Dump Status: good Filename: /var/crash/info.last Dump header from device /dev/label/swap0 Architecture: amd64 Architecture Version: 1 Dump Length: 71680B (0 MB) Blocksize: 512 Dumptime: Thu Nov 17 20:57:27 2016 Hostname: hq-int-1.hq.informaticsinc.com Magic: FreeBSD Text Dump Version String: FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense Panic String: Dump Parity: 2893441565 Bounds: 0 Dump Status: good Filename: /var/crash/minfree 2048 Filename: /var/crash/textdump.tar.0 ddb.txt06000014000013013414667 7077 ustarrootwheeldb:0:kdb.enter.default> run lockinfo db:1:lockinfo> show locks No such command db:1:locks> show alllocks No such command db:1:alllocks> show lockedvnods Locked vnodes db:0:kdb.enter.default> show pcpu cpuid = 3 dynamic pcpu = 0xfffffe026b8c8500 curthread = 0xfffff8000d25a960: pid 46850 "pfctl" curpcb = 0xfffffe0235a1dcc0 fpcurthread = none idlethread = 0xfffff80005240000: tid 100006 "idle: cpu3" curpmap = 0xfffff80115c12d78 tssp = 0xffffffff821135c8 commontssp = 0xffffffff821135c8 rsp0 = 0xfffffe0235a1dcc0 gs32p = 0xffffffff82115020 ldt = 0xffffffff82115060 tss = 0xffffffff82115050 db:0:kdb.enter.default> bt Tracing pid 46850 tid 100166 td 0xfffff8000d25a960 m_tag_delete_chain() at m_tag_delete_chain+0x38/frame 0xfffffe0235a1ca30 uma_zfree_arg() at uma_zfree_arg+0x3e/frame 0xfffffe0235a1caa0 m_freem() at m_freem+0x63/frame 0xfffffe0235a1cac0 hfsc_purgeq() at hfsc_purgeq+0x37/frame 0xfffffe0235a1cae0 hfsc_class_destroy() at hfsc_class_destroy+0x75/frame 0xfffffe0235a1cb00 hfsc_remove_altq() at hfsc_remove_altq+0x66/frame 0xfffffe0235a1cb30 pf_commit_altq() at pf_commit_altq+0x221/frame 0xfffffe0235a1cb80 pfioctl() at pfioctl+0x4e63/frame 0xfffffe0235a1d920 devfs_ioctl_f() at devfs_ioctl_f+0x139/frame 0xfffffe0235a1d980 kern_ioctl() at kern_ioctl+0x255/frame 0xfffffe0235a1d9f0 sys_ioctl() at sys_ioctl+0x15c/frame 0xfffffe0235a1dad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235a1dbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235a1dbf0 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800dce6ea, rsp = 0x7fffffffd9b8, rbp = 0x7fffffffd9d0 --- db:0:kdb.enter.default> ps pid ppid pgrp uid state wmesg wchan cmd 47065 280 280 0 R CPU 1 php-fpm 46850 322 21 0 R+ CPU 3 pfctl 45872 280 280 0 S lockf 0xfffff80115a3bc80 php-fpm 39690 1 39690 0 Ls (threaded) ntpd 100164 S kqread 0xfffff801153b4700 ntpd 100160 L *pf rules 0xfffff800054b4000 ntpd 38583 1 38449 65534 S select 0xfffff80115950640 dnsmasq 34765 1 34765 0 Ss nanslp 0xffffffff82001170 cron 33967 33473 33473 0 S kqread 0xfffff8011530f900 nginx 33716 33473 33473 0 S kqread 0xfffff8011530f400 nginx 33473 1 33473 0 Ss pause 0xfffff8011594ba88 nginx 23430 280 280 0 S lockf 0xfffff80115410e00 php-fpm 18600 1 18600 0 Rs (threaded) dpinger 100153 S accept 0xfffff8000d1f488e dpinger 100152 S nanslp 0xffffffff82001172 dpinger 100151 Run CPU 0 dpinger 100150 S sbwait 0xfffff8000d1f4c24 dpinger 100128 S uwait 0xfffff8000d14f800 dpinger 18225 1 18225 0 Rs (threaded) dpinger 100149 S accept 0xfffff8000d1f531e dpinger 100148 RunQ dpinger 100147 L *pf rules 0xfffff800054b4000 dpinger 100146 S sbwait 0xfffff8000d1f596c dpinger 100135 S uwait 0xfffff8000d150000 dpinger 17658 1 17658 0 Rs (threaded) dpinger 100145 S accept 0xfffff801158f031e dpinger 100144 RunQ dpinger 100143 L *pf rules 0xfffff800054b4000 dpinger 100142 S sbwait 0xfffff801158f06b4 dpinger 100137 S uwait 0xfffff8000d14dc00 dpinger 15517 1 15517 0 Ss select 0xfffff8000db19cc0 xinetd 15124 1 15124 0 Ss bpf 0xfffff8011530ae00 filterlog 9174 3699 9174 0 Ss (threaded) sshlockout_pf 100141 S nanslp 0xffffffff82001173 sshlockout_pf 100108 S piperd 0xfffff8000d0d4000 sshlockout_pf 9167 1 9167 0 Ss select 0xfffff8000db1a7c0 sshd 3699 1 3699 0 Ss select 0xfffff8000d14fdc0 syslogd 332 1 332 0 Ss select 0xfffff80005fa4c40 devd 322 21 21 0 S+ piperd 0xfffff8000d1285d0 php-cgi 320 318 318 0 S kqread 0xfffff80005fe8200 check_reload_status 318 1 318 0 Ss kqread 0xfffff80005fe8400 check_reload_status 281 280 280 0 S nanslp 0xffffffff82001172 php-fpm 280 1 280 0 Ss kqread 0xfffff80005f7a500 php-fpm 51 0 0 0 DL mdwait 0xfffff8000d012000 [md0] 21 1 21 0 Ss+ pause 0xfffff8000d039a88 sh 20 0 0 0 DL syncer 0xffffffff82052108 [syncer] 19 0 0 0 DL vlruwt 0xfffff80005f869e0 [vnlru] 18 0 0 0 DL (threaded) [bufdaemon] 100111 D sdflush 0xfffff8000d1030e8 [/ worker] 100100 D psleep 0xffffffff82051304 [bufdaemon] 17 0 0 0 DL pgzero 0xffffffff8206243c [pagezero] 9 0 0 0 DL pollid 0xffffffff81fffa90 [idlepoll] 8 0 0 0 DL psleep 0xffffffff820617c0 [vmdaemon] 7 0 0 0 DL (threaded) [pagedaemon] 100106 D umarcl 0xffffffff82061140 [uma] 100096 D psleep 0xffffffff82112804 [pagedaemon] 6 0 0 0 DL waiting_ 0xffffffff821032c0 [sctp_iterator] 5 0 0 0 LL *pf rules 0xfffff800054b4000 [pf purge] 16 0 0 0 DL (threaded) [usb] 100085 D - 0xfffffe0000c56e70 [usbus0] 100084 D - 0xfffffe0000c56e18 [usbus0] 100083 D - 0xfffffe0000c56dc0 [usbus0] 100082 D - 0xfffffe0000c56d68 [usbus0] 100081 D - 0xfffffe0000c56d10 [usbus0] 4 0 0 0 DL (threaded) [cam] 100095 D - 0xffffffff81f35cc8 [scanner] 100025 D - 0xffffffff81f35e80 [doneq0] 15 0 0 0 DL - 0xffffffff81f575c0 [rand_harvestq] 3 0 0 0 DL crypto_r 0xffffffff8205fc98 [crypto returns] 2 0 0 0 DL crypto_w 0xffffffff8205fb40 [crypto] 14 0 0 0 DL (threaded) [geom] 100019 D - 0xffffffff820f79f8 [g_down] 100018 D - 0xffffffff820f79f0 [g_up] 100017 D - 0xffffffff820f79e8 [g_event] 13 0 0 0 DL (threaded) [ng_queue] 100016 D sleep 0xffffffff81ef4318 [ng_queue3] 100015 D sleep 0xffffffff81ef4318 [ng_queue2] 100014 D sleep 0xffffffff81ef4318 [ng_queue1] 100013 D sleep 0xffffffff81ef4318 [ng_queue0] 12 0 0 0 RL (threaded) [intr] 100105 I [swi1: netisr 3] 100104 I [swi1: netisr 2] 100103 I [swi1: netisr 1] 100093 I [swi1: pfsync] 100091 I [swi1: pf send] 100088 I [swi0: uart uart] 100087 I [irq283: ahci1] 100086 I [irq282: ahci0] 100080 I [irq22: ehci0] 100079 I [irq281: igb5:link] 100077 I [irq280: igb5:que 3] 100075 I [irq279: igb5:que 2] 100073 I [irq278: igb5:que 1] 100071 I [irq277: igb5:que 0] 100070 I [irq276: igb4:link] 100068 I [irq275: igb4:que 3] 100066 I [irq274: igb4:que 2] 100064 I [irq273: igb4:que 1] 100062 I [irq272: igb4:que 0] 100061 I [irq271: igb3:link] 100059 I [irq270: igb3:que 3] 100057 I [irq269: igb3:que 2] 100055 I [irq268: igb3:que 1] 100053 I [irq267: igb3:que 0] 100052 I [irq266: igb2:link] 100050 I [irq265: igb2:que 3] 100048 Run CPU 2 [irq264: igb2:que 2] 100046 L *pf rules 0xfffff800054b4000 [irq263: igb2:que 1] 100044 I [irq262: igb2:que 0] 100043 I [irq261: igb1:link] 100041 I [irq260: igb1:que 1] 100039 I [irq259: igb1:que 0] 100038 I [irq258: igb0:link] 100036 I [irq257: igb0:que 1] 100034 I [irq256: igb0:que 0] 100033 I [swi5: fast taskq] 100031 I [swi6: Giant taskq] 100029 I [swi6: task queue] 100012 I [swi4: clock] 100011 I [swi4: clock] 100010 I [swi4: clock] 100009 I [swi4: clock] 100008 I [swi1: netisr 0] 100007 I [swi3: vm] 11 0 0 0 RL (threaded) [idle] 100006 CanRun [idle: cpu3] 100005 CanRun [idle: cpu2] 100004 CanRun [idle: cpu1] 100003 CanRun [idle: cpu0] 1 0 1 0 SLs wait 0xfffff8000523e4f0 [init] 10 0 0 0 DL audit_wo 0xffffffff82109d60 [audit] 0 0 0 0 DLs (threaded) [kernel] 100094 D - 0xfffff80005260900 [CAM taskq] 100089 D - 0xfffff800054f2700 [mca taskq] 100078 D - 0xfffff800054c8200 [igb5 que] 100076 D - 0xfffff800054c8500 [igb5 que] 100074 D - 0xfffff800054c8800 [igb5 que] 100072 D - 0xfffff800054c8b00 [igb5 que] 100069 D - 0xfffff800054caa00 [igb4 que] 100067 D - 0xfffff800054cad00 [igb4 que] 100065 D - 0xfffff800054cb100 [igb4 que] 100063 D - 0xfffff800054cb400 [igb4 que] 100060 D - 0xfffff800054a6100 [igb3 que] 100058 D - 0xfffff800054a6400 [igb3 que] 100056 D - 0xfffff800054a6700 [igb3 que] 100054 D - 0xfffff800054a6a00 [igb3 que] 100051 D - 0xfffff8000543d700 [igb2 que] 100049 D - 0xfffff8000543da00 [igb2 que] 100047 D - 0xfffff8000543dd00 [igb2 que] 100045 D - 0xfffff80005480100 [igb2 que] 100042 D - 0xfffff80005482000 [igb1 que] 100040 D - 0xfffff80005482300 [igb1 que] 100037 D - 0xfffff80005262600 [igb0 que] 100035 D - 0xfffff80005262900 [igb0 que] 100032 D - 0xfffff8000525f800 [thread] 100030 D - 0xfffff8000525fd00 [ffs_trim taskq] 100028 D - 0xfffff80005260500 [acpi_task_2] 100027 D - 0xfffff80005260500 [acpi_task_1] 100026 D - 0xfffff80005260500 [acpi_task_0] 100024 D - 0xfffff80005260b00 [kqueue taskq] 100020 D - 0xfffff80005230100 [firmware taskq] 100000 D swapin 0xffffffff820f7af8 [swapper] db:0:kdb.enter.default> alltrace Tracing command php-fpm pid 47065 tid 100167 td 0xfffff8011594d4b0 cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe01ed7ebcf0 ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xfffffe01ed7ebd10 trap() at trap+0x3a/frame 0xfffffe01ed7ebf20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe01ed7ebf20 --- trap 0x13, rip = 0x55f269, rsp = 0x7fffffff8220, rbp = 0x7fffffff82f0 --- Tracing command pfctl pid 46850 tid 100166 td 0xfffff8000d25a960 m_tag_delete_chain() at m_tag_delete_chain+0x38/frame 0xfffffe0235a1ca30 uma_zfree_arg() at uma_zfree_arg+0x3e/frame 0xfffffe0235a1caa0 m_freem() at m_freem+0x63/frame 0xfffffe0235a1cac0 hfsc_purgeq() at hfsc_purgeq+0x37/frame 0xfffffe0235a1cae0 hfsc_class_destroy() at hfsc_class_destroy+0x75/frame 0xfffffe0235a1cb00 hfsc_remove_altq() at hfsc_remove_altq+0x66/frame 0xfffffe0235a1cb30 pf_commit_altq() at pf_commit_altq+0x221/frame 0xfffffe0235a1cb80 pfioctl() at pfioctl+0x4e63/frame 0xfffffe0235a1d920 devfs_ioctl_f() at devfs_ioctl_f+0x139/frame 0xfffffe0235a1d980 kern_ioctl() at kern_ioctl+0x255/frame 0xfffffe0235a1d9f0 sys_ioctl() at sys_ioctl+0x15c/frame 0xfffffe0235a1dad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235a1dbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235a1dbf0 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800dce6ea, rsp = 0x7fffffffd9b8, rbp = 0x7fffffffd9d0 --- Tracing command php-fpm pid 45872 tid 100165 td 0xfffff8000d25b000 sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235a18690 mi_switch() at mi_switch+0xe1/frame 0xfffffe0235a186d0 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235a18750 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235a18780 _sleep() at _sleep+0x27d/frame 0xfffffe0235a18800 lf_advlockasync() at lf_advlockasync+0xb4a/frame 0xfffffe0235a188c0 lf_advlock() at lf_advlock+0x45/frame 0xfffffe0235a18910 vop_stdadvlock() at vop_stdadvlock+0xc7/frame 0xfffffe0235a18a00 VOP_ADVLOCK_APV() at VOP_ADVLOCK_APV+0xa7/frame 0xfffffe0235a18a30 sys_flock() at sys_flock+0x137/frame 0xfffffe0235a18ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235a18bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235a18bf0 --- syscall (131, FreeBSD ELF64, sys_flock), rip = 0x801ae69ba, rsp = 0x7fffffffab08, rbp = 0x7fffffffab40 --- Tracing command ntpd pid 39690 tid 100164 td 0xfffff8011594d960 sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235a13670 mi_switch() at mi_switch+0xe1/frame 0xfffffe0235a136b0 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235a13730 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0235a13760 _sleep() at _sleep+0x238/frame 0xfffffe0235a137e0 kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe0235a139c0 kern_kevent() at kern_kevent+0x9f/frame 0xfffffe0235a13a20 sys_kevent() at sys_kevent+0x12a/frame 0xfffffe0235a13ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235a13bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235a13bf0 --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x8015f88fa, rsp = 0x7fffdfffc5c8, rbp = 0x7fffdfffcef0 --- Tracing command ntpd pid 39690 tid 100160 td 0xfffff80115ac24b0 sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359fd230 mi_switch() at mi_switch+0xe1/frame 0xfffffe02359fd270 turnstile_wait() at turnstile_wait+0x42a/frame 0xfffffe02359fd2c0 __rw_rlock() at __rw_rlock+0x319/frame 0xfffffe02359fd360 pf_test() at pf_test+0x191/frame 0xfffffe02359fd680 pf_check_out() at pf_check_out+0x1d/frame 0xfffffe02359fd6a0 pfil_run_hooks() at pfil_run_hooks+0x8d/frame 0xfffffe02359fd730 ip_output() at ip_output+0x9c0/frame 0xfffffe02359fd830 udp_send() at udp_send+0x972/frame 0xfffffe02359fd900 sosend_dgram() at sosend_dgram+0x406/frame 0xfffffe02359fd980 kern_sendit() at kern_sendit+0x205/frame 0xfffffe02359fda30 sendit() at sendit+0x129/frame 0xfffffe02359fda80 sys_sendto() at sys_sendto+0x4d/frame 0xfffffe02359fdad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359fdbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359fdbf0 --- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x8015f88ba, rsp = 0x7fffffffddf8, rbp = 0x7fffffffde40 --- Tracing command dnsmasq pid 38583 tid 100162 td 0xfffff80115ac2000 sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235a09790 mi_switch() at mi_switch+0xe1/frame 0xfffffe0235a097d0 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235a09850 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235a09880 _cv_wait_sig() at _cv_wait_sig+0x17a/frame 0xfffffe0235a098d0 seltdwait() at seltdwait+0xae/frame 0xfffffe0235a09910 kern_poll() at kern_poll+0x464/frame 0xfffffe0235a09ab0 sys_poll() at sys_poll+0x61/frame 0xfffffe0235a09ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235a09bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235a09bf0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x801463efa, rsp = 0x7fffffffeb08, rbp = 0x7fffffffec10 --- Tracing command cron pid 34765 tid 100154 td 0xfffff8011548b000 sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359df8c0 mi_switch() at mi_switch+0xe1/frame 0xfffffe02359df900 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359df980 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe02359df9b0 _sleep() at _sleep+0x238/frame 0xfffffe02359dfa30 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe02359dfa90 sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe02359dfad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359dfbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359dfbf0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800d19f3a, rsp = 0x7fffffffea88, rbp = 0x7fffffffecb0 --- Tracing command nginx pid 33967 tid 100110 td 0xfffff80005f804b0 sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235903670 mi_switch() at mi_switch+0xe1/frame 0xfffffe02359036b0 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235903730 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0235903760 _sleep() at _sleep+0x238/frame 0xfffffe02359037e0 kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe02359039c0 kern_kevent() at kern_kevent+0x9f/frame 0xfffffe0235903a20 sys_kevent() at sys_kevent+0x12a/frame 0xfffffe0235903ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235903bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235903bf0 --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x8018fd8fa, rsp = 0x7fffffffe768, rbp = 0x7fffffffe7b0 --- Tracing command nginx pid 33716 tid 100156 td 0xfffff80115a9e960 sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359e9670 mi_switch() at mi_switch+0xe1/frame 0xfffffe02359e96b0 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359e9730 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359e9760 _sleep() at _sleep+0x27d/frame 0xfffffe02359e97e0 kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe02359e99c0 kern_kevent() at kern_kevent+0x9f/frame 0xfffffe02359e9a20 sys_kevent() at sys_kevent+0x12a/frame 0xfffffe02359e9ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359e9bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359e9bf0 --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x8018fd8fa, rsp = 0x7fffffffe768, rbp = 0x7fffffffe7b0 --- Tracing command nginx pid 33473 tid 100155 td 0xfffff8011594e960 sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359e48d0 mi_switch() at mi_switch+0xe1/frame 0xfffffe02359e4910 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359e4990 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359e49c0 _sleep() at _sleep+0x27d/frame 0xfffffe02359e4a40 kern_sigsuspend() at kern_sigsuspend+0xf4/frame 0xfffffe02359e4aa0 sys_sigsuspend() at sys_sigsuspend+0x31/frame 0xfffffe02359e4ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359e4bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359e4bf0 --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x8018f2dfa, rsp = 0x7fffffffe908, rbp = 0x7fffffffe940 --- Tracing command php-fpm pid 23430 tid 100114 td 0xfffff8000d042960 sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235917690 mi_switch() at mi_switch+0xe1/frame 0xfffffe02359176d0 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235917750 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235917780 _sleep() at _sleep+0x27d/frame 0xfffffe0235917800 lf_advlockasync() at lf_advlockasync+0xb4a/frame 0xfffffe02359178c0 lf_advlock() at lf_advlock+0x45/frame 0xfffffe0235917910 vop_stdadvlock() at vop_stdadvlock+0xc7/frame 0xfffffe0235917a00 VOP_ADVLOCK_APV() at VOP_ADVLOCK_APV+0xa7/frame 0xfffffe0235917a30 sys_flock() at sys_flock+0x137/frame 0xfffffe0235917ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235917bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235917bf0 --- syscall (131, FreeBSD ELF64, sys_flock), rip = 0x801ae69ba, rsp = 0x7fffffffab08, rbp = 0x7fffffffab40 --- Tracing command dpinger pid 18600 tid 100153 td 0xfffff8011548b4b0 sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359da850 mi_switch() at mi_switch+0xe1/frame 0xfffffe02359da890 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359da910 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359da940 _sleep() at _sleep+0x27d/frame 0xfffffe02359da9c0 kern_accept4() at kern_accept4+0x1b1/frame 0xfffffe02359daa70 accept1() at accept1+0x92/frame 0xfffffe02359daad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359dabf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359dabf0 --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x800b1bfda, rsp = 0x7fffdf9fae78, rbp = 0x7fffdf9faeb0 --- Tracing command dpinger pid 18600 tid 100152 td 0xfffff8011548b960 sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359d58c0 mi_switch() at mi_switch+0xe1/frame 0xfffffe02359d5900 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359d5980 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe02359d59b0 _sleep() at _sleep+0x238/frame 0xfffffe02359d5a30 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe02359d5a90 sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe02359d5ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359d5bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359d5bf0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b1bf3a, rsp = 0x7fffdfbfbef8, rbp = 0x7fffdfbfbf20 --- Tracing command dpinger pid 18600 tid 100151 td 0xfffff80115904000 cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff820669e0 ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xffffffff82066a00 trap() at trap+0x3a/frame 0xffffffff82066c10 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff82066c10 --- trap 0x13, rip = 0xffffffff80b090b2, rsp = 0xfffffe02359d0300, rbp = 0xfffffe02359d0390 --- __rw_rlock() at __rw_rlock+0x1f2/frame 0xfffffe02359d0390 pf_test() at pf_test+0x191/frame 0xfffffe02359d06b0 pf_check_out() at pf_check_out+0x1d/frame 0xfffffe02359d06d0 pfil_run_hooks() at pfil_run_hooks+0x8d/frame 0xfffffe02359d0760 ip_output() at ip_output+0x9c0/frame 0xfffffe02359d0860 rip_output() at rip_output+0x2cd/frame 0xfffffe02359d08b0 sosend_generic() at sosend_generic+0x476/frame 0xfffffe02359d0980 kern_sendit() at kern_sendit+0x205/frame 0xfffffe02359d0a30 sendit() at sendit+0x129/frame 0xfffffe02359d0a80 sys_sendto() at sys_sendto+0x4d/frame 0xfffffe02359d0ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359d0bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359d0bf0 --- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x800b268ba, rsp = 0x7fffdfdfcf38, rbp = 0x7fffdfdfcf80 --- Tracing command dpinger pid 18600 tid 100150 td 0xfffff801159044b0 sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359cb740 mi_switch() at mi_switch+0xe1/frame 0xfffffe02359cb780 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359cb800 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359cb830 _sleep() at _sleep+0x27d/frame 0xfffffe02359cb8b0 sbwait() at sbwait+0x4f/frame 0xfffffe02359cb8d0 soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe02359cb9a0 kern_recvit() at kern_recvit+0x1f2/frame 0xfffffe02359cba60 sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe02359cbad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359cbbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359cbbf0 --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b268da, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 --- Tracing command dpinger pid 18600 tid 100128 td 0xfffff8000d1e5000 sched_switch() at sched_switch+0x2b3/frame 0xfffffe023595d840 mi_switch() at mi_switch+0xe1/frame 0xfffffe023595d880 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023595d900 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe023595d930 _sleep() at _sleep+0x27d/frame 0xfffffe023595d9b0 umtxq_sleep() at umtxq_sleep+0x125/frame 0xfffffe023595da10 do_wait() at do_wait+0x387/frame 0xfffffe023595da90 __umtx_op_wait() at __umtx_op_wait+0x7d/frame 0xfffffe023595dad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023595dbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe023595dbf0 --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083813c, rsp = 0x7fffffffe798, rbp = 0x7fffffffe810 --- Tracing command dpinger pid 18225 tid 100149 td 0xfffff80115904960 sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359c6850 mi_switch() at mi_switch+0xe1/frame 0xfffffe02359c6890 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359c6910 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359c6940 _sleep() at _sleep+0x27d/frame 0xfffffe02359c69c0 kern_accept4() at kern_accept4+0x1b1/frame 0xfffffe02359c6a70 accept1() at accept1+0x92/frame 0xfffffe02359c6ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359c6bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359c6bf0 --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x800b1bfda, rsp = 0x7fffdf9fae78, rbp = 0x7fffdf9faeb0 --- Tracing command dpinger pid 18225 tid 100148 td 0xfffff80115905000 sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359c18c0 mi_switch() at mi_switch+0xe1/frame 0xfffffe02359c1900 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359c1980 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe02359c19b0 _sleep() at _sleep+0x238/frame 0xfffffe02359c1a30 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe02359c1a90 sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe02359c1ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359c1bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359c1bf0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b1bf3a, rsp = 0x7fffdfbfbef8, rbp = 0x7fffdfbfbf20 --- Tracing command dpinger pid 18225 tid 100147 td 0xfffff801159054b0 sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359bc260 mi_switch() at mi_switch+0xe1/frame 0xfffffe02359bc2a0 turnstile_wait() at turnstile_wait+0x42a/frame 0xfffffe02359bc2f0 __rw_rlock() at __rw_rlock+0x319/frame 0xfffffe02359bc390 pf_test() at pf_test+0x191/frame 0xfffffe02359bc6b0 pf_check_out() at pf_check_out+0x1d/frame 0xfffffe02359bc6d0 pfil_run_hooks() at pfil_run_hooks+0x8d/frame 0xfffffe02359bc760 ip_output() at ip_output+0x9c0/frame 0xfffffe02359bc860 rip_output() at rip_output+0x2cd/frame 0xfffffe02359bc8b0 sosend_generic() at sosend_generic+0x476/frame 0xfffffe02359bc980 kern_sendit() at kern_sendit+0x205/frame 0xfffffe02359bca30 sendit() at sendit+0x129/frame 0xfffffe02359bca80 sys_sendto() at sys_sendto+0x4d/frame 0xfffffe02359bcad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359bcbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359bcbf0 --- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x800b268ba, rsp = 0x7fffdfdfcf38, rbp = 0x7fffdfdfcf80 --- Tracing command dpinger pid 18225 tid 100146 td 0xfffff80115905960 sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359b7740 mi_switch() at mi_switch+0xe1/frame 0xfffffe02359b7780 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359b7800 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359b7830 _sleep() at _sleep+0x27d/frame 0xfffffe02359b78b0 sbwait() at sbwait+0x4f/frame 0xfffffe02359b78d0 soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe02359b79a0 kern_recvit() at kern_recvit+0x1f2/frame 0xfffffe02359b7a60 sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe02359b7ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359b7bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359b7bf0 --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b268da, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 --- Tracing command dpinger pid 18225 tid 100135 td 0xfffff8000d1f8000 sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235980840 mi_switch() at mi_switch+0xe1/frame 0xfffffe0235980880 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235980900 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235980930 _sleep() at _sleep+0x27d/frame 0xfffffe02359809b0 umtxq_sleep() at umtxq_sleep+0x125/frame 0xfffffe0235980a10 do_wait() at do_wait+0x387/frame 0xfffffe0235980a90 __umtx_op_wait() at __umtx_op_wait+0x7d/frame 0xfffffe0235980ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235980bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235980bf0 --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083813c, rsp = 0x7fffffffe798, rbp = 0x7fffffffe810 --- Tracing command dpinger pid 17658 tid 100145 td 0xfffff8000d48a4b0 sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359b2850 mi_switch() at mi_switch+0xe1/frame 0xfffffe02359b2890 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359b2910 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359b2940 _sleep() at _sleep+0x27d/frame 0xfffffe02359b29c0 kern_accept4() at kern_accept4+0x1b1/frame 0xfffffe02359b2a70 accept1() at accept1+0x92/frame 0xfffffe02359b2ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359b2bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359b2bf0 --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x800b1bfda, rsp = 0x7fffdf9fae78, rbp = 0x7fffdf9faeb0 --- Tracing command dpinger pid 17658 tid 100144 td 0xfffff8000d48a960 sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359ad8c0 mi_switch() at mi_switch+0xe1/frame 0xfffffe02359ad900 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359ad980 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe02359ad9b0 _sleep() at _sleep+0x238/frame 0xfffffe02359ada30 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe02359ada90 sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe02359adad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359adbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359adbf0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b1bf3a, rsp = 0x7fffdfbfbef8, rbp = 0x7fffdfbfbf20 --- Tracing command dpinger pid 17658 tid 100143 td 0xfffff8000d48b000 sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359a8260 mi_switch() at mi_switch+0xe1/frame 0xfffffe02359a82a0 turnstile_wait() at turnstile_wait+0x42a/frame 0xfffffe02359a82f0 __rw_rlock() at __rw_rlock+0x319/frame 0xfffffe02359a8390 pf_test() at pf_test+0x191/frame 0xfffffe02359a86b0 pf_check_out() at pf_check_out+0x1d/frame 0xfffffe02359a86d0 pfil_run_hooks() at pfil_run_hooks+0x8d/frame 0xfffffe02359a8760 ip_output() at ip_output+0x9c0/frame 0xfffffe02359a8860 rip_output() at rip_output+0x2cd/frame 0xfffffe02359a88b0 sosend_generic() at sosend_generic+0x476/frame 0xfffffe02359a8980 kern_sendit() at kern_sendit+0x205/frame 0xfffffe02359a8a30 sendit() at sendit+0x129/frame 0xfffffe02359a8a80 sys_sendto() at sys_sendto+0x4d/frame 0xfffffe02359a8ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359a8bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359a8bf0 --- syscall (133, FreeBSD ELF64, sys_sendto), rip = 0x800b268ba, rsp = 0x7fffdfdfcf38, rbp = 0x7fffdfdfcf80 --- Tracing command dpinger pid 17658 tid 100142 td 0xfffff8000d48b4b0 sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359a3740 mi_switch() at mi_switch+0xe1/frame 0xfffffe02359a3780 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359a3800 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02359a3830 _sleep() at _sleep+0x27d/frame 0xfffffe02359a38b0 sbwait() at sbwait+0x4f/frame 0xfffffe02359a38d0 soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe02359a39a0 kern_recvit() at kern_recvit+0x1f2/frame 0xfffffe02359a3a60 sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe02359a3ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02359a3bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02359a3bf0 --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b268da, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 --- Tracing command dpinger pid 17658 tid 100137 td 0xfffff8000d4aa4b0 sched_switch() at sched_switch+0x2b3/frame 0xfffffe023598a840 mi_switch() at mi_switch+0xe1/frame 0xfffffe023598a880 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023598a900 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe023598a930 _sleep() at _sleep+0x27d/frame 0xfffffe023598a9b0 umtxq_sleep() at umtxq_sleep+0x125/frame 0xfffffe023598aa10 do_wait() at do_wait+0x387/frame 0xfffffe023598aa90 __umtx_op_wait() at __umtx_op_wait+0x7d/frame 0xfffffe023598aad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023598abf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe023598abf0 --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083813c, rsp = 0x7fffffffe7a8, rbp = 0x7fffffffe820 --- Tracing command xinetd pid 15517 tid 100134 td 0xfffff8000d1f84b0 sched_switch() at sched_switch+0x2b3/frame 0xfffffe023597b710 mi_switch() at mi_switch+0xe1/frame 0xfffffe023597b750 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023597b7d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe023597b800 _cv_wait_sig() at _cv_wait_sig+0x17a/frame 0xfffffe023597b850 seltdwait() at seltdwait+0xae/frame 0xfffffe023597b890 kern_select() at kern_select+0x8fa/frame 0xfffffe023597ba90 sys_select() at sys_select+0x54/frame 0xfffffe023597bad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023597bbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe023597bbf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800fca66a, rsp = 0x7fffffffeb08, rbp = 0x7fffffffec40 --- Tracing command filterlog pid 15124 tid 100140 td 0xfffff8000d03a4b0 sched_switch() at sched_switch+0x2b3/frame 0xfffffe02359997c0 mi_switch() at mi_switch+0xe1/frame 0xfffffe0235999800 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235999880 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe02359998b0 _sleep() at _sleep+0x238/frame 0xfffffe0235999930 bpfread() at bpfread+0x1c7/frame 0xfffffe0235999980 devfs_read_f() at devfs_read_f+0xd8/frame 0xfffffe02359999e0 dofileread() at dofileread+0x95/frame 0xfffffe0235999a30 kern_readv() at kern_readv+0x68/frame 0xfffffe0235999a80 sys_read() at sys_read+0x63/frame 0xfffffe0235999ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235999bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235999bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800da168a, rsp = 0x7fffffffeac8, rbp = 0x7fffffffec00 --- Tracing command sshlockout_pf pid 9174 tid 100141 td 0xfffff8000d03a000 sched_switch() at sched_switch+0x2b3/frame 0xfffffe023599e8c0 mi_switch() at mi_switch+0xe1/frame 0xfffffe023599e900 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023599e980 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe023599e9b0 _sleep() at _sleep+0x238/frame 0xfffffe023599ea30 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe023599ea90 sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe023599ead0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023599ebf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe023599ebf0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b18f3a, rsp = 0x7fffdfffdf48, rbp = 0x7fffdfffdf70 --- Tracing command sshlockout_pf pid 9174 tid 100108 td 0xfffff80005fa94b0 sched_switch() at sched_switch+0x2b3/frame 0xfffffe02358f9810 mi_switch() at mi_switch+0xe1/frame 0xfffffe02358f9850 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02358f98d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe02358f9900 _sleep() at _sleep+0x27d/frame 0xfffffe02358f9980 pipe_read() at pipe_read+0x2aa/frame 0xfffffe02358f99e0 dofileread() at dofileread+0x95/frame 0xfffffe02358f9a30 kern_readv() at kern_readv+0x68/frame 0xfffffe02358f9a80 sys_read() at sys_read+0x63/frame 0xfffffe02358f9ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe02358f9bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe02358f9bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b7b68a, rsp = 0x7fffffffe7d8, rbp = 0x7fffffffe7f0 --- Tracing command sshd pid 9167 tid 100132 td 0xfffff8000d1e4000 sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235971710 mi_switch() at mi_switch+0xe1/frame 0xfffffe0235971750 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359717d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235971800 _cv_wait_sig() at _cv_wait_sig+0x17a/frame 0xfffffe0235971850 seltdwait() at seltdwait+0xae/frame 0xfffffe0235971890 kern_select() at kern_select+0x8fa/frame 0xfffffe0235971a90 sys_select() at sys_select+0x54/frame 0xfffffe0235971ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235971bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235971bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x80327966a, rsp = 0x7fffffffe228, rbp = 0x7fffffffe260 --- Tracing command syslogd pid 3699 tid 100139 td 0xfffff8000d48b960 sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235994710 mi_switch() at mi_switch+0xe1/frame 0xfffffe0235994750 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359947d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235994800 _cv_wait_sig() at _cv_wait_sig+0x17a/frame 0xfffffe0235994850 seltdwait() at seltdwait+0xae/frame 0xfffffe0235994890 kern_select() at kern_select+0x8fa/frame 0xfffffe0235994a90 sys_select() at sys_select+0x54/frame 0xfffffe0235994ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235994bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235994bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800b7166a, rsp = 0x7fffffffdc08, rbp = 0x7fffffffec20 --- Tracing command devd pid 332 tid 100123 td 0xfffff80005f97000 sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235944700 mi_switch() at mi_switch+0xe1/frame 0xfffffe0235944740 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359447c0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe02359447f0 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x19e/frame 0xfffffe0235944850 seltdwait() at seltdwait+0xa4/frame 0xfffffe0235944890 kern_select() at kern_select+0x8fa/frame 0xfffffe0235944a90 sys_select() at sys_select+0x54/frame 0xfffffe0235944ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235944bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235944bf0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x4a8caa, rsp = 0x7fffffffcb48, rbp = 0x7fffffffecd0 --- Tracing command php-cgi pid 322 tid 100124 td 0xfffff8000d1e54b0 sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235949810 mi_switch() at mi_switch+0xe1/frame 0xfffffe0235949850 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe02359498d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe0235949900 _sleep() at _sleep+0x27d/frame 0xfffffe0235949980 pipe_read() at pipe_read+0x2aa/frame 0xfffffe02359499e0 dofileread() at dofileread+0x95/frame 0xfffffe0235949a30 kern_readv() at kern_readv+0x68/frame 0xfffffe0235949a80 sys_read() at sys_read+0x63/frame 0xfffffe0235949ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235949bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235949bf0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x801b9968a, rsp = 0x7fffffffcb48, rbp = 0x7fffffffcb80 --- Tracing command check_reload_status pid 320 tid 100125 td 0xfffff8000d25b960 sched_switch() at sched_switch+0x2b3/frame 0xfffffe023594e670 mi_switch() at mi_switch+0xe1/frame 0xfffffe023594e6b0 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023594e730 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe023594e760 _sleep() at _sleep+0x27d/frame 0xfffffe023594e7e0 kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe023594e9c0 kern_kevent() at kern_kevent+0x9f/frame 0xfffffe023594ea20 sys_kevent() at sys_kevent+0x12a/frame 0xfffffe023594ead0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023594ebf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe023594ebf0 --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x800d478fa, rsp = 0x7fffffffe9e8, rbp = 0x7fffffffea30 --- Tracing command check_reload_status pid 318 tid 100129 td 0xfffff8000d03a960 sched_switch() at sched_switch+0x2b3/frame 0xfffffe0235962670 mi_switch() at mi_switch+0xe1/frame 0xfffffe02359626b0 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe0235962730 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe0235962760 _sleep() at _sleep+0x238/frame 0xfffffe02359627e0 kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe02359629c0 kern_kevent() at kern_kevent+0x9f/frame 0xfffffe0235962a20 sys_kevent() at sys_kevent+0x12a/frame 0xfffffe0235962ad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe0235962bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0235962bf0 --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x800d478fa, rsp = 0x7fffffffe8a8, rbp = 0x7fffffffe8f0 --- Tracing command php-fpm pid 281 tid 100122 td 0xfffff80005f974b0 sched_switch() at sched_switch+0x2b3/frame 0xfffffe023593f8c0 mi_switch() at mi_switch+0xe1/frame 0xfffffe023593f900 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023593f980 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe023593f9b0 _sleep() at _sleep+0x238/frame 0xfffffe023593fa30 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe023593fa90 sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe023593fad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023593fbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe023593fbf0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x801b4af3a, rsp = 0x7fffffffae78, rbp = 0x7fffffffaea0 --- Tracing command php-fpm pid 280 tid 100118 td 0xfffff80005fa8960 sched_switch() at sched_switch+0x2b3/frame 0xfffffe023592b670 mi_switch() at mi_switch+0xe1/frame 0xfffffe023592b6b0 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe023592b730 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe023592b760 _sleep() at _sleep+0x238/frame 0xfffffe023592b7e0 kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe023592b9c0 kern_kevent() at kern_kevent+0x9f/frame 0xfffffe023592ba20 sys_kevent() at sys_kevent+0x12a/frame 0xfffffe023592bad0 amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe023592b[/thread] -
Seems like it might be https://redmine.pfsense.org/issues/6257 but it's not quite an exact match for the ones we saw hitting that previously.
-
Ironic, I manually rebuilt the config and when I added the Shaper it crashed. the 2.2.6 Shaper must not be compatible in 2.3.
Thanks!!
