Crash when enabling traffic shaper on more than 1 port
-
traffic shaper crashing happens to me too on sg-1000 as well as apu2 but only when i saturate the line above 50mbps
-
I read that thread as well when looking if it was a known issue.
But I’m quite possitive the origin lies with the shaper.I run several days now with the shaper enabled on the wan link.
As soon as I enable it on a 2nd interface, no matter which one, (lan, dmz, lab), pfSense crashes. -
Please post the entire crash dump, the summary is not enough information to go on. At least the backtrace and last section of the message buffer are required, but the more info we have, the better.
Can you post the details of your shaper configuration? Preferably also including the shaper queue/rules from /tmp/rules.debug.
Does this only happen with HFSC or does it also happen with PRIQ?
On the system with igb NICs, try setting hw.igb.num_queues=1 in /boot/loader.conf.local and then reboot, see if it stabilizes.
-
Current configuration is 3 shapers, WAN, LAN and DMZ. (See attached picture)
In general, the shapers are similar to the configurations from the wizard.
WAN - HFSC, 160Mbps
qInternet - Prio 3, 160 Mbps
- qAck - prio 6, 25% borrow
- qVoip - prio 6, 256kbps
- qGold - prio 5, 15%, RED, ECN, borrow
- qSilver - prio 4, 44%, RED, borrow, default queue
- qBronze - prio 3, 15%, RED, borrowLan and DMZ have similar configuration except for the VoIP queue
LAN - HFSC, 1Gbps
qSilver - prio 4, 60%, RED, borrow, default queue
qOther - prio 4, 40%
- qAck - prio 6, 50%, borrow
- qVoip - prio 6, 256kbps
- qGold - prio 5, 25%, RED, ECN, borrow
- qBronze - prio 3, 24%, RED, borrowThe first thing I did was reading this post FreeBSD Network Performance Tuning @ Calomel.org and changed hw.igb.num_queues=1 in /boot/loader.conf.local
It now looks like this:Accept intel license
legal.intel_iwi.license_ack=1
kern.cam.boot_delay=10000
kern.ipc.nmbclusters="1000000"ipmi_load="YES"
hw.igb.enable_msix=1hw.igb.buf_ring_size="32768"
hw.igb.num_queues="1"
hw.igb.rx_process_limit="4096"
hw.igb.tx_process_limit="4096"
hw.igb.rxd="4096"
hw.igb.txd="4096"
net.link.ifqmaxlen="40960"
net.pf.states_hashsize="16777216"Disable flow control
dev.igb.0.fc=0
dev.igb.1.fc=0
dev.igb.2.fc=0
dev.igb.3.fc=0That did not solve the problen
After some crashes I changed the configuration from HFSC to CBQ but that didn't help either.Unfortunately I didn't save the crashdump so I cannot post one now. But I uploaded two.
They can be identified by the name "firewall.familievink.nl"If required I will enable the LAN shaper as well to create a crash dump, but that requires some planning.
 -
Can you remove all of the extra tuning and just use num_queues?
And can you try PRIQ to see if it also crashes?
-
First removed all extra configurations except for bandwidth and prio. Also no borrow from other queues.
Enabled, both Wan and Lan
The good thing is that the firewall is not crashing immediately.
Let it run for 10 minutes, no problems so far, which is better than what I had.Then hhanged the WAN and LAN queues to PRIQ
qAck, prio 7
qVoip, prio 6
qGold, prio 5
qSilver, prio 4, default
qBronze, prio 3This runs fine as well.
So, next thing is to find out where it goes wrong. My first guess will be the borrow option.
-
Unfortunately, I was too optimistic.
The CBQ configuration crashed after the traffic increased a bit. Not even that much. at 12Mbps… poooffThe crashdump was uploaded for inspection, here are the first 350 lines
Crash report begins. Anonymous machine information:
amd64
11.0-RELEASE-p12
FreeBSD 11.0-RELEASE-p12 #44 eed6fd412cc(RELENG_2_4_0): Fri Sep 15 16:11:43 CDT 2017 root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSenseCrash report details:
No PHP errors found.
Filename: /var/crash/bounds
2Filename: /var/crash/info.0
Dump header from device: /dev/label/swap0
Architecture: amd64
Architecture Version: 1
Dump Length: 87552
Blocksize: 512
Dumptime: Tue Sep 19 19:59:49 2017
Hostname: firewall.familievink.nl
Magic: FreeBSD Text Dump
Version String: FreeBSD 11.0-RELEASE-p12 #44 eed6fd412cc(RELENG_2_4_0): Fri Sep 15 16:11:43 CDT 2017
root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense
Panic String: _rmc_wrr_dequeue_next
Dump Parity: 2036107568
Bounds: 0
Dump Status: goodFilename: /var/crash/info.1
Dump header from device: /dev/label/swap0
Architecture: amd64
Architecture Version: 1
Dump Length: 101376
Blocksize: 512
Dumptime: Tue Sep 19 20:03:10 2017
Hostname: firewall.familievink.nl
Magic: FreeBSD Text Dump
Version String: FreeBSD 11.0-RELEASE-p12 #44 eed6fd412cc(RELENG_2_4_0): Fri Sep 15 16:11:43 CDT 2017
root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense
Panic String: _rmc_wrr_dequeue_next
Dump Parity: 2995161392
Bounds: 1
Dump Status: goodFilename: /var/crash/info.last
Dump header from device: /dev/label/swap0
Architecture: amd64
Architecture Version: 1
Dump Length: 101376
Blocksize: 512
Dumptime: Tue Sep 19 20:03:10 2017
Hostname: firewall.familievink.nl
Magic: FreeBSD Text Dump
Version String: FreeBSD 11.0-RELEASE-p12 #44 eed6fd412cc(RELENG_2_4_0): Fri Sep 15 16:11:43 CDT 2017
root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense
Panic String: _rmc_wrr_dequeue_next
Dump Parity: 2995161392
Bounds: 1
Dump Status: goodFilename: /var/crash/textdump.tar.0
ddb.txt���������������������������������������������������������������������������������������������0600����0�������0�������140000������13160255425� 7075� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������db:0:kdb.enter.default> run lockinfo
db:1:lockinfo> show locks
No such command
db:1:locks> show alllocks
No such command
db:1:alllocks> show lockedvnods
Locked vnodes
db:0:kdb.enter.default> show pcpu
cpuid = 2
dynamic pcpu = 0xfffffe02a6ca5200
curthread = 0xfffff80006535a00: pid 12 "irq268: igb1:que 0"
curpcb = 0xfffffe02293deb80
fpcurthread = none
idlethread = 0xfffff80006219500: tid 100005 "idle: cpu2"
curpmap = 0xffffffff829e6300
tssp = 0xffffffff82a1ebe0
commontssp = 0xffffffff82a1ebe0
rsp0 = 0xfffffe02293deb80
gs32p = 0xffffffff82a25438
ldt = 0xffffffff82a25478
tss = 0xffffffff82a25468
db:0:kdb.enter.default> bt
Tracing pid 12 tid 100043 td 0xfffff80006535a00
kdb_enter() at kdb_enter+0x3b/frame 0xfffffe02293ddfe0
vpanic() at vpanic+0x19f/frame 0xfffffe02293de060
panic() at panic+0x43/frame 0xfffffe02293de0c0
rmc_dequeue_next() at rmc_dequeue_next+0x4b2/frame 0xfffffe02293de110
cbq_dequeue() at cbq_dequeue+0x26/frame 0xfffffe02293de150
tbr_dequeue() at tbr_dequeue+0xdf/frame 0xfffffe02293de1a0
vlan_start() at vlan_start+0x12c/frame 0xfffffe02293de200
if_transmit() at if_transmit+0x1a5/frame 0xfffffe02293de250
ether_output() at ether_output+0x728/frame 0xfffffe02293de2e0
ip_output() at ip_output+0x1746/frame 0xfffffe02293de430
phyint_send() at phyint_send+0xb8/frame 0xfffffe02293de4a0
ip_mdq() at ip_mdq+0x306/frame 0xfffffe02293de510
X_ip_mforward() at X_ip_mforward+0x356/frame 0xfffffe02293de590
ip_input() at ip_input+0x6b0/frame 0xfffffe02293de5f0
netisr_dispatch_src() at netisr_dispatch_src+0xa5/frame 0xfffffe02293de650
ether_demux() at ether_demux+0x15c/frame 0xfffffe02293de680
ether_nh_input() at ether_nh_input+0x317/frame 0xfffffe02293de6e0
netisr_dispatch_src() at netisr_dispatch_src+0xa5/frame 0xfffffe02293de740
ether_input() at ether_input+0x26/frame 0xfffffe02293de760
vlan_input() at vlan_input+0x1cb/frame 0xfffffe02293de7e0
ether_demux() at ether_demux+0xa9/frame 0xfffffe02293de810
ether_nh_input() at ether_nh_input+0x317/frame 0xfffffe02293de870
netisr_dispatch_src() at netisr_dispatch_src+0xa5/frame 0xfffffe02293de8d0
ether_input() at ether_input+0x26/frame 0xfffffe02293de8f0
igb_rxeof() at igb_rxeof+0x81c/frame 0xfffffe02293de990
igb_msix_que() at igb_msix_que+0x152/frame 0xfffffe02293de9e0
intr_event_execute_handlers() at intr_event_execute_handlers+0x20f/frame 0xfffffe02293dea20
ithread_loop() at ithread_loop+0xc6/frame 0xfffffe02293dea70
fork_exit() at fork_exit+0x85/frame 0xfffffe02293deab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe02293deab0
–- trap 0, rip = 0, rsp = 0, rbp = 0 ---
db:0:kdb.enter.default> ps
pid ppid pgrp uid state wmesg wchan cmd
33038 298 298 0 S accept 0xfffff8009007a06c php-fpm
29694 35607 298 0 S nanslp 0xffffffff82866b33 sleep
6612 1 6612 0 Ss accept 0xfffff801b2ff506c qstats
35607 1 298 0 S wait 0xfffff801b4018528 sh
93608 92629 93608 0 S+ ttyin 0xfffff800069e60a8 tcsh
92629 91887 92629 0 Ss+ wait 0xfffff801b4b18000 sh
92350 62701 92350 0 Ss (threaded) sshlockout_pf
100245 S piperd 0xfffff8018dbb58e8 sshlockout_pf
100757 S nanslp 0xffffffff82866b31 sshlockout_pf
91887 7656 91887 0 Ss select 0xfffff8015f7c1140 sshd
22994 1 298 0 S nanslp 0xffffffff82866b33 php
16380 1 16380 0 Ss (threaded) radiusd
100299 S kqread 0xfffff801b9520600 radiusd
100447 S usem 0xfffff801fbf0b380 radiusd
100448 S usem 0xfffff80009dc5c80 radiusd
100449 S usem 0xfffff801fbf0b680 radiusd
100450 S usem 0xfffff8015f1a9600 radiusd
100451 S usem 0xfffff80061904000 radiusd
94578 1 94578 0 Ss (threaded) snort
101223 S bpf 0xfffff801a75b7000 snort
100462 S nanslp 0xffffffff82866b32 snort
100463 S sbwait 0xfffff8012bb5a144 snort
23943 1 23943 0 Ss (threaded) dpinger
100197 S uwait 0xfffff8012bce0900 dpinger
101214 S sbwait 0xfffff801bc3d64a4 dpinger
101215 S nanslp 0xffffffff82866b31 dpinger
101216 S nanslp 0xffffffff82866b33 dpinger
101221 S accept 0xfffff800cb7bda8c dpinger
23331 1 23331 0 Ss (threaded) dpinger
100321 S uwait 0xfffff80090385200 dpinger
101181 S sbwait 0xfffff8012be6c4a4 dpinger
101211 S nanslp 0xffffffff82866b31 dpinger
101212 S nanslp 0xffffffff82866b32 dpinger
101213 S accept 0xfffff8015faf63cc dpinger
23238 1 23238 0 Ss (threaded) dpinger
100466 S uwait 0xfffff80009c1c280 dpinger
100347 S sbwait 0xfffff80090392b64 dpinger
100348 S nanslp 0xffffffff82866b32 dpinger
100349 S nanslp 0xffffffff82866b32 dpinger
101180 S accept 0xfffff8015f1053cc dpinger
23049 1 23049 0 Ss (threaded) dpinger
100130 S uwait 0xfffff80090387c00 dpinger
100334 S sbwait 0xfffff801b2fedb64 dpinger
100335 S nanslp 0xffffffff82866b33 dpinger
100336 S nanslp 0xffffffff82866b33 dpinger
100337 S accept 0xfffff800c9bae3cc dpinger
18455 1 11660 0 S select 0xfffff8015fa81140 dhcp6c
10995 96200 96200 0 S piperd 0xfffff8006b1ce8e8 cron
90119 1 90119 0 Ss kqread 0xfffff800bc831200 dhcpleases6
89812 1 89812 1002 Ss select 0xfffff8015f588140 dhcpd
88853 1 88853 1002 Ss select 0xfffff8015f1a3b40 dhcpd
79029 1 79029 59 Ss (threaded) unbound
100319 S kqread 0xfffff801811ce600 unbound
100430 S kqread 0xfffff8019614d500 unbound
100434 S kqread 0xfffff801a142ea00 unbound
100455 S kqread 0xfffff801cd747800 unbound
86160 1 298 0 S select 0xfffff801fa40dcc0 igmpproxy
59524 1 59524 0 Ss (threaded) filterdns
100324 S uwait 0xfffff8015f1a4180 chats.vpn.mrv.com
100325 S uwait 0xfffff8015f588000 bos.vpn.mrv.com
100326 S uwait 0xfffff800900f8d80 194.90.138.234
100327 S uwait 0xfffff80090138100 olpair.com
100328 S uwait 0xfffff8015fa76880 plex.tv
100329 S uwait 0xfffff8015fa76c80 plex.tv
100330 S uwait 0xfffff8015fa76e00 www.plex.tv
100331 S uwait 0xfffff8015fa76d00 SIP.xs4all.nl
100332 S uwait 0xfffff8012bbac200 signal-thread
1275 894 1275 0 S+ ttyin 0xfffff80009b334a8 sh
1067 695 1067 0 S+ ttyin 0xfffff800069e24a8 sh
894 99064 894 0 SW+ wait 0xfffff80009c99528 sh
695 202 695 0 SW+ wait 0xfffff801b4111000 sh
202 1 202 0 SWs+ wait 0xfffff80009deca50 login
154 1 154 0 Ss+ ttyin 0xfffff80009b8a0a8 getty
153 1 153 0 Ss+ ttyin 0xfffff80009b8a4a8 getty
99798 1 99798 0 Ss+ ttyin 0xfffff80009b8a8a8 getty
99556 1 99556 0 Ss+ ttyin 0xfffff800097c94a8 getty
99553 1 99553 0 Ss+ ttyin 0xfffff800097c98a8 getty
99312 1 99312 0 Ss+ ttyin 0xfffff800097c9ca8 getty
99239 1 99239 0 Ss+ ttyin 0xfffff80009b310a8 getty
99064 1 99064 0 SWs+ wait 0xfffff80009cb1a50 login
77672 1 77531 0 S kqread 0xfffff8015fabdc00 lighttpd_pfb
77073 76871 76871 0 S nanslp 0xffffffff82866b33 minicron
76871 1 76871 0 SWs wait 0xfffff801b4110000 minicron
76832 76765 76765 0 S nanslp 0xffffffff82866b30 minicron
76765 1 76765 0 SWs wait 0xfffff80009c99a50 minicron
76121 76090 76090 0 S nanslp 0xffffffff82866b31 minicron
76090 1 76090 0 SWs wait 0xfffff80009cb0000 minicron
62701 1 62701 0 Ss select 0xfffff8015f9a3440 syslogd
20127 1 20127 0 Ss select 0xfffff8015fa777c0 powerd
8847 8587 8847 0 Ss (threaded) charon
100133 S sigwait 0xfffff8015fa5b000 charon
100218 S select 0xfffff8015fd2b740 charon
100219 S uwait 0xfffff80009c1c480 charon
100220 S select 0xfffff8015fa627c0 charon
100221 S uwait 0xfffff80009c13680 charon
100222 S uwait 0xfffff80009c1c000 charon
100223 S uwait 0xfffff80009c13580 charon
100224 S uwait 0xfffff801178a5580 charon
100225 S uwait 0xfffff8015fd2b800 charon
100226 S uwait 0xfffff8015fd2b900 charon
100227 S uwait 0xfffff8015fd2ba00 charon
100228 S uwait 0xfffff8015fd2bb00 charon
100229 S uwait 0xfffff8015fd2bc00 charon
100230 S uwait 0xfffff8015fd2bd00 charon
100231 S uwait 0xfffff8015fd2be00 charon
100232 S uwait 0xfffff8015fd2c380 charon
100233 S uwait 0xfffff8015fd2c180 charon
8587 1 8587 0 Ss select 0xfffff8015fa628c0 starter
4647 1 4545 0 S select 0xfffff8015ef423c0 radvd
96807 1 96807 0 Ss (threaded) ntpd
100114 S select 0xfffff8015f1a55c0 ntpd
96200 1 96200 0 Ss nanslp 0xffffffff82866b30 cron
95969 95454 95454 0 S kqread 0xfffff8015f63cd00 nginx
95711 95454 95454 0 S kqread 0xfffff8015ed92200 nginx
95454 1 95454 0 SWs pause 0xfffff80009ec95d0 nginx
64781 1 64781 0 Ss select 0xfffff80149e54240 openvpn
61617 1 61617 0 Ss select 0xfffff80009c12540 openvpn
60126 1 60126 0 Ss select 0xfffff80149e55a40 openvpn
57281 1 57281 0 Ss select 0xfffff80009bd1540 openvpn
55041 1 55041 0 Ss select 0xfffff80009e6eb40 openvpn
46288 1 46288 65 Ss select 0xfffff8012bc58f40 dhclient
42347 1 42347 0 Ss select 0xfffff80090385740 dhclient
22284 1 22284 0 Ss bpf 0xfffff8012baf0400 filterlog
16036 1 16036 0 Ss (threaded) mpd5
100135 S select 0xfffff80009c1ce40 mpd5
7930 1 7930 0 Ss (threaded) sshlockout_pf
100127 S uwait 0xfffff80009c1ed80 sshlockout_pf
100131 S nanslp 0xffffffff82866b32 sshlockout_pf
7656 1 7656 0 Ss select 0xfffff80009c1e5c0 sshd
352 1 352 0 Ss select 0xfffff80009e6d740 devd
338 336 336 0 SW kqread 0xfffff80009c7e700 check_reload_status
336 1 336 0 Ss kqread 0xfffff80009b7e500 check_reload_status
298 1 298 0 Ss kqread 0xfffff80009ba8300 php-fpm
58 0 0 0 DL mdwait 0xfffff80009bd9800 [md0]
26 0 0 0 DL vlruwt 0xfffff80009b44528 [vnlru]
25 0 0 0 DL syncer 0xffffffff82960680 [syncer]
24 0 0 0 DL - 0xffffffff8295fbb4 [bufspacedaemon]
23 0 0 0 DL (threaded) [bufdaemon]
100081 D psleep 0xffffffff8295ef04 [bufdaemon]
100100 D sdflush 0xfffff80009bd6ce8 [/ worker]
22 0 0 0 DL pgzero 0xffffffff829742e4 [pagezero]
21 0 0 0 DL psleep 0xffffffff8297078c [vmdaemon]
20 0 0 0 DL (threaded) [pagedaemon]
100078 D psleep 0xffffffff82a1df05 [pagedaemon]
100085 D umarcl 0xffffffff829700b8 [uma]
19 0 0 0 DL - 0xffffffff8295e714 [soaiod4]
18 0 0 0 DL - 0xffffffff8295e714 [soaiod3]
9 0 0 0 DL - 0xffffffff8295e714 [soaiod2]
8 0 0 0 DL - 0xffffffff8295e714 [soaiod1]
7 0 0 0 DL - 0xffffffff8273b2f0 [rand_harvestq]
17 0 0 0 DL cooling 0xfffff800068b9d58 [acpi_cooling1]
16 0 0 0 DL tzpoll 0xffffffff82621f38 [acpi_thermal]
6 0 0 0 DL pftm 0xffffffff80f4c4f0 [pf purge]
5 0 0 0 DL waiting_ 0xffffffff82a0d540 [sctp_iterator]
15 0 0 0 DL (threaded) [usb]
100034 D - 0xfffffe0001121460 [usbus0]
100035 D - 0xfffffe00011214b8 [usbus0]
100036 D - 0xfffffe0001121510 [usbus0]
100037 D - 0xfffffe0001121568 [usbus0]
100038 D - 0xfffffe00011215c0 [usbus0]
100056 D - 0xfffffe0000f85d10 [usbus1]
100057 D - 0xfffffe0000f85d68 [usbus1]
100058 D - 0xfffffe0000f85dc0 [usbus1]
100059 D - 0xfffffe0000f85e18 [usbus1]
100060 D - 0xfffffe0000f85e70 [usbus1]
4 0 0 0 DL (threaded) [cam]
100029 D - 0xffffffff8260e180 [doneq0]
100073 D - 0xffffffff8260dfc8 [scanner]
3 0 0 0 DL crypto_r 0xffffffff8296ec70 [crypto returns]
2 0 0 0 DL crypto_w 0xffffffff8296eb18 [crypto]
14 0 0 0 DL (threaded) [geom]
100023 D - 0xffffffff829e5740 [g_event]
100024 D - 0xffffffff829e5748 [g_up]
100025 D - 0xffffffff829e5750 [g_down]
13 0 0 0 DL (threaded) [ng_queue]
100019 D sleep 0xffffffff825cbfd0 [ng_queue0]
100020 D sleep 0xffffffff825cbfd0 [ng_queue1]
100021 D sleep 0xffffffff825cbfd0 [ng_queue2]
100022 D sleep 0xffffffff825cbfd0 [ng_queue3]
12 0 0 0 RL (threaded) [intr]
100007 I [swi1: netisr 0]
100008 I [swi3: vm]
100009 I [swi4: clock (0)]
100010 I [swi4: clock (1)]
100011 I [swi4: clock (2)]
100012 I [swi4: clock (3)]
100015 I [swi5: fast taskq]
100017 I [swi6: task queue]
100018 I [swi6: Giant taskq]
100033 I [irq264: xhci0]
100039 I [irq265: hdac0]
100040 I [irq266: igb0:que 0]
100042 I [irq267: igb0:link]
100043 Run CPU 2 [irq268: igb1:que 0]
100045 I [irq269: igb1:link]
100046 I [irq270: igb2:que 0]
100048 I [irq271: igb2:link]
100051 I [irq272: iwn0]
100052 I [irq273: igb3:que 0]
100054 I [irq274: igb3:link]
100055 I [irq23: ehci0]
100061 I [irq275: ahci0]
100062 I [swi0: uart]
100063 I [irq1: atkbd0]
100067 I [swi1: pf send]
100068 I [swi1: pfsync]
100086 I [swi1: netisr 1]
100087 I [swi1: netisr 2]
100088 Run CPU 0 [swi1: netisr 3]
11 0 0 0 RL (threaded) [idle]
100003 CanRun [idle: cpu0]
100004 Run CPU 1 [idle: cpu1]
100005 CanRun [idle: cpu2]
100006 Run CPU 3 [idle: cpu3]
1 0 1 0 SLs wait 0xfffff80006217528 [init]
10 0 0 0 DL audit_wo 0xffffffff82a146e0 [audit]
0 0 0 0 DLs (threaded) [kernel]
100000 D swapin 0xffffffff829e5778 [swapper]
100013 D - 0xfffff8000623f100 [aiod_kick taskq]
100014 D - 0xfffff8000623ee00 -
in my case when i use num_queues it instantly crashes on HFSC, without it im able to use till i dont go over 50mbps
-
I would test on 2.4.1.
The igb driver on FreeBSD 11.0 (pfSense 2,4) is missing 'hw.igb.buf_ring_size' tunable so setting it on 2.4 probably has no affect, and igb had some bugs which are fixed in FreeBSD 11.1 (pfSense 2.4.1.). The sysctl 'hw.igb.buf_ring_size' is also reinstated.
-
I started getting one or two crash reboot a day on 2.4RC. I have submitted every one of them. Here is the latest:
Crash report begins. Anonymous machine information: amd64 11.0-RELEASE-p12 FreeBSD 11.0-RELEASE-p12 #58 daf8d77a4c1(RELENG_2_4_0): Fri Sep 22 20:48:22 CDT 2017 root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense Crash report details: No PHP errors found. Filename: /var/crash/bounds 1 Filename: /var/crash/info.0 Dump header from device: /dev/gptid/09925e84-90d4-11e7-84bc-80ee73c2ffc4 Architecture: amd64 Architecture Version: 1 Dump Length: 79360 Blocksize: 512 Dumptime: Sat Sep 23 12:45:13 2017 Hostname: alix1.nowayout.ca Magic: FreeBSD Text Dump Version String: FreeBSD 11.0-RELEASE-p12 #58 daf8d77a4c1(RELENG_2_4_0): Fri Sep 22 20:48:22 CDT 2017 root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense Panic String: hfsc_dequeue: Dump Parity: 3690452271 Bounds: 0 Dump Status: good Filename: /var/crash/info.last Dump header from device: /dev/gptid/09925e84-90d4-11e7-84bc-80ee73c2ffc4 Architecture: amd64 Architecture Version: 1 Dump Length: 79360 Blocksize: 512 Dumptime: Sat Sep 23 12:45:13 2017 Hostname: alix1.nowayout.ca Magic: FreeBSD Text Dump Version String: FreeBSD 11.0-RELEASE-p12 #58 daf8d77a4c1(RELENG_2_4_0): Fri Sep 22 20:48:22 CDT 2017 root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense Panic String: hfsc_dequeue: Dump Parity: 3690452271 Bounds: 0 Dump Status: good Filename: /var/crash/minfree 2048 Filename: /var/crash/textdump.tar.0 ddb.txt06000014000013161507631 7075 ustarrootwheeldb:0:kdb.enter.default> run lockinfo db:1:lockinfo> show locks No such command db:1:locks> show alllocks No such command db:1:alllocks> show lockedvnods Locked vnodes db:0:kdb.enter.default> show pcpu cpuid = 1 dynamic pcpu = 0xfffffe0170be4200 curthread = 0xfffff80003983000: pid 0 "em0 que" curpcb = 0xfffffe0117d60c80 fpcurthread = none idlethread = 0xfffff8000395fa00: tid 100004 "idle: cpu1" curpmap = 0xffffffff829e6300 tssp = 0xffffffff82a1eb78 commontssp = 0xffffffff82a1eb78 rsp0 = 0xfffffe0117d60c80 gs32p = 0xffffffff82a253d0 ldt = 0xffffffff82a25410 tss = 0xffffffff82a25400 db:0:kdb.enter.default> bt Tracing pid 0 tid 100027 td 0xfffff80003983000 kdb_enter() at kdb_enter+0x3b/frame 0xfffffe0117d60370 vpanic() at vpanic+0x19f/frame 0xfffffe0117d603f0 panic() at panic+0x43/frame 0xfffffe0117d60450 hfsc_dequeue() at hfsc_dequeue+0x633/frame 0xfffffe0117d60490 tbr_dequeue() at tbr_dequeue+0xdf/frame 0xfffffe0117d604e0 vlan_start() at vlan_start+0x12c/frame 0xfffffe0117d60540 if_transmit() at if_transmit+0x1a5/frame 0xfffffe0117d60590 ether_output() at ether_output+0x728/frame 0xfffffe0117d60620 ip_tryforward() at ip_tryforward+0x726/frame 0xfffffe0117d606a0 ip_input() at ip_input+0x3a4/frame 0xfffffe0117d60700 netisr_dispatch_src() at netisr_dispatch_src+0xa5/frame 0xfffffe0117d60760 ether_demux() at ether_demux+0x15c/frame 0xfffffe0117d60790 ether_nh_input() at ether_nh_input+0x317/frame 0xfffffe0117d607f0 netisr_dispatch_src() at netisr_dispatch_src+0xa5/frame 0xfffffe0117d60850 ether_input() at ether_input+0x26/frame 0xfffffe0117d60870 vlan_input() at vlan_input+0x1cb/frame 0xfffffe0117d608f0 ether_demux() at ether_demux+0xa9/frame 0xfffffe0117d60920 ether_nh_input() at ether_nh_input+0x317/frame 0xfffffe0117d60980 netisr_dispatch_src() at netisr_dispatch_src+0xa5/frame 0xfffffe0117d609e0 ether_input() at ether_input+0x26/frame 0xfffffe0117d60a00 if_input() at if_input+0xa/frame 0xfffffe0117d60a10 em_rxeof() at em_rxeof+0x352/frame 0xfffffe0117d60a90 em_handle_que() at em_handle_que+0x43/frame 0xfffffe0117d60ae0 taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame 0xfffffe0117d60b40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame 0xfffffe0117d60b70 fork_exit() at fork_exit+0x85/frame 0xfffffe0117d60bb0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0117d60bb0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db:0:kdb.enter.default> ps pid ppid pgrp uid state wmesg wchan cmd 82872 90006 552 0 S nanslp 0xffffffff82866b30 sleep 80573 552 552 0 S accept 0xfffff8000e29172c php-fpm 67765 1 67616 558 S select 0xfffff8000eae9f40 avahi-daemon 67480 1 552 0 S nanslp 0xffffffff82866b31 php 63593 1 63593 556 Ss select 0xfffff800616e2d40 dbus-daemon 90006 1 552 0 S wait 0xfffff8000e137000 sh 89044 1 89044 136 Ss select 0xfffff80061027dc0 dhcpd 76084 1 76084 59 Ss (threaded) unbound 100251 S kqread 0xfffff8000e6f3500 unbound 100272 S kqread 0xfffff8000e964700 unbound 53080 1 53080 0 Ss (threaded) dpinger 100157 S uwait 0xfffff8000e2b2500 dpinger 100268 S sbwait 0xfffff80085533144 dpinger 100269 S nanslp 0xffffffff82866b30 dpinger 100270 S nanslp 0xffffffff82866b31 dpinger 100271 S accept 0xfffff8008550f72c dpinger 52697 1 52697 0 Ss (threaded) dpinger 100218 S uwait 0xfffff8000ea68080 dpinger 100264 S sbwait 0xfffff8007b0e7144 dpinger 100265 S nanslp 0xffffffff82866b31 dpinger 100266 S nanslp 0xffffffff82866b31 dpinger 100267 S accept 0xfffff80061ad93cc dpinger 51997 1 51997 0 Ss (threaded) dpinger 100142 S uwait 0xfffff8000e027780 dpinger 100260 S sbwait 0xfffff800855084a4 dpinger 100261 S nanslp 0xffffffff82866b30 dpinger 100262 S nanslp 0xffffffff82866b31 dpinger 100263 S accept 0xfffff8001460aa8c dpinger 6177 1 6177 0 Rs CPU 0 openvpn 57344 1 57344 0 Ss+ ttyin 0xfffff8000badc8a8 getty 57229 1 57229 0 Ss+ ttyin 0xfffff8000badcca8 getty 56954 1 56954 0 Ss+ ttyin 0xfffff8000badd8a8 getty 56790 1 56790 0 Ss+ ttyin 0xfffff8000badd0a8 getty 56606 1 56606 0 Ss+ ttyin 0xfffff8000badd4a8 getty 56492 1 56492 0 Ss+ ttyin 0xfffff8000badf4a8 getty 56169 1 56169 0 Ss+ ttyin 0xfffff8000badf8a8 getty 56044 1 56044 0 Ss+ ttyin 0xfffff8000badfca8 getty 9047 8128 8128 0 S nanslp 0xffffffff82866b31 minicron 8128 1 8128 0 Ss wait 0xfffff800856a8528 minicron 7894 7188 7188 0 S nanslp 0xffffffff82866b30 minicron 7188 1 7188 0 Ss wait 0xfffff80021a3b000 minicron 6870 6432 6432 0 S nanslp 0xffffffff82866b31 minicron 6432 1 6432 0 Ss wait 0xfffff80085203528 minicron 6187 1 6187 0 Ss (threaded) filterdns 100227 S uwait 0xfffff80061029900 ns1.he.net 100228 S uwait 0xfffff8006107b000 ns2.he.net 100229 S uwait 0xfffff8006107a400 ns3.he.net 100230 S uwait 0xfffff8000ee31d00 ns4.he.net 100231 S uwait 0xfffff80061734d00 ns5.he.net 100232 S uwait 0xfffff80061736600 filterdns 100233 S uwait 0xfffff800616ffd80 filterdns 100234 S uwait 0xfffff8006171e680 filterdns 100235 S uwait 0xfffff8006171ff00 filterdns 100236 S uwait 0xfffff800616b1d80 filterdns 100237 S uwait 0xfffff8000efb3300 signal-thread 2653 1 2358 0 S kqread 0xfffff8000e64d300 lighttpd_pfb 82428 1 82428 556 Ss select 0xfffff80061734f40 dbus-daemon 70117 1 70117 0 Ss select 0xfffff8000e663b40 syslogd 42738 1 42738 0 Ss select 0xfffff8000ee5b6c0 miniupnpd 37886 1 37886 0 Ss select 0xfffff8000e91b0c0 powerd 70983 1 70983 0 Ss (threaded) ntpd 100203 S select 0xfffff8000ee04540 ntpd 100205 S usem 0xfffff8000e663680 ntpd 70411 1 70411 0 Ss nanslp 0xffffffff82866b30 cron 69785 69275 69275 0 S kqread 0xfffff8000ec6f200 nginx 69509 69275 69275 0 S kqread 0xfffff8000ec8ae00 nginx 69275 1 69275 0 Ss pause 0xfffff8000e1260a8 nginx 48784 1 48784 0 Ss bpf 0xfffff80061192800 filterlog 44364 1 44364 0 Ss select 0xfffff800612d47c0 openvpn 29529 1 29529 65 Ss select 0xfffff8000e662c40 dhclient 24129 1 24129 0 Ss select 0xfffff8000e5e88c0 dhclient 16032 1 16032 0 Ss (threaded) sshlockout_pf 100143 S uwait 0xfffff8000e027880 sshlockout_pf 100156 S nanslp 0xffffffff82866b30 sshlockout_pf 15937 1 15937 0 Ss select 0xfffff80061663340 sshd 580 1 580 0 Ss select 0xfffff800616032c0 devd 568 566 566 0 S kqread 0xfffff800611b5200 check_reload_status 566 1 566 0 Ss kqread 0xfffff80061149e00 check_reload_status 552 1 552 0 Ss kqread 0xfffff800611b4700 php-fpm 59 0 0 0 DL mdwait 0xfffff80003bba800 [md0] 27 0 0 0 DL syncer 0xffffffff82960680 [syncer] 26 0 0 0 DL vlruwt 0xfffff8000bb1e528 [vnlru] 25 0 0 0 DL (threaded) [bufdaemon] 100120 D psleep 0xffffffff8295ef04 [bufdaemon] 100132 D sdflush 0xfffff8000bbb28e8 [/ worker] 24 0 0 0 DL - 0xffffffff8295fbb4 [bufspacedaemon] 23 0 0 0 DL pgzero 0xffffffff829742e4 [pagezero] 22 0 0 0 DL psleep 0xffffffff8297078c [vmdaemon] 21 0 0 0 DL (threaded) [pagedaemon] 100116 D psleep 0xffffffff82a1df05 [pagedaemon] 100126 D umarcl 0xffffffff829700b8 [uma] 20 0 0 0 DL idle 0xfffff8000bb20528 [enc_daemon0] 19 0 0 0 DL - 0xffffffff8295e714 [soaiod4] 18 0 0 0 DL - 0xffffffff8295e714 [soaiod3] 17 0 0 0 DL - 0xffffffff8295e714 [soaiod2] 9 0 0 0 DL - 0xffffffff8295e714 [soaiod1] 16 0 0 0 DL tzpoll 0xffffffff82621f38 [acpi_thermal] 8 0 0 0 DL - 0xffffffff8273b2f0 [rand_harvestq] 7 0 0 0 DL pftm 0xffffffff80f4c510 [pf purge] 6 0 0 0 DL waiting_ 0xffffffff82a0d540 [sctp_iterator] 5 0 0 0 SL - 0xfffffe0000fde000 [fw0_probe] 15 0 0 0 DL (threaded) [usb] 100030 D - 0xfffffe0000f22de8 [usbus0] 100031 D - 0xfffffe0000f22e40 [usbus0] 100032 D - 0xfffffe0000f22e98 [usbus0] 100033 D - 0xfffffe0000f22ef0 [usbus0] 100034 D - 0xfffffe0000f22f48 [usbus0] 100036 D - 0xfffffe0000f3ede8 [usbus1] 100037 D - 0xfffffe0000f3ee40 [usbus1] 100038 D - 0xfffffe0000f3ee98 [usbus1] 100039 D - 0xfffffe0000f3eef0 [usbus1] 100040 D - 0xfffffe0000f3ef48 [usbus1] 100042 D - 0xfffffe0000f4ede8 [usbus2] 100043 D - 0xfffffe0000f4ee40 [usbus2] 100044 D - 0xfffffe0000f4ee98 [usbus2] 100045 D - 0xfffffe0000f4eef0 [usbus2] 100046 D - 0xfffffe0000f4ef48 [usbus2] 100047 D - 0xfffffe0000f5ed10 [usbus3] 100048 D - 0xfffffe0000f5ed68 [usbus3] 100049 D - 0xfffffe0000f5edc0 [usbus3] 100050 D - 0xfffffe0000f5ee18 [usbus3] 100051 D - 0xfffffe0000f5ee70 [usbus3] 100059 D - 0xfffffe0000f6ede8 [usbus4] 100060 D - 0xfffffe0000f6ee40 [usbus4] 100061 D - 0xfffffe0000f6ee98 [usbus4] 100062 D - 0xfffffe0000f6eef0 [usbus4] 100063 D - 0xfffffe0000f6ef48 [usbus4] 100064 D - 0xfffffe0000fb4de8 [usbus5] 100065 D - 0xfffffe0000fb4e40 [usbus5] 100066 D - 0xfffffe0000fb4e98 [usbus5] 100067 D - 0xfffffe0000fb4ef0 [usbus5] 100068 D - 0xfffffe0000fb4f48 [usbus5] 100069 D - 0xfffffe0000fc4de8 [usbus6] 100070 D - 0xfffffe0000fc4e40 [usbus6] 100071 D - 0xfffffe0000fc4e98 [usbus6] 100072 D - 0xfffffe0000fc4ef0 [usbus6] 100073 D - 0xfffffe0000fc4f48 [usbus6] 100074 D - 0xfffffe0000fd4d10 [usbus7] 100075 D - 0xfffffe0000fd4d68 [usbus7] 100076 D - 0xfffffe0000fd4dc0 [usbus7] 100077 D - 0xfffffe0000fd4e18 [usbus7] 100078 D - 0xfffffe0000fd4e70 [usbus7] 4 0 0 0 DL (threaded) [cam] 100026 D - 0xffffffff8260e180 [doneq0] 100110 D - 0xffffffff8260dfc8 [scanner] 3 0 0 0 DL crypto_r 0xffffffff8296ec70 [crypto returns] 2 0 0 0 DL crypto_w 0xffffffff8296eb18 [crypto] 14 0 0 0 DL (threaded) [geom] 100017 D - 0xffffffff829e5740 [g_event] 100018 D - 0xffffffff829e5748 [g_up] 100019 D - 0xffffffff829e5750 [g_down] 13 0 0 0 DL (threaded) [ng_queue] 100015 D sleep 0xffffffff825cbfd0 [ng_queue0] 100016 D sleep 0xffffffff825cbfd0 [ng_queue1] 12 0 0 0 WL (threaded) [intr] 100005 I [swi3: vm] 100006 I [swi4: clock (0)] 100007 I [swi4: clock (1)] 100008 I [swi1: netisr 0] 100009 I [swi6: task queue] 100010 I [swi6: Giant taskq] 100013 I [swi5: fast taskq] 100029 I [irq20: hpet0 uhci0*] 100035 I [irq21: uhci1 uhci4] 100041 I [irq22: uhci2 ehci0+] 100052 I [irq257: hdac0] 100053 I [irq258: pcib1] 100056 I [irq259: iwn0] 100057 I [irq260: pcib2] 100058 I [irq261: pcib3] 100079 I [irq17: fwohci0] 100082 I [irq18: sdhci_pci0+] 100083 I [irq262: ahci0:ch0] 100084 I [irq263: ahci0:ch1] 100085 I [irq264: ahci0:ch2] 100086 I [irq265: ahci0:ch3] 100087 I [irq266: ahci0:ch4] 100088 I [irq267: ahci0:ch5] 100089 I [irq268: ahci0:6] 100090 I [irq269: ahci0:7] 100091 I [irq270: ahci0:8] 100092 I [irq271: ahci0:9] 100093 I [irq272: ahci0:10] 100094 I [irq273: ahci0:11] 100095 I [irq274: ahci0:12] 100096 I [irq275: ahci0:13] 100097 I [irq276: ahci0:14] 100098 I [irq277: ahci0:15] 100099 I [irq1: atkbd0] 100100 I [irq12: psm0] 100101 I [swi0: uart] 100105 I [swi1: pf send] 100106 I [swi1: pfsync] 100123 I [swi1: netisr 1] 11 0 0 0 RL (threaded) [idle] 100003 CanRun [idle: cpu0] 100004 CanRun [idle: cpu1] 1 0 1 0 SLs wait 0xfffff8000395d528 [init] 10 0 0 0 DL audit_wo 0xffffffff82a146e0 [audit] 0 0 0 0 RLs (threaded) [kernel] 100000 D swapin 0xffffffff829e5778 [swapper] 100011 D - 0xfffff8000394d000 [aiod_kick taskq] 100012 D - 0xfffff8000394cd00 [thread] 100014 D - 0xfffff8000394c800 [kqueue_ctx taskq] 100020 D - 0xfffff80003989d00 [firmware taskq] 100023 D - 0xfffff80003988600 [acpi_task_0] 100024 D - 0xfffff80003988600 [acpi_task_1] 100025 D - 0xfffff80003988600 [acpi_task_2] 100027 Run CPU 1 [em0 que] 100028 D - 0xfffff80003ba7700 [em0 txq] 100054 D - 0xfffff8000516a500 [iwn0 net80211 taskq] 100055 D - 0xfffff8000516a300 [iwn_taskq] 100080 D - 0xfffff80003bf9600 [fw0_taskq] 100102 D - 0xfffff8000b098a00 [mca taskq] 100109 D - 0xfffff80003988200 [CAM taskq] 100124 D - 0xfffff8000baf5d00 [if_config_tqg_0] 100125 D - 0xfffff8000bb79b00 [if_io_tqg_0] 100127 D - 0xfffff8000bb79900 [if_io_tqg_1] db:0:kdb.enter.default> alltrace Tracing command sleep pid 82872 tid 100226 td 0xfffff8008572a500 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f068880 mi_switch() at mi_switch+0xd2/frame 0xfffffe011f0688b0 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f068930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011f068960 _sleep() at _sleep+0x26f/frame 0xfffffe011f0689f0 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011f068a50 sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011f068a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f068bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f068bb0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8009008fa, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 --- Tracing command php-fpm pid 80573 tid 100225 td 0xfffff8008572aa00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f063810 mi_switch() at mi_switch+0xd2/frame 0xfffffe011f063840 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0638c0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011f0638f0 _sleep() at _sleep+0x294/frame 0xfffffe011f063980 kern_accept4() at kern_accept4+0x1f2/frame 0xfffffe011f063a30 accept1() at accept1+0x60/frame 0xfffffe011f063a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f063bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f063bb0 --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x801d939ba, rsp = 0x7ffffffec848, rbp = 0x7ffffffec880 --- Tracing command avahi-daemon pid 67765 tid 100255 td 0xfffff80021e6fa00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f0f4730 mi_switch() at mi_switch+0xd2/frame 0xfffffe011f0f4760 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0f47e0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011f0f4810 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe011f0f4880 seltdwait() at seltdwait+0xc7/frame 0xfffffe011f0f48d0 kern_poll() at kern_poll+0x296/frame 0xfffffe011f0f4a70 sys_poll() at sys_poll+0x61/frame 0xfffffe011f0f4a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f0f4bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f0f4bb0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x801814f7a, rsp = 0x7fffffffe7d8, rbp = 0x7fffffffe810 --- Tracing command php pid 67480 tid 100254 td 0xfffff80021e86000 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f0ef880 mi_switch() at mi_switch+0xd2/frame 0xfffffe011f0ef8b0 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0ef930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011f0ef960 _sleep() at _sleep+0x26f/frame 0xfffffe011f0ef9f0 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011f0efa50 sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011f0efa90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f0efbb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f0efbb0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x801d848fa, rsp = 0x7fffffffc7c8, rbp = 0x7fffffffc7f0 --- Tracing command dbus-daemon pid 63593 tid 100250 td 0xfffff80021e87500 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f0db750 mi_switch() at mi_switch+0xd2/frame 0xfffffe011f0db780 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0db800 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011f0db830 _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011f0db880 seltdwait() at seltdwait+0x10f/frame 0xfffffe011f0db8d0 kern_poll() at kern_poll+0x296/frame 0xfffffe011f0dba70 sys_poll() at sys_poll+0x61/frame 0xfffffe011f0dba90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f0dbbb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f0dbbb0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8011dcf7a, rsp = 0x7fffffffe858, rbp = 0x7fffffffe890 --- Tracing command sh pid 90006 tid 100146 td 0xfffff8000e0d7000 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef4d660 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef4d690 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef4d710 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef4d740 _sleep() at _sleep+0x294/frame 0xfffffe011ef4d7d0 kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe011ef4d8a0 sys_wait4() at sys_wait4+0x73/frame 0xfffffe011ef4da90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef4dbb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef4dbb0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800b5d75a, rsp = 0x7fffffffe868, rbp = 0x7fffffffe920 --- Tracing command dhcpd pid 89044 tid 100148 td 0xfffff8000e13b500 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef576e0 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef57710 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef57790 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef577c0 _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011ef57810 seltdwait() at seltdwait+0x10f/frame 0xfffffe011ef57860 kern_select() at kern_select+0x979/frame 0xfffffe011ef57a50 sys_select() at sys_select+0x54/frame 0xfffffe011ef57a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef57bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef57bb0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800d641ca, rsp = 0x7fffffffea08, rbp = 0x7fffffffea30 --- Tracing command unbound pid 76084 tid 100251 td 0xfffff80021e87000 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f0e0620 mi_switch() at mi_switch+0xd2/frame 0xfffffe011f0e0650 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0e06d0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011f0e0700 _sleep() at _sleep+0x26f/frame 0xfffffe011f0e0790 kqueue_kevent() at kqueue_kevent+0x336/frame 0xfffffe011f0e0970 kern_kevent() at kern_kevent+0xb5/frame 0xfffffe011f0e09e0 sys_kevent() at sys_kevent+0x11c/frame 0xfffffe011f0e0a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f0e0bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f0e0bb0 --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x801b7293a, rsp = 0x7fffffffea18, rbp = 0x7fffffffea60 --- Tracing command unbound pid 76084 tid 100272 td 0xfffff800854e0a00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f103620 mi_switch() at mi_switch+0xd2/frame 0xfffffe011f103650 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f1036d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011f103700 _sleep() at _sleep+0x294/frame 0xfffffe011f103790 kqueue_kevent() at kqueue_kevent+0x336/frame 0xfffffe011f103970 kern_kevent() at kern_kevent+0xb5/frame 0xfffffe011f1039e0 sys_kevent() at sys_kevent+0x11c/frame 0xfffffe011f103a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f103bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f103bb0 --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x801b7293a, rsp = 0x7fffdfffde18, rbp = 0x7fffdfffde60 --- Tracing command dpinger pid 53080 tid 100157 td 0xfffff8007b0e5500 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef847f0 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef84820 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef848a0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef848d0 _sleep() at _sleep+0x294/frame 0xfffffe011ef84960 umtxq_sleep() at umtxq_sleep+0x122/frame 0xfffffe011ef849c0 do_wait() at do_wait+0x419/frame 0xfffffe011ef84a50 __umtx_op_wait() at __umtx_op_wait+0x77/frame 0xfffffe011ef84a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef84bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef84bb0 --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083c67c, rsp = 0x7fffffffe8f8, rbp = 0x7fffffffe980 --- Tracing command dpinger pid 53080 tid 100268 td 0xfffff800854df500 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efb16f0 mi_switch() at mi_switch+0xd2/frame 0xfffffe011efb1720 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efb17a0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011efb17d0 _sleep() at _sleep+0x294/frame 0xfffffe011efb1860 sbwait() at sbwait+0x4f/frame 0xfffffe011efb1880 soreceive_generic() at soreceive_generic+0x1ae/frame 0xfffffe011efb1950 kern_recvit() at kern_recvit+0x1d6/frame 0xfffffe011efb1a20 sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe011efb1a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efb1bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efb1bb0 --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b3cf5a, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 --- Tracing command dpinger pid 53080 tid 100269 td 0xfffff800854df000 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efb6880 mi_switch() at mi_switch+0xd2/frame 0xfffffe011efb68b0 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efb6930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011efb6960 _sleep() at _sleep+0x26f/frame 0xfffffe011efb69f0 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011efb6a50 sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011efb6a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efb6bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efb6bb0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2b8fa, rsp = 0x7fffdfdfcf58, rbp = 0x7fffdfdfcf80 --- Tracing command dpinger pid 53080 tid 100270 td 0xfffff800c31cfa00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efbb880 mi_switch() at mi_switch+0xd2/frame 0xfffffe011efbb8b0 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efbb930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011efbb960 _sleep() at _sleep+0x26f/frame 0xfffffe011efbb9f0 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011efbba50 sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011efbba90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efbbbb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efbbbb0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2b8fa, rsp = 0x7fffdfbfbef8, rbp = 0x7fffdfbfbf20 --- Tracing command dpinger pid 53080 tid 100271 td 0xfffff800c31cf500 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efc0810 mi_switch() at mi_switch+0xd2/frame 0xfffffe011efc0840 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efc08c0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011efc08f0 _sleep() at _sleep+0x294/frame 0xfffffe011efc0980 kern_accept4() at kern_accept4+0x1f2/frame 0xfffffe011efc0a30 accept1() at accept1+0xf6/frame 0xfffffe011efc0a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efc0bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efc0bb0 --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x800b2b9ba, rsp = 0x7fffdf9fae78, rbp = 0x7fffdf9faeb0 --- Tracing command dpinger pid 52697 tid 100218 td 0xfffff8007b294000 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f0407f0 mi_switch() at mi_switch+0xd2/frame 0xfffffe011f040820 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0408a0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011f0408d0 _sleep() at _sleep+0x294/frame 0xfffffe011f040960 umtxq_sleep() at umtxq_sleep+0x122/frame 0xfffffe011f0409c0 do_wait() at do_wait+0x419/frame 0xfffffe011f040a50 __umtx_op_wait() at __umtx_op_wait+0x77/frame 0xfffffe011f040a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f040bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f040bb0 --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083c67c, rsp = 0x7fffffffe908, rbp = 0x7fffffffe990 --- Tracing command dpinger pid 52697 tid 100264 td 0xfffff8007b24aa00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef936f0 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef93720 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef937a0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef937d0 _sleep() at _sleep+0x294/frame 0xfffffe011ef93860 sbwait() at sbwait+0x4f/frame 0xfffffe011ef93880 soreceive_generic() at soreceive_generic+0x1ae/frame 0xfffffe011ef93950 kern_recvit() at kern_recvit+0x1d6/frame 0xfffffe011ef93a20 sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe011ef93a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef93bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef93bb0 --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b3cf5a, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 --- Tracing command dpinger pid 52697 tid 100265 td 0xfffff8000bb7da00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef98880 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef988b0 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef98930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011ef98960 _sleep() at _sleep+0x26f/frame 0xfffffe011ef989f0 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011ef98a50 sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011ef98a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef98bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef98bb0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2b8fa, rsp = 0x7fffdfdfcf58, rbp = 0x7fffdfdfcf80 --- Tracing command dpinger pid 52697 tid 100266 td 0xfffff8007b0e4000 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef9d880 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef9d8b0 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef9d930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011ef9d960 _sleep() at _sleep+0x26f/frame 0xfffffe011ef9d9f0 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011ef9da50 sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011ef9da90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef9dbb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef9dbb0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2b8fa, rsp = 0x7fffdfbfbef8, rbp = 0x7fffdfbfbf20 --- Tracing command dpinger pid 52697 tid 100267 td 0xfffff8007b13ba00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efa2810 mi_switch() at mi_switch+0xd2/frame 0xfffffe011efa2840 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efa28c0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011efa28f0 _sleep() at _sleep+0x294/frame 0xfffffe011efa2980 kern_accept4() at kern_accept4+0x1f2/frame 0xfffffe011efa2a30 accept1() at accept1+0xf6/frame 0xfffffe011efa2a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efa2bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efa2bb0 --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x800b2b9ba, rsp = 0x7fffdf9fae78, rbp = 0x7fffdf9faeb0 --- Tracing command dpinger pid 51997 tid 100142 td 0xfffff8000e121500 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef397f0 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef39820 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef398a0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef398d0 _sleep() at _sleep+0x294/frame 0xfffffe011ef39960 umtxq_sleep() at umtxq_sleep+0x122/frame 0xfffffe011ef399c0 do_wait() at do_wait+0x419/frame 0xfffffe011ef39a50 __umtx_op_wait() at __umtx_op_wait+0x77/frame 0xfffffe011ef39a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef39bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef39bb0 --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083c67c, rsp = 0x7fffffffe908, rbp = 0x7fffffffe990 --- Tracing command dpinger pid 51997 tid 100260 td 0xfffff8007b196500 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efd96f0 mi_switch() at mi_switch+0xd2/frame 0xfffffe011efd9720 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efd97a0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011efd97d0 _sleep() at _sleep+0x294/frame 0xfffffe011efd9860 sbwait() at sbwait+0x4f/frame 0xfffffe011efd9880 soreceive_generic() at soreceive_generic+0x1ae/frame 0xfffffe011efd9950 kern_recvit() at kern_recvit+0x1d6/frame 0xfffffe011efd9a20 sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe011efd9a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efd9bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efd9bb0 --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b3cf5a, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 --- Tracing command dpinger pid 51997 tid 100261 td 0xfffff8007b19fa00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efde880 mi_switch() at mi_switch+0xd2/frame 0xfffffe011efde8b0 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efde930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011efde960 _sleep() at _sleep+0x26f/frame 0xfffffe011efde9f0 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011efdea50 sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011efdea90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efdebb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efdebb0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2b8fa, rsp = 0x7fffdfdfcf58, rbp = 0x7fffdfdfcf80 --- Tracing command dpinger pid 51997 tid 100262 td 0xfffff800854e0000 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efe3880 mi_switch() at mi_switch+0xd2/frame 0xfffffe011efe38b0 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efe3930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011efe3960 _sleep() at _sleep+0x26f/frame 0xfffffe011efe39f0 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011efe3a50 sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011efe3a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efe3bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efe3bb0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2b8fa, rsp = 0x7fffdfbfbef8, rbp = 0x7fffdfbfbf20 --- Tracing command dpinger pid 51997 tid 100263 td 0xfffff800854dfa00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efe8810 mi_switch() at mi_switch+0xd2/frame 0xfffffe011efe8840 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efe88c0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011efe88f0 _sleep() at _sleep+0x294/frame 0xfffffe011efe8980 kern_accept4() at kern_accept4+0x1f2/frame 0xfffffe011efe8a30 accept1() at accept1+0xf6/frame 0xfffffe011efe8a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efe8bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efe8bb0 --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x800b2b9ba, rsp = 0x7fffdf9fae78, rbp = 0x7fffdf9faeb0 --- Tracing command openvpn pid 6177 tid 100241 td 0xfffff80021dc1500 cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff82978530 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xffffffff82978550 trap() at trap+0x3a/frame 0xffffffff82978760 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff82978760 --- trap 0x13, rip = 0xffffffff80cf8666, rsp = 0xfffffe011f0ae880, rbp = 0xfffffe011f0ae8d0 --- sopoll_generic() at sopoll_generic+0x46/frame 0xfffffe011f0ae8d0 kern_poll() at kern_poll+0x650/frame 0xfffffe011f0aea70 sys_poll() at sys_poll+0x61/frame 0xfffffe011f0aea90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f0aebb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f0aebb0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x80155cf7a, rsp = 0x7fffffffdd78, rbp = 0x7fffffffdde0 --- Tracing command getty pid 57344 tid 100150 td 0xfffff8000e13aa00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef61760 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef61790 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef61810 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef61840 _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011ef61890 tty_wait() at tty_wait+0x1c/frame 0xfffffe011ef618b0 ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe011ef61910 ttydev_read() at ttydev_read+0x34/frame 0xfffffe011ef61940 devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe011ef619a0 dofileread() at dofileread+0x98/frame 0xfffffe011ef619f0 kern_readv() at kern_readv+0x68/frame 0xfffffe011ef61a40 sys_read() at sys_read+0x84/frame 0xfffffe011ef61a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef61bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef61bb0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 --- Tracing command getty pid 57229 tid 100151 td 0xfffff8000e13a500 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef66760 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef66790 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef66810 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef66840 _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011ef66890 tty_wait() at tty_wait+0x1c/frame 0xfffffe011ef668b0 ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe011ef66910 ttydev_read() at ttydev_read+0x34/frame 0xfffffe011ef66940 devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe011ef669a0 dofileread() at dofileread+0x98/frame 0xfffffe011ef669f0 kern_readv() at kern_readv+0x68/frame 0xfffffe011ef66a40 sys_read() at sys_read+0x84/frame 0xfffffe011ef66a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef66bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef66bb0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 --- Tracing command getty pid 56954 tid 100240 td 0xfffff80021e87a00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f0a9760 mi_switch() at mi_switch+0xd2/frame 0xfffffe011f0a9790 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0a9810 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011f0a9840 _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011f0a9890 tty_wait() at tty_wait+0x1c/frame 0xfffffe011f0a98b0 ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe011f0a9910 ttydev_read() at ttydev_read+0x34/frame 0xfffffe011f0a9940 devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe011f0a99a0 dofileread() at dofileread+0x98/frame 0xfffffe011f0a99f0 kern_readv() at kern_readv+0x68/frame 0xfffffe011f0a9a40 sys_read() at sys_read+0x84/frame 0xfffffe011f0a9a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f0a9bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f0a9bb0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 --- Tracing command getty pid 56790 tid 100135 td 0xfffff8000e040a00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef16760 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef16790 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef16810 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef16840 _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011ef16890 tty_wait() at tty_wait+0x1c/frame 0xfffffe011ef168b0 ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe011ef16910 ttydev_read() at ttydev_read+0x34/frame 0xfffffe011ef16940 devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe011ef169a0 dofileread() at dofileread+0x98/frame 0xfffffe011ef169f0 kern_readv() at kern_readv+0x68/frame 0xfffffe011ef16a40 sys_read() at sys_read+0x84/frame 0xfffffe011ef16a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef16bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef16bb0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 --- Tracing command getty pid 56606 tid 100224 td 0xfffff8008572b000 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f05e760 mi_switch() at mi_switch+0xd2/f[/thread] -
I would test on 2.4.1.
The igb driver on FreeBSD 11.0 (pfSense 2,4) is missing 'hw.igb.buf_ring_size' tunable so setting it on 2.4 probably has no affect, and igb had some bugs which are fixed in FreeBSD 11.1 (pfSense 2.4.1.). The sysctl 'hw.igb.buf_ring_size' is also reinstated.
isnt it possible to backport the igb drivers from 11.1 for now rather than ship pfsense 2.4 with broken traffic shaping?
-
Did someone test the traffic shaping on 2.4.1?
I didn’t have the time to do it this week, and next week I don’t have the time either. -
@avink:
Did someone test the traffic shaping on 2.4.1?
I didn’t have the time to do it this week, and next week I don’t have the time either.it needs you guys with the problem to test, no point in me testing as I dont get the crash on 2.4 anyway.
Its far easier for you to test than for someone to backport the driver, (note the fixes are also in the kernel rather than the driver).
Do a config backup.
Download 2.4.1 snapshot from pfsense servers.
Install 2.4.1
Restore the backup. -
but the main question is how stable is 2.4.1 coz on my box on 2.4 it doesnt crash and on this i can test but where it crashes is a client box in a remote location so if 2.4.1 isnt stable then it will be cause a lot of headache for me bringing it back to 2.4
-
I tried 2.4.1 with 'em' E1000 nics on ESXi.. for me it crashes..
Version 2.4.1-DEVELOPMENT (amd64) built on Sat Sep 23 19:46:59 CDT 2017 FreeBSD 11.1-RELEASE-p1amd64 11.1-RELEASE-p1 FreeBSD 11.1-RELEASE-p1 #299 r313908+13ee0afae40(RELENG_2_4): Sat Sep 23 19:54:08 CDT 2017 root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense Crash report details: No PHP errors found. Filename: /var/crash/bounds 1 Filename: /var/crash/info.0 Dump header from device: /dev/da0s1b Architecture: amd64 Architecture Version: 1 Dump Length: 86016 Blocksize: 512 Dumptime: Sun Sep 24 19:07:58 2017 Hostname: pfsense_3.local Magic: FreeBSD Text Dump Version String: FreeBSD 11.1-RELEASE-p1 #299 r313908+13ee0afae40(RELENG_2_4): Sat Sep 23 19:54:08 CDT 2017 root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense Panic String: hfsc_dequeue: Dump Parity: 1670814533 Bounds: 0 Dump Status: good Filename: /var/crash/info.last Dump header from device: /dev/da0s1b Architecture: amd64 Architecture Version: 1 Dump Length: 86016 Blocksize: 512 Dumptime: Sun Sep 24 19:07:58 2017 Hostname: pfsense_3.local Magic: FreeBSD Text Dump Version String: FreeBSD 11.1-RELEASE-p1 #299 r313908+13ee0afae40(RELENG_2_4): Sat Sep 23 19:54:08 CDT 2017 root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense Panic String: hfsc_dequeue: Dump Parity: 1670814533 Bounds: 0 Dump Status: good Filename: /var/crash/minfree 2048 Filename: /var/crash/textdump.tar.0 ddb.txt06000014000013161763156 7103 ustarrootwheeldb:0:kdb.enter.default> run lockinfo db:1:lockinfo> show locks No such command db:1:locks> show alllocks No such command db:1:alllocks> show lockedvnods Locked vnodes db:0:kdb.enter.default> show pcpu cpuid = 1 dynamic pcpu = 0xfffffe00981a3200 curthread = 0xfffff80003215560: pid 0 "em4 taskq" curpcb = 0xfffffe002b4f4b80 fpcurthread = none idlethread = 0xfffff800031df560: tid 100004 "idle: cpu1" curpmap = 0xffffffff82a3a7c0 tssp = 0xffffffff82a738f8 commontssp = 0xffffffff82a738f8 rsp0 = 0xfffffe002b4f4b80 gs32p = 0xffffffff82a7a150 ldt = 0xffffffff82a7a190 tss = 0xffffffff82a7a180 db:0:kdb.enter.default> bt Tracing pid 0 tid 100036 td 0xfffff80003215560 kdb_enter() at kdb_enter+0x3b/frame 0xfffffe002b4f3cd0 vpanic() at vpanic+0x1a3/frame 0xfffffe002b4f3d50 panic() at panic+0x43/frame 0xfffffe002b4f3db0 hfsc_dequeue() at hfsc_dequeue+0x1a6/frame 0xfffffe002b4f3df0 tbr_dequeue() at tbr_dequeue+0xea/frame 0xfffffe002b4f3e40 vlan_start() at vlan_start+0x196/frame 0xfffffe002b4f3e90 if_transmit() at if_transmit+0x15c/frame 0xfffffe002b4f3ed0 ether_output() at ether_output+0x718/frame 0xfffffe002b4f3f60 ip_output() at ip_output+0x154a/frame 0xfffffe002b4f40b0 ip_forward() at ip_forward+0x323/frame 0xfffffe002b4f4150 ip_input() at ip_input+0x75a/frame 0xfffffe002b4f41b0 netisr_dispatch_src() at netisr_dispatch_src+0xa0/frame 0xfffffe002b4f4200 ng_iface_rcvdata() at ng_iface_rcvdata+0x11a/frame 0xfffffe002b4f4230 ng_apply_item() at ng_apply_item+0xde/frame 0xfffffe002b4f42b0 ng_snd_item() at ng_snd_item+0x18c/frame 0xfffffe002b4f42f0 ng_apply_item() at ng_apply_item+0xde/frame 0xfffffe002b4f4370 ng_snd_item() at ng_snd_item+0x18c/frame 0xfffffe002b4f43b0 ng_ppp_rcvdata() at ng_ppp_rcvdata+0xe6b/frame 0xfffffe002b4f4430 ng_apply_item() at ng_apply_item+0xde/frame 0xfffffe002b4f44b0 ng_snd_item() at ng_snd_item+0x18c/frame 0xfffffe002b4f44f0 ng_apply_item() at ng_apply_item+0xde/frame 0xfffffe002b4f4570 ng_snd_item() at ng_snd_item+0x18c/frame 0xfffffe002b4f45b0 ng_apply_item() at ng_apply_item+0xde/frame 0xfffffe002b4f4630 ng_snd_item() at ng_snd_item+0x18c/frame 0xfffffe002b4f4670 ether_demux() at ether_demux+0x240/frame 0xfffffe002b4f46a0 ether_nh_input() at ether_nh_input+0x310/frame 0xfffffe002b4f4700 netisr_dispatch_src() at netisr_dispatch_src+0xa0/frame 0xfffffe002b4f4750 ether_input() at ether_input+0x26/frame 0xfffffe002b4f4770 vlan_input() at vlan_input+0x1f0/frame 0xfffffe002b4f47f0 ether_demux() at ether_demux+0x156/frame 0xfffffe002b4f4820 ether_nh_input() at ether_nh_input+0x310/frame 0xfffffe002b4f4880 netisr_dispatch_src() at netisr_dispatch_src+0xa0/frame 0xfffffe002b4f48d0 ether_input() at ether_input+0x26/frame 0xfffffe002b4f48f0 if_input() at if_input+0xa/frame 0xfffffe002b4f4900 lem_rxeof() at lem_rxeof+0x3df/frame 0xfffffe002b4f49a0 lem_handle_rxtx() at lem_handle_rxtx+0x32/frame 0xfffffe002b4f49e0 taskqueue_run_locked() at taskqueue_run_locked+0x127/frame 0xfffffe002b4f4a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xc8/frame 0xfffffe002b4f4a70 fork_exit() at fork_exit+0x85/frame 0xfffffe002b4f4ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe002b4f4ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- -
Afraid this is really an uphill battle and altq is pretty much dead… Not sure what's the plan in FreeBSD but this buggy mess is not maintainable by Netgate alone.
-
Yes its reliant on FreeBSD upstream, pfSense may refine the code, but you entering a dangerous situation if you start making large en masse changes from the upstream product as this would need maintaining outside of upstream and be a pain on every upstream update.
FreeBSD made a decision to not keep up with openbsd in regards to PF and ALTQ so they made the rod on their own back with that decision, the good news is that they have started to fix FreeBSD specific PF issues now and I would hope this includes ALTQ as well.
The problem in this case there is no specific confirmed bug, just a few guys reporting crashes at high load, it needs lots of testing to dig deeper.
My advice was on the basis I am aware of scheduler panic fixes and igb panic fixes in FreeBSD 11.1 upstream hence what I posted.
-
FreeBSD made a decision to not keep up with openbsd in regards to PF and ALTQ so they made the rod on their own back with that decision, the good news is that they have started to fix FreeBSD specific PF issues now and I would hope this includes ALTQ as well.
ALTQ has been gone from OpenBSD for years. http://undeadly.org/cgi?action=article&sid=20140419151959
-
my crash log attached as follows, im also having same issue on client box when i enable shaping, sorry the log was too long to capture completely
-
A quick update from my side.
Both ALTQ and HFSC crash on 2.4.1 as well.
It doesn’t have to have any load. It crashes only seconds after reboot.Will try to upload the crash info tonight.
