Crash when enabling traffic shaper on more than 1 port
-
I started getting one or two crash reboot a day on 2.4RC. I have submitted every one of them. Here is the latest:
Crash report begins. Anonymous machine information: amd64 11.0-RELEASE-p12 FreeBSD 11.0-RELEASE-p12 #58 daf8d77a4c1(RELENG_2_4_0): Fri Sep 22 20:48:22 CDT 2017 root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense Crash report details: No PHP errors found. Filename: /var/crash/bounds 1 Filename: /var/crash/info.0 Dump header from device: /dev/gptid/09925e84-90d4-11e7-84bc-80ee73c2ffc4 Architecture: amd64 Architecture Version: 1 Dump Length: 79360 Blocksize: 512 Dumptime: Sat Sep 23 12:45:13 2017 Hostname: alix1.nowayout.ca Magic: FreeBSD Text Dump Version String: FreeBSD 11.0-RELEASE-p12 #58 daf8d77a4c1(RELENG_2_4_0): Fri Sep 22 20:48:22 CDT 2017 root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense Panic String: hfsc_dequeue: Dump Parity: 3690452271 Bounds: 0 Dump Status: good Filename: /var/crash/info.last Dump header from device: /dev/gptid/09925e84-90d4-11e7-84bc-80ee73c2ffc4 Architecture: amd64 Architecture Version: 1 Dump Length: 79360 Blocksize: 512 Dumptime: Sat Sep 23 12:45:13 2017 Hostname: alix1.nowayout.ca Magic: FreeBSD Text Dump Version String: FreeBSD 11.0-RELEASE-p12 #58 daf8d77a4c1(RELENG_2_4_0): Fri Sep 22 20:48:22 CDT 2017 root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense Panic String: hfsc_dequeue: Dump Parity: 3690452271 Bounds: 0 Dump Status: good Filename: /var/crash/minfree 2048 Filename: /var/crash/textdump.tar.0 ddb.txt06000014000013161507631 7075 ustarrootwheeldb:0:kdb.enter.default> run lockinfo db:1:lockinfo> show locks No such command db:1:locks> show alllocks No such command db:1:alllocks> show lockedvnods Locked vnodes db:0:kdb.enter.default> show pcpu cpuid = 1 dynamic pcpu = 0xfffffe0170be4200 curthread = 0xfffff80003983000: pid 0 "em0 que" curpcb = 0xfffffe0117d60c80 fpcurthread = none idlethread = 0xfffff8000395fa00: tid 100004 "idle: cpu1" curpmap = 0xffffffff829e6300 tssp = 0xffffffff82a1eb78 commontssp = 0xffffffff82a1eb78 rsp0 = 0xfffffe0117d60c80 gs32p = 0xffffffff82a253d0 ldt = 0xffffffff82a25410 tss = 0xffffffff82a25400 db:0:kdb.enter.default> bt Tracing pid 0 tid 100027 td 0xfffff80003983000 kdb_enter() at kdb_enter+0x3b/frame 0xfffffe0117d60370 vpanic() at vpanic+0x19f/frame 0xfffffe0117d603f0 panic() at panic+0x43/frame 0xfffffe0117d60450 hfsc_dequeue() at hfsc_dequeue+0x633/frame 0xfffffe0117d60490 tbr_dequeue() at tbr_dequeue+0xdf/frame 0xfffffe0117d604e0 vlan_start() at vlan_start+0x12c/frame 0xfffffe0117d60540 if_transmit() at if_transmit+0x1a5/frame 0xfffffe0117d60590 ether_output() at ether_output+0x728/frame 0xfffffe0117d60620 ip_tryforward() at ip_tryforward+0x726/frame 0xfffffe0117d606a0 ip_input() at ip_input+0x3a4/frame 0xfffffe0117d60700 netisr_dispatch_src() at netisr_dispatch_src+0xa5/frame 0xfffffe0117d60760 ether_demux() at ether_demux+0x15c/frame 0xfffffe0117d60790 ether_nh_input() at ether_nh_input+0x317/frame 0xfffffe0117d607f0 netisr_dispatch_src() at netisr_dispatch_src+0xa5/frame 0xfffffe0117d60850 ether_input() at ether_input+0x26/frame 0xfffffe0117d60870 vlan_input() at vlan_input+0x1cb/frame 0xfffffe0117d608f0 ether_demux() at ether_demux+0xa9/frame 0xfffffe0117d60920 ether_nh_input() at ether_nh_input+0x317/frame 0xfffffe0117d60980 netisr_dispatch_src() at netisr_dispatch_src+0xa5/frame 0xfffffe0117d609e0 ether_input() at ether_input+0x26/frame 0xfffffe0117d60a00 if_input() at if_input+0xa/frame 0xfffffe0117d60a10 em_rxeof() at em_rxeof+0x352/frame 0xfffffe0117d60a90 em_handle_que() at em_handle_que+0x43/frame 0xfffffe0117d60ae0 taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame 0xfffffe0117d60b40 taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame 0xfffffe0117d60b70 fork_exit() at fork_exit+0x85/frame 0xfffffe0117d60bb0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0117d60bb0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db:0:kdb.enter.default> ps pid ppid pgrp uid state wmesg wchan cmd 82872 90006 552 0 S nanslp 0xffffffff82866b30 sleep 80573 552 552 0 S accept 0xfffff8000e29172c php-fpm 67765 1 67616 558 S select 0xfffff8000eae9f40 avahi-daemon 67480 1 552 0 S nanslp 0xffffffff82866b31 php 63593 1 63593 556 Ss select 0xfffff800616e2d40 dbus-daemon 90006 1 552 0 S wait 0xfffff8000e137000 sh 89044 1 89044 136 Ss select 0xfffff80061027dc0 dhcpd 76084 1 76084 59 Ss (threaded) unbound 100251 S kqread 0xfffff8000e6f3500 unbound 100272 S kqread 0xfffff8000e964700 unbound 53080 1 53080 0 Ss (threaded) dpinger 100157 S uwait 0xfffff8000e2b2500 dpinger 100268 S sbwait 0xfffff80085533144 dpinger 100269 S nanslp 0xffffffff82866b30 dpinger 100270 S nanslp 0xffffffff82866b31 dpinger 100271 S accept 0xfffff8008550f72c dpinger 52697 1 52697 0 Ss (threaded) dpinger 100218 S uwait 0xfffff8000ea68080 dpinger 100264 S sbwait 0xfffff8007b0e7144 dpinger 100265 S nanslp 0xffffffff82866b31 dpinger 100266 S nanslp 0xffffffff82866b31 dpinger 100267 S accept 0xfffff80061ad93cc dpinger 51997 1 51997 0 Ss (threaded) dpinger 100142 S uwait 0xfffff8000e027780 dpinger 100260 S sbwait 0xfffff800855084a4 dpinger 100261 S nanslp 0xffffffff82866b30 dpinger 100262 S nanslp 0xffffffff82866b31 dpinger 100263 S accept 0xfffff8001460aa8c dpinger 6177 1 6177 0 Rs CPU 0 openvpn 57344 1 57344 0 Ss+ ttyin 0xfffff8000badc8a8 getty 57229 1 57229 0 Ss+ ttyin 0xfffff8000badcca8 getty 56954 1 56954 0 Ss+ ttyin 0xfffff8000badd8a8 getty 56790 1 56790 0 Ss+ ttyin 0xfffff8000badd0a8 getty 56606 1 56606 0 Ss+ ttyin 0xfffff8000badd4a8 getty 56492 1 56492 0 Ss+ ttyin 0xfffff8000badf4a8 getty 56169 1 56169 0 Ss+ ttyin 0xfffff8000badf8a8 getty 56044 1 56044 0 Ss+ ttyin 0xfffff8000badfca8 getty 9047 8128 8128 0 S nanslp 0xffffffff82866b31 minicron 8128 1 8128 0 Ss wait 0xfffff800856a8528 minicron 7894 7188 7188 0 S nanslp 0xffffffff82866b30 minicron 7188 1 7188 0 Ss wait 0xfffff80021a3b000 minicron 6870 6432 6432 0 S nanslp 0xffffffff82866b31 minicron 6432 1 6432 0 Ss wait 0xfffff80085203528 minicron 6187 1 6187 0 Ss (threaded) filterdns 100227 S uwait 0xfffff80061029900 ns1.he.net 100228 S uwait 0xfffff8006107b000 ns2.he.net 100229 S uwait 0xfffff8006107a400 ns3.he.net 100230 S uwait 0xfffff8000ee31d00 ns4.he.net 100231 S uwait 0xfffff80061734d00 ns5.he.net 100232 S uwait 0xfffff80061736600 filterdns 100233 S uwait 0xfffff800616ffd80 filterdns 100234 S uwait 0xfffff8006171e680 filterdns 100235 S uwait 0xfffff8006171ff00 filterdns 100236 S uwait 0xfffff800616b1d80 filterdns 100237 S uwait 0xfffff8000efb3300 signal-thread 2653 1 2358 0 S kqread 0xfffff8000e64d300 lighttpd_pfb 82428 1 82428 556 Ss select 0xfffff80061734f40 dbus-daemon 70117 1 70117 0 Ss select 0xfffff8000e663b40 syslogd 42738 1 42738 0 Ss select 0xfffff8000ee5b6c0 miniupnpd 37886 1 37886 0 Ss select 0xfffff8000e91b0c0 powerd 70983 1 70983 0 Ss (threaded) ntpd 100203 S select 0xfffff8000ee04540 ntpd 100205 S usem 0xfffff8000e663680 ntpd 70411 1 70411 0 Ss nanslp 0xffffffff82866b30 cron 69785 69275 69275 0 S kqread 0xfffff8000ec6f200 nginx 69509 69275 69275 0 S kqread 0xfffff8000ec8ae00 nginx 69275 1 69275 0 Ss pause 0xfffff8000e1260a8 nginx 48784 1 48784 0 Ss bpf 0xfffff80061192800 filterlog 44364 1 44364 0 Ss select 0xfffff800612d47c0 openvpn 29529 1 29529 65 Ss select 0xfffff8000e662c40 dhclient 24129 1 24129 0 Ss select 0xfffff8000e5e88c0 dhclient 16032 1 16032 0 Ss (threaded) sshlockout_pf 100143 S uwait 0xfffff8000e027880 sshlockout_pf 100156 S nanslp 0xffffffff82866b30 sshlockout_pf 15937 1 15937 0 Ss select 0xfffff80061663340 sshd 580 1 580 0 Ss select 0xfffff800616032c0 devd 568 566 566 0 S kqread 0xfffff800611b5200 check_reload_status 566 1 566 0 Ss kqread 0xfffff80061149e00 check_reload_status 552 1 552 0 Ss kqread 0xfffff800611b4700 php-fpm 59 0 0 0 DL mdwait 0xfffff80003bba800 [md0] 27 0 0 0 DL syncer 0xffffffff82960680 [syncer] 26 0 0 0 DL vlruwt 0xfffff8000bb1e528 [vnlru] 25 0 0 0 DL (threaded) [bufdaemon] 100120 D psleep 0xffffffff8295ef04 [bufdaemon] 100132 D sdflush 0xfffff8000bbb28e8 [/ worker] 24 0 0 0 DL - 0xffffffff8295fbb4 [bufspacedaemon] 23 0 0 0 DL pgzero 0xffffffff829742e4 [pagezero] 22 0 0 0 DL psleep 0xffffffff8297078c [vmdaemon] 21 0 0 0 DL (threaded) [pagedaemon] 100116 D psleep 0xffffffff82a1df05 [pagedaemon] 100126 D umarcl 0xffffffff829700b8 [uma] 20 0 0 0 DL idle 0xfffff8000bb20528 [enc_daemon0] 19 0 0 0 DL - 0xffffffff8295e714 [soaiod4] 18 0 0 0 DL - 0xffffffff8295e714 [soaiod3] 17 0 0 0 DL - 0xffffffff8295e714 [soaiod2] 9 0 0 0 DL - 0xffffffff8295e714 [soaiod1] 16 0 0 0 DL tzpoll 0xffffffff82621f38 [acpi_thermal] 8 0 0 0 DL - 0xffffffff8273b2f0 [rand_harvestq] 7 0 0 0 DL pftm 0xffffffff80f4c510 [pf purge] 6 0 0 0 DL waiting_ 0xffffffff82a0d540 [sctp_iterator] 5 0 0 0 SL - 0xfffffe0000fde000 [fw0_probe] 15 0 0 0 DL (threaded) [usb] 100030 D - 0xfffffe0000f22de8 [usbus0] 100031 D - 0xfffffe0000f22e40 [usbus0] 100032 D - 0xfffffe0000f22e98 [usbus0] 100033 D - 0xfffffe0000f22ef0 [usbus0] 100034 D - 0xfffffe0000f22f48 [usbus0] 100036 D - 0xfffffe0000f3ede8 [usbus1] 100037 D - 0xfffffe0000f3ee40 [usbus1] 100038 D - 0xfffffe0000f3ee98 [usbus1] 100039 D - 0xfffffe0000f3eef0 [usbus1] 100040 D - 0xfffffe0000f3ef48 [usbus1] 100042 D - 0xfffffe0000f4ede8 [usbus2] 100043 D - 0xfffffe0000f4ee40 [usbus2] 100044 D - 0xfffffe0000f4ee98 [usbus2] 100045 D - 0xfffffe0000f4eef0 [usbus2] 100046 D - 0xfffffe0000f4ef48 [usbus2] 100047 D - 0xfffffe0000f5ed10 [usbus3] 100048 D - 0xfffffe0000f5ed68 [usbus3] 100049 D - 0xfffffe0000f5edc0 [usbus3] 100050 D - 0xfffffe0000f5ee18 [usbus3] 100051 D - 0xfffffe0000f5ee70 [usbus3] 100059 D - 0xfffffe0000f6ede8 [usbus4] 100060 D - 0xfffffe0000f6ee40 [usbus4] 100061 D - 0xfffffe0000f6ee98 [usbus4] 100062 D - 0xfffffe0000f6eef0 [usbus4] 100063 D - 0xfffffe0000f6ef48 [usbus4] 100064 D - 0xfffffe0000fb4de8 [usbus5] 100065 D - 0xfffffe0000fb4e40 [usbus5] 100066 D - 0xfffffe0000fb4e98 [usbus5] 100067 D - 0xfffffe0000fb4ef0 [usbus5] 100068 D - 0xfffffe0000fb4f48 [usbus5] 100069 D - 0xfffffe0000fc4de8 [usbus6] 100070 D - 0xfffffe0000fc4e40 [usbus6] 100071 D - 0xfffffe0000fc4e98 [usbus6] 100072 D - 0xfffffe0000fc4ef0 [usbus6] 100073 D - 0xfffffe0000fc4f48 [usbus6] 100074 D - 0xfffffe0000fd4d10 [usbus7] 100075 D - 0xfffffe0000fd4d68 [usbus7] 100076 D - 0xfffffe0000fd4dc0 [usbus7] 100077 D - 0xfffffe0000fd4e18 [usbus7] 100078 D - 0xfffffe0000fd4e70 [usbus7] 4 0 0 0 DL (threaded) [cam] 100026 D - 0xffffffff8260e180 [doneq0] 100110 D - 0xffffffff8260dfc8 [scanner] 3 0 0 0 DL crypto_r 0xffffffff8296ec70 [crypto returns] 2 0 0 0 DL crypto_w 0xffffffff8296eb18 [crypto] 14 0 0 0 DL (threaded) [geom] 100017 D - 0xffffffff829e5740 [g_event] 100018 D - 0xffffffff829e5748 [g_up] 100019 D - 0xffffffff829e5750 [g_down] 13 0 0 0 DL (threaded) [ng_queue] 100015 D sleep 0xffffffff825cbfd0 [ng_queue0] 100016 D sleep 0xffffffff825cbfd0 [ng_queue1] 12 0 0 0 WL (threaded) [intr] 100005 I [swi3: vm] 100006 I [swi4: clock (0)] 100007 I [swi4: clock (1)] 100008 I [swi1: netisr 0] 100009 I [swi6: task queue] 100010 I [swi6: Giant taskq] 100013 I [swi5: fast taskq] 100029 I [irq20: hpet0 uhci0*] 100035 I [irq21: uhci1 uhci4] 100041 I [irq22: uhci2 ehci0+] 100052 I [irq257: hdac0] 100053 I [irq258: pcib1] 100056 I [irq259: iwn0] 100057 I [irq260: pcib2] 100058 I [irq261: pcib3] 100079 I [irq17: fwohci0] 100082 I [irq18: sdhci_pci0+] 100083 I [irq262: ahci0:ch0] 100084 I [irq263: ahci0:ch1] 100085 I [irq264: ahci0:ch2] 100086 I [irq265: ahci0:ch3] 100087 I [irq266: ahci0:ch4] 100088 I [irq267: ahci0:ch5] 100089 I [irq268: ahci0:6] 100090 I [irq269: ahci0:7] 100091 I [irq270: ahci0:8] 100092 I [irq271: ahci0:9] 100093 I [irq272: ahci0:10] 100094 I [irq273: ahci0:11] 100095 I [irq274: ahci0:12] 100096 I [irq275: ahci0:13] 100097 I [irq276: ahci0:14] 100098 I [irq277: ahci0:15] 100099 I [irq1: atkbd0] 100100 I [irq12: psm0] 100101 I [swi0: uart] 100105 I [swi1: pf send] 100106 I [swi1: pfsync] 100123 I [swi1: netisr 1] 11 0 0 0 RL (threaded) [idle] 100003 CanRun [idle: cpu0] 100004 CanRun [idle: cpu1] 1 0 1 0 SLs wait 0xfffff8000395d528 [init] 10 0 0 0 DL audit_wo 0xffffffff82a146e0 [audit] 0 0 0 0 RLs (threaded) [kernel] 100000 D swapin 0xffffffff829e5778 [swapper] 100011 D - 0xfffff8000394d000 [aiod_kick taskq] 100012 D - 0xfffff8000394cd00 [thread] 100014 D - 0xfffff8000394c800 [kqueue_ctx taskq] 100020 D - 0xfffff80003989d00 [firmware taskq] 100023 D - 0xfffff80003988600 [acpi_task_0] 100024 D - 0xfffff80003988600 [acpi_task_1] 100025 D - 0xfffff80003988600 [acpi_task_2] 100027 Run CPU 1 [em0 que] 100028 D - 0xfffff80003ba7700 [em0 txq] 100054 D - 0xfffff8000516a500 [iwn0 net80211 taskq] 100055 D - 0xfffff8000516a300 [iwn_taskq] 100080 D - 0xfffff80003bf9600 [fw0_taskq] 100102 D - 0xfffff8000b098a00 [mca taskq] 100109 D - 0xfffff80003988200 [CAM taskq] 100124 D - 0xfffff8000baf5d00 [if_config_tqg_0] 100125 D - 0xfffff8000bb79b00 [if_io_tqg_0] 100127 D - 0xfffff8000bb79900 [if_io_tqg_1] db:0:kdb.enter.default> alltrace Tracing command sleep pid 82872 tid 100226 td 0xfffff8008572a500 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f068880 mi_switch() at mi_switch+0xd2/frame 0xfffffe011f0688b0 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f068930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011f068960 _sleep() at _sleep+0x26f/frame 0xfffffe011f0689f0 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011f068a50 sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011f068a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f068bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f068bb0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8009008fa, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 --- Tracing command php-fpm pid 80573 tid 100225 td 0xfffff8008572aa00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f063810 mi_switch() at mi_switch+0xd2/frame 0xfffffe011f063840 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0638c0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011f0638f0 _sleep() at _sleep+0x294/frame 0xfffffe011f063980 kern_accept4() at kern_accept4+0x1f2/frame 0xfffffe011f063a30 accept1() at accept1+0x60/frame 0xfffffe011f063a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f063bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f063bb0 --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x801d939ba, rsp = 0x7ffffffec848, rbp = 0x7ffffffec880 --- Tracing command avahi-daemon pid 67765 tid 100255 td 0xfffff80021e6fa00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f0f4730 mi_switch() at mi_switch+0xd2/frame 0xfffffe011f0f4760 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0f47e0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011f0f4810 _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe011f0f4880 seltdwait() at seltdwait+0xc7/frame 0xfffffe011f0f48d0 kern_poll() at kern_poll+0x296/frame 0xfffffe011f0f4a70 sys_poll() at sys_poll+0x61/frame 0xfffffe011f0f4a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f0f4bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f0f4bb0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x801814f7a, rsp = 0x7fffffffe7d8, rbp = 0x7fffffffe810 --- Tracing command php pid 67480 tid 100254 td 0xfffff80021e86000 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f0ef880 mi_switch() at mi_switch+0xd2/frame 0xfffffe011f0ef8b0 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0ef930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011f0ef960 _sleep() at _sleep+0x26f/frame 0xfffffe011f0ef9f0 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011f0efa50 sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011f0efa90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f0efbb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f0efbb0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x801d848fa, rsp = 0x7fffffffc7c8, rbp = 0x7fffffffc7f0 --- Tracing command dbus-daemon pid 63593 tid 100250 td 0xfffff80021e87500 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f0db750 mi_switch() at mi_switch+0xd2/frame 0xfffffe011f0db780 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0db800 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011f0db830 _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011f0db880 seltdwait() at seltdwait+0x10f/frame 0xfffffe011f0db8d0 kern_poll() at kern_poll+0x296/frame 0xfffffe011f0dba70 sys_poll() at sys_poll+0x61/frame 0xfffffe011f0dba90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f0dbbb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f0dbbb0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8011dcf7a, rsp = 0x7fffffffe858, rbp = 0x7fffffffe890 --- Tracing command sh pid 90006 tid 100146 td 0xfffff8000e0d7000 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef4d660 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef4d690 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef4d710 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef4d740 _sleep() at _sleep+0x294/frame 0xfffffe011ef4d7d0 kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe011ef4d8a0 sys_wait4() at sys_wait4+0x73/frame 0xfffffe011ef4da90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef4dbb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef4dbb0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800b5d75a, rsp = 0x7fffffffe868, rbp = 0x7fffffffe920 --- Tracing command dhcpd pid 89044 tid 100148 td 0xfffff8000e13b500 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef576e0 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef57710 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef57790 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef577c0 _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011ef57810 seltdwait() at seltdwait+0x10f/frame 0xfffffe011ef57860 kern_select() at kern_select+0x979/frame 0xfffffe011ef57a50 sys_select() at sys_select+0x54/frame 0xfffffe011ef57a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef57bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef57bb0 --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800d641ca, rsp = 0x7fffffffea08, rbp = 0x7fffffffea30 --- Tracing command unbound pid 76084 tid 100251 td 0xfffff80021e87000 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f0e0620 mi_switch() at mi_switch+0xd2/frame 0xfffffe011f0e0650 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0e06d0 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011f0e0700 _sleep() at _sleep+0x26f/frame 0xfffffe011f0e0790 kqueue_kevent() at kqueue_kevent+0x336/frame 0xfffffe011f0e0970 kern_kevent() at kern_kevent+0xb5/frame 0xfffffe011f0e09e0 sys_kevent() at sys_kevent+0x11c/frame 0xfffffe011f0e0a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f0e0bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f0e0bb0 --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x801b7293a, rsp = 0x7fffffffea18, rbp = 0x7fffffffea60 --- Tracing command unbound pid 76084 tid 100272 td 0xfffff800854e0a00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f103620 mi_switch() at mi_switch+0xd2/frame 0xfffffe011f103650 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f1036d0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011f103700 _sleep() at _sleep+0x294/frame 0xfffffe011f103790 kqueue_kevent() at kqueue_kevent+0x336/frame 0xfffffe011f103970 kern_kevent() at kern_kevent+0xb5/frame 0xfffffe011f1039e0 sys_kevent() at sys_kevent+0x11c/frame 0xfffffe011f103a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f103bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f103bb0 --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x801b7293a, rsp = 0x7fffdfffde18, rbp = 0x7fffdfffde60 --- Tracing command dpinger pid 53080 tid 100157 td 0xfffff8007b0e5500 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef847f0 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef84820 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef848a0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef848d0 _sleep() at _sleep+0x294/frame 0xfffffe011ef84960 umtxq_sleep() at umtxq_sleep+0x122/frame 0xfffffe011ef849c0 do_wait() at do_wait+0x419/frame 0xfffffe011ef84a50 __umtx_op_wait() at __umtx_op_wait+0x77/frame 0xfffffe011ef84a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef84bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef84bb0 --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083c67c, rsp = 0x7fffffffe8f8, rbp = 0x7fffffffe980 --- Tracing command dpinger pid 53080 tid 100268 td 0xfffff800854df500 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efb16f0 mi_switch() at mi_switch+0xd2/frame 0xfffffe011efb1720 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efb17a0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011efb17d0 _sleep() at _sleep+0x294/frame 0xfffffe011efb1860 sbwait() at sbwait+0x4f/frame 0xfffffe011efb1880 soreceive_generic() at soreceive_generic+0x1ae/frame 0xfffffe011efb1950 kern_recvit() at kern_recvit+0x1d6/frame 0xfffffe011efb1a20 sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe011efb1a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efb1bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efb1bb0 --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b3cf5a, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 --- Tracing command dpinger pid 53080 tid 100269 td 0xfffff800854df000 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efb6880 mi_switch() at mi_switch+0xd2/frame 0xfffffe011efb68b0 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efb6930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011efb6960 _sleep() at _sleep+0x26f/frame 0xfffffe011efb69f0 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011efb6a50 sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011efb6a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efb6bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efb6bb0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2b8fa, rsp = 0x7fffdfdfcf58, rbp = 0x7fffdfdfcf80 --- Tracing command dpinger pid 53080 tid 100270 td 0xfffff800c31cfa00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efbb880 mi_switch() at mi_switch+0xd2/frame 0xfffffe011efbb8b0 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efbb930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011efbb960 _sleep() at _sleep+0x26f/frame 0xfffffe011efbb9f0 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011efbba50 sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011efbba90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efbbbb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efbbbb0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2b8fa, rsp = 0x7fffdfbfbef8, rbp = 0x7fffdfbfbf20 --- Tracing command dpinger pid 53080 tid 100271 td 0xfffff800c31cf500 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efc0810 mi_switch() at mi_switch+0xd2/frame 0xfffffe011efc0840 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efc08c0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011efc08f0 _sleep() at _sleep+0x294/frame 0xfffffe011efc0980 kern_accept4() at kern_accept4+0x1f2/frame 0xfffffe011efc0a30 accept1() at accept1+0xf6/frame 0xfffffe011efc0a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efc0bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efc0bb0 --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x800b2b9ba, rsp = 0x7fffdf9fae78, rbp = 0x7fffdf9faeb0 --- Tracing command dpinger pid 52697 tid 100218 td 0xfffff8007b294000 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f0407f0 mi_switch() at mi_switch+0xd2/frame 0xfffffe011f040820 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0408a0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011f0408d0 _sleep() at _sleep+0x294/frame 0xfffffe011f040960 umtxq_sleep() at umtxq_sleep+0x122/frame 0xfffffe011f0409c0 do_wait() at do_wait+0x419/frame 0xfffffe011f040a50 __umtx_op_wait() at __umtx_op_wait+0x77/frame 0xfffffe011f040a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f040bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f040bb0 --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083c67c, rsp = 0x7fffffffe908, rbp = 0x7fffffffe990 --- Tracing command dpinger pid 52697 tid 100264 td 0xfffff8007b24aa00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef936f0 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef93720 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef937a0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef937d0 _sleep() at _sleep+0x294/frame 0xfffffe011ef93860 sbwait() at sbwait+0x4f/frame 0xfffffe011ef93880 soreceive_generic() at soreceive_generic+0x1ae/frame 0xfffffe011ef93950 kern_recvit() at kern_recvit+0x1d6/frame 0xfffffe011ef93a20 sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe011ef93a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef93bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef93bb0 --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b3cf5a, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 --- Tracing command dpinger pid 52697 tid 100265 td 0xfffff8000bb7da00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef98880 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef988b0 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef98930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011ef98960 _sleep() at _sleep+0x26f/frame 0xfffffe011ef989f0 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011ef98a50 sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011ef98a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef98bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef98bb0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2b8fa, rsp = 0x7fffdfdfcf58, rbp = 0x7fffdfdfcf80 --- Tracing command dpinger pid 52697 tid 100266 td 0xfffff8007b0e4000 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef9d880 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef9d8b0 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef9d930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011ef9d960 _sleep() at _sleep+0x26f/frame 0xfffffe011ef9d9f0 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011ef9da50 sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011ef9da90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef9dbb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef9dbb0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2b8fa, rsp = 0x7fffdfbfbef8, rbp = 0x7fffdfbfbf20 --- Tracing command dpinger pid 52697 tid 100267 td 0xfffff8007b13ba00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efa2810 mi_switch() at mi_switch+0xd2/frame 0xfffffe011efa2840 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efa28c0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011efa28f0 _sleep() at _sleep+0x294/frame 0xfffffe011efa2980 kern_accept4() at kern_accept4+0x1f2/frame 0xfffffe011efa2a30 accept1() at accept1+0xf6/frame 0xfffffe011efa2a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efa2bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efa2bb0 --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x800b2b9ba, rsp = 0x7fffdf9fae78, rbp = 0x7fffdf9faeb0 --- Tracing command dpinger pid 51997 tid 100142 td 0xfffff8000e121500 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef397f0 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef39820 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef398a0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef398d0 _sleep() at _sleep+0x294/frame 0xfffffe011ef39960 umtxq_sleep() at umtxq_sleep+0x122/frame 0xfffffe011ef399c0 do_wait() at do_wait+0x419/frame 0xfffffe011ef39a50 __umtx_op_wait() at __umtx_op_wait+0x77/frame 0xfffffe011ef39a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef39bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef39bb0 --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083c67c, rsp = 0x7fffffffe908, rbp = 0x7fffffffe990 --- Tracing command dpinger pid 51997 tid 100260 td 0xfffff8007b196500 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efd96f0 mi_switch() at mi_switch+0xd2/frame 0xfffffe011efd9720 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efd97a0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011efd97d0 _sleep() at _sleep+0x294/frame 0xfffffe011efd9860 sbwait() at sbwait+0x4f/frame 0xfffffe011efd9880 soreceive_generic() at soreceive_generic+0x1ae/frame 0xfffffe011efd9950 kern_recvit() at kern_recvit+0x1d6/frame 0xfffffe011efd9a20 sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe011efd9a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efd9bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efd9bb0 --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b3cf5a, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 --- Tracing command dpinger pid 51997 tid 100261 td 0xfffff8007b19fa00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efde880 mi_switch() at mi_switch+0xd2/frame 0xfffffe011efde8b0 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efde930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011efde960 _sleep() at _sleep+0x26f/frame 0xfffffe011efde9f0 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011efdea50 sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011efdea90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efdebb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efdebb0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2b8fa, rsp = 0x7fffdfdfcf58, rbp = 0x7fffdfdfcf80 --- Tracing command dpinger pid 51997 tid 100262 td 0xfffff800854e0000 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efe3880 mi_switch() at mi_switch+0xd2/frame 0xfffffe011efe38b0 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efe3930 sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011efe3960 _sleep() at _sleep+0x26f/frame 0xfffffe011efe39f0 kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011efe3a50 sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011efe3a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efe3bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efe3bb0 --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2b8fa, rsp = 0x7fffdfbfbef8, rbp = 0x7fffdfbfbf20 --- Tracing command dpinger pid 51997 tid 100263 td 0xfffff800854dfa00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efe8810 mi_switch() at mi_switch+0xd2/frame 0xfffffe011efe8840 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efe88c0 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011efe88f0 _sleep() at _sleep+0x294/frame 0xfffffe011efe8980 kern_accept4() at kern_accept4+0x1f2/frame 0xfffffe011efe8a30 accept1() at accept1+0xf6/frame 0xfffffe011efe8a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efe8bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efe8bb0 --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x800b2b9ba, rsp = 0x7fffdf9fae78, rbp = 0x7fffdf9faeb0 --- Tracing command openvpn pid 6177 tid 100241 td 0xfffff80021dc1500 cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff82978530 ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xffffffff82978550 trap() at trap+0x3a/frame 0xffffffff82978760 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff82978760 --- trap 0x13, rip = 0xffffffff80cf8666, rsp = 0xfffffe011f0ae880, rbp = 0xfffffe011f0ae8d0 --- sopoll_generic() at sopoll_generic+0x46/frame 0xfffffe011f0ae8d0 kern_poll() at kern_poll+0x650/frame 0xfffffe011f0aea70 sys_poll() at sys_poll+0x61/frame 0xfffffe011f0aea90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f0aebb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f0aebb0 --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x80155cf7a, rsp = 0x7fffffffdd78, rbp = 0x7fffffffdde0 --- Tracing command getty pid 57344 tid 100150 td 0xfffff8000e13aa00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef61760 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef61790 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef61810 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef61840 _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011ef61890 tty_wait() at tty_wait+0x1c/frame 0xfffffe011ef618b0 ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe011ef61910 ttydev_read() at ttydev_read+0x34/frame 0xfffffe011ef61940 devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe011ef619a0 dofileread() at dofileread+0x98/frame 0xfffffe011ef619f0 kern_readv() at kern_readv+0x68/frame 0xfffffe011ef61a40 sys_read() at sys_read+0x84/frame 0xfffffe011ef61a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef61bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef61bb0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 --- Tracing command getty pid 57229 tid 100151 td 0xfffff8000e13a500 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef66760 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef66790 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef66810 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef66840 _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011ef66890 tty_wait() at tty_wait+0x1c/frame 0xfffffe011ef668b0 ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe011ef66910 ttydev_read() at ttydev_read+0x34/frame 0xfffffe011ef66940 devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe011ef669a0 dofileread() at dofileread+0x98/frame 0xfffffe011ef669f0 kern_readv() at kern_readv+0x68/frame 0xfffffe011ef66a40 sys_read() at sys_read+0x84/frame 0xfffffe011ef66a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef66bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef66bb0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 --- Tracing command getty pid 56954 tid 100240 td 0xfffff80021e87a00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f0a9760 mi_switch() at mi_switch+0xd2/frame 0xfffffe011f0a9790 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0a9810 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011f0a9840 _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011f0a9890 tty_wait() at tty_wait+0x1c/frame 0xfffffe011f0a98b0 ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe011f0a9910 ttydev_read() at ttydev_read+0x34/frame 0xfffffe011f0a9940 devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe011f0a99a0 dofileread() at dofileread+0x98/frame 0xfffffe011f0a99f0 kern_readv() at kern_readv+0x68/frame 0xfffffe011f0a9a40 sys_read() at sys_read+0x84/frame 0xfffffe011f0a9a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f0a9bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f0a9bb0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 --- Tracing command getty pid 56790 tid 100135 td 0xfffff8000e040a00 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef16760 mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef16790 sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef16810 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef16840 _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011ef16890 tty_wait() at tty_wait+0x1c/frame 0xfffffe011ef168b0 ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe011ef16910 ttydev_read() at ttydev_read+0x34/frame 0xfffffe011ef16940 devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe011ef169a0 dofileread() at dofileread+0x98/frame 0xfffffe011ef169f0 kern_readv() at kern_readv+0x68/frame 0xfffffe011ef16a40 sys_read() at sys_read+0x84/frame 0xfffffe011ef16a90 amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef16bb0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef16bb0 --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 --- Tracing command getty pid 56606 tid 100224 td 0xfffff8008572b000 sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f05e760 mi_switch() at mi_switch+0xd2/f[/thread] -
I would test on 2.4.1.
The igb driver on FreeBSD 11.0 (pfSense 2,4) is missing 'hw.igb.buf_ring_size' tunable so setting it on 2.4 probably has no affect, and igb had some bugs which are fixed in FreeBSD 11.1 (pfSense 2.4.1.). The sysctl 'hw.igb.buf_ring_size' is also reinstated.
isnt it possible to backport the igb drivers from 11.1 for now rather than ship pfsense 2.4 with broken traffic shaping?
-
Did someone test the traffic shaping on 2.4.1?
I didn’t have the time to do it this week, and next week I don’t have the time either. -
@avink:
Did someone test the traffic shaping on 2.4.1?
I didn’t have the time to do it this week, and next week I don’t have the time either.it needs you guys with the problem to test, no point in me testing as I dont get the crash on 2.4 anyway.
Its far easier for you to test than for someone to backport the driver, (note the fixes are also in the kernel rather than the driver).
Do a config backup.
Download 2.4.1 snapshot from pfsense servers.
Install 2.4.1
Restore the backup. -
but the main question is how stable is 2.4.1 coz on my box on 2.4 it doesnt crash and on this i can test but where it crashes is a client box in a remote location so if 2.4.1 isnt stable then it will be cause a lot of headache for me bringing it back to 2.4
-
I tried 2.4.1 with 'em' E1000 nics on ESXi.. for me it crashes..
Version 2.4.1-DEVELOPMENT (amd64) built on Sat Sep 23 19:46:59 CDT 2017 FreeBSD 11.1-RELEASE-p1amd64 11.1-RELEASE-p1 FreeBSD 11.1-RELEASE-p1 #299 r313908+13ee0afae40(RELENG_2_4): Sat Sep 23 19:54:08 CDT 2017 root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense Crash report details: No PHP errors found. Filename: /var/crash/bounds 1 Filename: /var/crash/info.0 Dump header from device: /dev/da0s1b Architecture: amd64 Architecture Version: 1 Dump Length: 86016 Blocksize: 512 Dumptime: Sun Sep 24 19:07:58 2017 Hostname: pfsense_3.local Magic: FreeBSD Text Dump Version String: FreeBSD 11.1-RELEASE-p1 #299 r313908+13ee0afae40(RELENG_2_4): Sat Sep 23 19:54:08 CDT 2017 root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense Panic String: hfsc_dequeue: Dump Parity: 1670814533 Bounds: 0 Dump Status: good Filename: /var/crash/info.last Dump header from device: /dev/da0s1b Architecture: amd64 Architecture Version: 1 Dump Length: 86016 Blocksize: 512 Dumptime: Sun Sep 24 19:07:58 2017 Hostname: pfsense_3.local Magic: FreeBSD Text Dump Version String: FreeBSD 11.1-RELEASE-p1 #299 r313908+13ee0afae40(RELENG_2_4): Sat Sep 23 19:54:08 CDT 2017 root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense Panic String: hfsc_dequeue: Dump Parity: 1670814533 Bounds: 0 Dump Status: good Filename: /var/crash/minfree 2048 Filename: /var/crash/textdump.tar.0 ddb.txt06000014000013161763156 7103 ustarrootwheeldb:0:kdb.enter.default> run lockinfo db:1:lockinfo> show locks No such command db:1:locks> show alllocks No such command db:1:alllocks> show lockedvnods Locked vnodes db:0:kdb.enter.default> show pcpu cpuid = 1 dynamic pcpu = 0xfffffe00981a3200 curthread = 0xfffff80003215560: pid 0 "em4 taskq" curpcb = 0xfffffe002b4f4b80 fpcurthread = none idlethread = 0xfffff800031df560: tid 100004 "idle: cpu1" curpmap = 0xffffffff82a3a7c0 tssp = 0xffffffff82a738f8 commontssp = 0xffffffff82a738f8 rsp0 = 0xfffffe002b4f4b80 gs32p = 0xffffffff82a7a150 ldt = 0xffffffff82a7a190 tss = 0xffffffff82a7a180 db:0:kdb.enter.default> bt Tracing pid 0 tid 100036 td 0xfffff80003215560 kdb_enter() at kdb_enter+0x3b/frame 0xfffffe002b4f3cd0 vpanic() at vpanic+0x1a3/frame 0xfffffe002b4f3d50 panic() at panic+0x43/frame 0xfffffe002b4f3db0 hfsc_dequeue() at hfsc_dequeue+0x1a6/frame 0xfffffe002b4f3df0 tbr_dequeue() at tbr_dequeue+0xea/frame 0xfffffe002b4f3e40 vlan_start() at vlan_start+0x196/frame 0xfffffe002b4f3e90 if_transmit() at if_transmit+0x15c/frame 0xfffffe002b4f3ed0 ether_output() at ether_output+0x718/frame 0xfffffe002b4f3f60 ip_output() at ip_output+0x154a/frame 0xfffffe002b4f40b0 ip_forward() at ip_forward+0x323/frame 0xfffffe002b4f4150 ip_input() at ip_input+0x75a/frame 0xfffffe002b4f41b0 netisr_dispatch_src() at netisr_dispatch_src+0xa0/frame 0xfffffe002b4f4200 ng_iface_rcvdata() at ng_iface_rcvdata+0x11a/frame 0xfffffe002b4f4230 ng_apply_item() at ng_apply_item+0xde/frame 0xfffffe002b4f42b0 ng_snd_item() at ng_snd_item+0x18c/frame 0xfffffe002b4f42f0 ng_apply_item() at ng_apply_item+0xde/frame 0xfffffe002b4f4370 ng_snd_item() at ng_snd_item+0x18c/frame 0xfffffe002b4f43b0 ng_ppp_rcvdata() at ng_ppp_rcvdata+0xe6b/frame 0xfffffe002b4f4430 ng_apply_item() at ng_apply_item+0xde/frame 0xfffffe002b4f44b0 ng_snd_item() at ng_snd_item+0x18c/frame 0xfffffe002b4f44f0 ng_apply_item() at ng_apply_item+0xde/frame 0xfffffe002b4f4570 ng_snd_item() at ng_snd_item+0x18c/frame 0xfffffe002b4f45b0 ng_apply_item() at ng_apply_item+0xde/frame 0xfffffe002b4f4630 ng_snd_item() at ng_snd_item+0x18c/frame 0xfffffe002b4f4670 ether_demux() at ether_demux+0x240/frame 0xfffffe002b4f46a0 ether_nh_input() at ether_nh_input+0x310/frame 0xfffffe002b4f4700 netisr_dispatch_src() at netisr_dispatch_src+0xa0/frame 0xfffffe002b4f4750 ether_input() at ether_input+0x26/frame 0xfffffe002b4f4770 vlan_input() at vlan_input+0x1f0/frame 0xfffffe002b4f47f0 ether_demux() at ether_demux+0x156/frame 0xfffffe002b4f4820 ether_nh_input() at ether_nh_input+0x310/frame 0xfffffe002b4f4880 netisr_dispatch_src() at netisr_dispatch_src+0xa0/frame 0xfffffe002b4f48d0 ether_input() at ether_input+0x26/frame 0xfffffe002b4f48f0 if_input() at if_input+0xa/frame 0xfffffe002b4f4900 lem_rxeof() at lem_rxeof+0x3df/frame 0xfffffe002b4f49a0 lem_handle_rxtx() at lem_handle_rxtx+0x32/frame 0xfffffe002b4f49e0 taskqueue_run_locked() at taskqueue_run_locked+0x127/frame 0xfffffe002b4f4a40 taskqueue_thread_loop() at taskqueue_thread_loop+0xc8/frame 0xfffffe002b4f4a70 fork_exit() at fork_exit+0x85/frame 0xfffffe002b4f4ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe002b4f4ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- -
Afraid this is really an uphill battle and altq is pretty much dead… Not sure what's the plan in FreeBSD but this buggy mess is not maintainable by Netgate alone.
-
Yes its reliant on FreeBSD upstream, pfSense may refine the code, but you entering a dangerous situation if you start making large en masse changes from the upstream product as this would need maintaining outside of upstream and be a pain on every upstream update.
FreeBSD made a decision to not keep up with openbsd in regards to PF and ALTQ so they made the rod on their own back with that decision, the good news is that they have started to fix FreeBSD specific PF issues now and I would hope this includes ALTQ as well.
The problem in this case there is no specific confirmed bug, just a few guys reporting crashes at high load, it needs lots of testing to dig deeper.
My advice was on the basis I am aware of scheduler panic fixes and igb panic fixes in FreeBSD 11.1 upstream hence what I posted.
-
FreeBSD made a decision to not keep up with openbsd in regards to PF and ALTQ so they made the rod on their own back with that decision, the good news is that they have started to fix FreeBSD specific PF issues now and I would hope this includes ALTQ as well.
ALTQ has been gone from OpenBSD for years. http://undeadly.org/cgi?action=article&sid=20140419151959
-
my crash log attached as follows, im also having same issue on client box when i enable shaping, sorry the log was too long to capture completely
-
A quick update from my side.
Both ALTQ and HFSC crash on 2.4.1 as well.
It doesn’t have to have any load. It crashes only seconds after reboot.Will try to upload the crash info tonight.
-
FreeBSD made a decision to not keep up with openbsd in regards to PF and ALTQ so they made the rod on their own back with that decision, the good news is that they have started to fix FreeBSD specific PF issues now and I would hope this includes ALTQ as well.
ALTQ has been gone from OpenBSD for years. http://undeadly.org/cgi?action=article&sid=20140419151959
It is gone now yes, but before it was removed, it was updated and improved over what the version in FreeBSD is.
-
Luiz fixed it.!. ;D
https://redmine.pfsense.org/issues/7879#note-3
At least after initial testing with and pushing some traffic through my queue's which seemed like a sure way to crash it before, i'm gone try and see how it holds up tomorrow :)
2.4.0-RC (amd64) built on Mon Sep 25 15:04:23 CDT 2017 FreeBSD 11.0-RELEASE-p12Perhaps some of you guy's here can also give it another testrun.? 8)
-
i tried on 2 apu2 boxes and now works all good, can any1 point me to the actual commit that fixed this?
-
It's running for a few hours now.
No crash since the reboot, so this looks promising.
