Recurring Crash 2.7.0
-
I believe this crash is happening during a large file transfer over IPSEC regularly. I apologize I am not adept in reading these reports in any way. Can someone please assist?
TIA!
-JoshCrash report begins. Anonymous machine information:
amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 RELENG_2_7_0-n255866-686c8d3c1f0: Wed Jun 28 04:21:19 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/obj/amd64/LwYAddCr/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/sources/FreeBSD-src-RELCrash report details:
No PHP errors found. Filename: /var/crash/info.0 Dump header from device: /dev/label/swap0 Architecture: amd64 Architecture Version: 4 Dump Length: 157696 Blocksize: 512 Compression: none Dumptime: 2023-12-14 04:23:22 +0000 Hostname: ManaFW1.manaproducts.local Magic: FreeBSD Text Dump Version String: FreeBSD 14.0-CURRENT #1 RELENG_2_7_0-n255866-686c8d3c1f0: Wed Jun 28 04:21:19 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/obj/amd64/LwYAddCr/var/jenkins/ Panic String: page fault Dump Parity: 1285908248 Bounds: 0 Dump Status: good Filename: /var/crash/textdump.tar.0 ddb.txt db:0:kdb.enter.default> show registers cs 0x20 ds 0x3b es 0x3b fs 0x13 gs 0x1b ss 0x28 rax 0x12 rcx 0x1 rdx 0xfffffe000379d5e0 rbx 0x100 rsp 0xfffffe000379d9c0 rbp 0xfffffe000379d9c0 rsi 0x19 rdi 0xffffffff82d82918 vt_conswindow+0x10 r8 0 r9 0x304f _binary_elf_vdso_so_1_size+0x2a3f r10 0xffffffff82d82a58 vt_consdev r11 0xcedfc2df9afff59c r12 0 r13 0xfffffe000379db40 r14 0xfffffe000379da50 r15 0xfffffe0003fd6720 rip 0xffffffff80d43122 kdb_enter+0x32 rflags 0x82 kdb_enter+0x32: movq $0,0x2347ce3(%rip) db:0:kdb.enter.default> run lockinfo db:1:lockinfo> show locks No such command; use "help" to list available commands db:1:lockinfo> show alllocks No such command; use "help" to list available commands db:1:lockinfo> show lockedvnods Locked vnodes db:0:kdb.enter.default> show pcpu cpuid = 4 dynamic pcpu = 0xfffffe0080f995c0 curthread = 0xfffffe0003fd6720: pid 11 tid 100007 critnest 2 "idle: cpu4" curpcb = 0xfffffe0003fd6c40 fpcurthread = none idlethread = 0xfffffe0003fd6720: tid 100007 "idle: cpu4" self = 0xffffffff83a14000 curpmap = 0xffffffff8303ef30 tssp = 0xffffffff83a14384 rsp0 = 0xfffffe000379e000 kcr3 = 0x800000000387c002 ucr3 = 0xffffffffffffffff scr3 = 0x45a89ffd gs32p = 0xffffffff83a14404 ldt = 0xffffffff83a14444 tss = 0xffffffff83a14434 curvnet = 0 db:0:kdb.enter.default> bt Tracing pid 11 tid 100007 td 0xfffffe0003fd6720 kdb_enter() at kdb_enter+0x32/frame 0xfffffe000379d9c0 vpanic() at vpanic+0x183/frame 0xfffffe000379da10 panic() at panic+0x43/frame 0xfffffe000379da70 trap_fatal() at trap_fatal+0x409/frame 0xfffffe000379dad0 trap_pfault() at trap_pfault+0x4f/frame 0xfffffe000379db30 calltrap() at calltrap+0x8/frame 0xfffffe000379db30 --- trap 0xc, rip = 0xffffffff80b05c80, rsp = 0xfffffe000379dc00, rbp = 0xfffffe000379dc00 --- vmxnet3_isc_txd_credits_update() at vmxnet3_isc_txd_credits_update+0x20/frame 0xfffffe000379dc00 iflib_fast_intr_rxtx() at iflib_fast_intr_rxtx+0xf7/frame 0xfffffe000379dc60 intr_event_handle() at intr_event_handle+0x123/frame 0xfffffe000379dcd0 intr_execute_handlers() at intr_execute_handlers+0x4a/frame 0xfffffe000379dd00 Xapic_isr1() at Xapic_isr1+0xdc/frame 0xfffffe000379dd00 --- interrupt, rip = 0xffffffff8125b026, rsp = 0xfffffe000379ddd0, rbp = 0xfffffe000379ddd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe000379ddd0 acpi_cpu_idle() at acpi_cpu_idle+0x2fe/frame 0xfffffe000379de10 cpu_idle_acpi() at cpu_idle_acpi+0x48/frame 0xfffffe000379de30 cpu_idle() at cpu_idle+0x9e/frame 0xfffffe000379de50 sched_idletd() at sched_idletd+0x4d1/frame 0xfffffe000379def0 fork_exit() at fork_exit+0x7d/frame 0xfffffe000379df30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000379df30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db:0:kdb.enter.default> ps pid ppid pgrp uid state wmesg wchan cmd 99985 35254 21 0 S nanslp 0xffffffff8305e5e2 sleep 33872 33542 32473 0 S piperd 0xfffffe00566726c0 sh 33542 32473 32473 0 S wait 0xfffffe008988eab8 sh 33451 32473 32473 0 S (threaded) sshg-blocker 100264 S piperd 0xfffffe0056671b60 sshg-blocker 100320 S nanslp 0xffffffff8305e5e4 sshg-blocker 33223 32473 32473 0 S piperd 0xfffffe0056841888 sshg-parser 33016 32473 32473 0 S piperd 0xfffffe00568415b0 cat 32473 58490 32473 0 Ss wait 0xfffffe008989d568 sh 80094 79814 78726 0 S nanslp 0xffffffff8305e5e3 sleep 79814 1 78726 0 S wait 0xfffffe00569fd568 sh 93334 1 93334 0 Ss+ ttyin 0xfffff800060590b0 getty 93144 1 93144 0 Ss+ ttyin 0xfffff800060598b0 getty 93089 1 93089 0 Ss+ ttyin 0xfffff800060580b0 getty 93061 1 93061 0 Ss+ ttyin 0xfffff800060588b0 getty 93048 1 93048 0 Ss+ ttyin 0xfffff800060570b0 getty 93006 1 93006 0 Ss+ ttyin 0xfffff800060578b0 getty 92953 1 92953 0 Ss+ ttyin 0xfffff800060560b0 getty 92910 1 92910 0 Ss+ ttyin 0xfffff800060568b0 getty 85867 1 83913 0 S (threaded) vmtoolsd 100243 S select 0xfffff800063eeb40 vmtoolsd 100250 S uwait 0xfffff80004971000 pool-spawner 100251 S uwait 0xfffff8000606cd00 HangDetector 61180 60755 60755 0 S nanslp 0xffffffff8305e5e4 minicron 60755 1 60755 0 Ss wait 0xfffffe008989b000 minicron 60699 60460 60460 0 S nanslp 0xffffffff8305e5e5 minicron 60460 1 60460 0 Ss wait 0xfffffe00568bfab8 minicron 60296 59998 59998 0 S nanslp 0xffffffff8305e5e1 minicron 59998 1 59998 0 Ss wait 0xfffffe005666d010 minicron 59703 59270 59270 0 S nanslp 0xffffffff8305e5e1 minicron 59270 1 59270 0 Ss wait 0xfffffe0056676008 minicron 58490 1 58490 0 Ss select 0xfffff80073a24640 syslogd 35254 1 21 0 S+ wait 0xfffffe0056678570 sh 32769 1 32769 0 Ss select 0xfffff800062f93c0 bsnmpd 31937 31653 31653 0 S (threaded) charon 100166 S sigwait 0xfffffe005666db70 charon 100200 S select 0xfffff800063ee940 charon 100201 S uwait 0xfffff80006521180 charon 100202 S select 0xfffff800062f9740 charon 100203 S uwait 0xfffff80073a24700 charon 100204 S uwait 0xfffff80073a24a00 charon 100205 S uwait 0xfffff80073a24b00 charon 100206 S uwait 0xfffff80073a24c00 charon 100207 S uwait 0xfffff80073a24d00 charon 100208 S uwait 0xfffff8000606b180 charon 100209 S uwait 0xfffff8000606b280 charon 100210 S uwait 0xfffff8000606b380 charon 100211 S uwait 0xfffff8000606b480 charon 100212 S uwait 0xfffff8000606b580 charon 100213 S uwait 0xfffff8000606b680 charon 100214 S uwait 0xfffff8000606b780 charon 100215 S uwait 0xfffff8000606b880 charon 31653 1 31653 0 Ss piperd 0xfffffe00566735b0 daemon 55681 55486 55486 0 S kqread 0xfffff80072e5ae00 nginx 55518 55486 55486 0 S kqread 0xfffff80073c78e00 nginx 55486 1 55486 0 Ss pause 0xfffffe0056659b68 nginx 54863 1 54863 0 Ss (threaded) ntpd 100180 S select 0xfffff800062eb140 ntpd 53693 1 53693 0 Ss select 0xfffff800062f84c0 sshd 51879 1 51879 0 Ss nanslp 0xffffffff8305e5e4 cron 50278 411 411 0 S accept 0xfffff80006771950 php-fpm 40031 1 40031 0 Ss (threaded) dpinger 100165 S uwait 0xfffff80004971500 dpinger 100185 S sbwait 0xfffff80006b951cc dpinger 100186 S nanslp 0xffffffff8305e5e3 dpinger 100187 S nanslp 0xffffffff8305e5e3 dpinger 100188 S accept 0xfffff80006ca31d0 dpinger 13150 1 13150 0 Ss bpf 0xfffff800739f5200 filterlog 683 1 683 0 Ss select 0xfffff80004972640 devd 452 450 450 0 S kqread 0xfffff8000629bb00 check_reload_status 450 1 450 0 Ss kqread 0xfffff80006067b00 check_reload_status 413 411 411 0 S accept 0xfffff80006771950 php-fpm 412 411 411 0 S accept 0xfffff80006771950 php-fpm 411 1 411 0 Ss kqread 0xfffff80006066000 php-fpm 20 0 0 0 DL aldslp 0xffffffff8303f950 [ALQ Daemon] 19 0 0 0 DL syncer 0xffffffff830ae968 [syncer] 18 0 0 0 DL vlruwt 0xfffffe0003fd5018 [vnlru] 17 0 0 0 DL (threaded) [bufdaemon] 100140 D psleep 0xffffffff830ad970 [bufdaemon] 100141 D - 0xffffffff82c0b140 [bufspacedaemon-0] 100150 D sdflush 0xfffff800062b58e8 [/ worker] 16 0 0 0 DL psleep 0xffffffff830d8e90 [vmdaemon] 9 0 0 0 DL (threaded) [pagedaemon] 100138 D psleep 0xffffffff830cd2f8 [dom0] 100142 D launds 0xffffffff830cd304 [laundry: dom0] 100143 D umarcl 0xffffffff8113c3c0 [uma] 8 0 0 0 DL - 0xffffffff82e9f688 [rand_harvestq] 7 0 0 0 DL pftm 0xffffffff80fadc20 [pf purge] 6 0 0 0 DL idle 0xfffffe005130a000 [mpt_recovery0] 5 0 0 0 DL - 0xffffffff830e2ad8 [busdma] 4 0 0 0 DL (threaded) [cam] 100084 D - 0xffffffff82e67fc0 [doneq0] 100085 D - 0xffffffff82e68040 [doneq1] 100086 D - 0xffffffff82e67f40 [async] 100136 D - 0xffffffff82e67e10 [scanner] 15 0 0 0 DL seqstat 0xfffff80004705c88 [sequencer 00] 3 0 0 0 DL (threaded) [crypto] 100075 D crypto_ 0xffffffff830ca900 [crypto] 100076 D crypto_ 0xfffff80004470830 [crypto returns 0] 100077 D crypto_ 0xfffff80004470880 [crypto returns 1] 100078 D crypto_ 0xfffff800044708d0 [crypto returns 2] 100079 D crypto_ 0xfffff80004470920 [crypto returns 3] 100080 D crypto_ 0xfffff80004470970 [crypto returns 4] 100081 D crypto_ 0xfffff800044709c0 [crypto returns 5] 14 0 0 0 DL (threaded) [geom] 100066 D - 0xffffffff8303dbb8 [g_event] 100067 D - 0xffffffff8303dbc0 [g_up] 100068 D - 0xffffffff8303dbc8 [g_down] 13 0 0 0 DL (threaded) [ng_queue] 100059 D sleep 0xffffffff82ddc670 [ng_queue0] 100060 D sleep 0xffffffff82ddc670 [ng_queue1] 100061 D sleep 0xffffffff82ddc670 [ng_queue2] 100062 D sleep 0xffffffff82ddc670 [ng_queue3] 100063 D sleep 0xffffffff82ddc670 [ng_queue4] 100064 D sleep 0xffffffff82ddc670 [ng_queue5] 2 0 0 0 WL (threaded) [clock] 100053 I [clock (0)] 100054 I [clock (1)] 100055 I [clock (2)] 100056 I [clock (3)] 100057 I [clock (4)] 100058 I [clock (5)] 12 0 0 0 RL (threaded) [intr] 100024 I [swi6: task queue] 100027 I [swi6: Giant taskq] 100031 I [swi5: fast taskq] 100047 RunQ [swi1: netisr 0] 100048 I [swi1: netisr 1] 100049 I [swi1: netisr 2] 100050 Run CPU 2 [swi1: netisr 3] 100051 I [swi1: netisr 4] 100052 I [swi1: netisr 5] 100088 I [irq14: ata0] 100089 I [irq15: ata1] 100090 I [irq24: pcib3] 100091 I [irq25: mpt0] 100093 I [irq26: pcib4] 100094 I [irq32: pcib5] 100095 I [irq33: pcib6] 100096 I [irq34: pcib7] 100097 I [irq35: pcib8] 100098 I [irq36: pcib9] 100099 I [irq37: pcib10] 100100 I [irq38: pcib11] 100101 I [irq44: pcib12] 100102 I [irq50: pcib13] 100103 I [irq51: pcib14] 100104 I [irq52: pcib15] 100105 I [irq53: pcib16] 100106 I [irq54: pcib17] 100107 I [irq55: pcib18] 100108 I [irq56: pcib19] 100109 I [irq62: pcib20] 100110 I [irq63: pcib21] 100111 I [irq64: pcib22] 100112 I [irq65: pcib23] 100113 I [irq66: pcib24] 100114 I [irq67: pcib25] 100115 I [irq68: pcib26] 100116 I [irq69: pcib27] 100117 I [irq75: pcib28] 100118 I [irq76: pcib29] 100119 I [irq77: pcib30] 100120 I [irq78: pcib31] 100121 I [irq79: pcib32] 100122 I [irq80: pcib33] 100123 I [irq81: pcib34] 100124 I [irq1: atkbd0] 100125 I [irq12: psm0] 100128 I [swi1: pf send] 100129 I [swi1: pfsync] 11 0 0 0 RL (threaded) [idle] 100003 Run CPU 0 [idle: cpu0] 100004 Run CPU 1 [idle: cpu1] 100005 CanRun [idle: cpu2] 100006 Run CPU 3 [idle: cpu3] 100007 Run CPU 4 [idle: cpu4] 100008 Run CPU 5 [idle: cpu5] 1 0 1 0 SLs wait 0xfffffe0003fd4010 [init] 10 0 0 0 DL audit_w 0xffffffff830cade8 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D swapin 0xffffffff8303e150 [swapper] 100009 D - 0xfffff80004468a00 [if_io_tqg_0] 100010 D - 0xfffff80004468800 [if_io_tqg_1] 100011 D - 0xfffff80004468600 [if_io_tqg_2] 100012 D - 0xfffff80004468400 [if_io_tqg_3] 100013 D - 0xfffff80004468200 [if_io_tqg_4] 100014 D - 0xfffff80004468000 [if_io_tqg_5] 100015 D - 0xfffff80004467d00 [if_config_tqg_0] 100016 D - 0xfffff80004467b00 [softirq_0] 100017 D - 0xfffff80004467900 [softirq_1] 100018 D - 0xfffff80004467700 [softirq_2] 100019 D - 0xfffff80004467500 [softirq_3] 100020 D - 0xfffff80004467300 [softirq_4] 100021 D - 0xfffff80004467100 [softirq_5] 100022 D - 0xfffff8000452a300 [linuxkpi_irq_wq] 100023 D - 0xfffff8000452a100 [pci_hp taskq] 100025 D - 0xfffff8000452bb00 [aiod_kick taskq] 100026 D - 0xfffff8000452b900 [deferred_unmount ta] 100028 D - 0xfffff8000452b400 [inm_free taskq] 100029 D - 0xfffff8000452b200 [thread taskq] 100030 D - 0xfffff8000452b000 [in6m_free taskq] 100032 D - 0xfffff8000452ca00 [kqueue_ctx taskq] 100033 D - 0xfffff8000452c800 [linuxkpi_short_wq_0] 100034 D - 0xfffff8000452c800 [linuxkpi_short_wq_1] 100035 D - 0xfffff8000452c800 [linuxkpi_short_wq_2] 100036 D - 0xfffff8000452c800 [linuxkpi_short_wq_3] 100037 D - 0xfffff8000452c800 [linuxkpi_short_wq_4] 100038 D - 0xfffff8000452c800 [linuxkpi_short_wq_5] 100039 D - 0xfffff8000452c800 [linuxkpi_short_wq_6] 100040 D - 0xfffff8000452c000 [linuxkpi_long_wq_0] 100041 D - 0xfffff8000452c000 [linuxkpi_long_wq_1] 100042 D - 0xfffff8000452c000 [linuxkpi_long_wq_2] 100043 D - 0xfffff8000452c000 [linuxkpi_long_wq_3] 100044 D - 0xfffff8000452c000 [linuxkpi_long_wq_4] 100045 D - 0xfffff8000452c000 [linuxkpi_long_wq_5] 100046 D - 0xfffff8000452c000 [linuxkpi_long_wq_6] 100065 D - 0xfffff800045d5d00 [firmware taskq] 100069 D - 0xfffff800045d6800 [crypto_0] 100070 D - 0xfffff800045d6800 [crypto_1] 100071 D - 0xfffff800045d6800 [crypto_2] 100072 D - 0xfffff800045d6800 [crypto_3] 100073 D - 0xfffff800045d6800 [crypto_4] 100074 D - 0xfffff800045d6800 [crypto_5] 100082 D - 0xfffff800045d7700 [t4_rst_thr] 100126 D - 0xfffff80004f9d100 [ip_mroute_tskq task] 100131 D - 0xfffff8000605c800 [acpi_task_0] 100132 D - 0xfffff8000605c800 [acpi_task_1] 100133 D - 0xfffff8000605c800 [acpi_task_2] 100134 D - 0xfffff8000446b100 [mca taskq] 100135 D - 0xfffff8000452ab00 [CAM taskq] 100137 D - 0xfffff8000605d500 [mlx4] db:0:kdb.enter.default> alltrace -
Backtrace:
db:0:kdb.enter.default> bt Tracing pid 11 tid 100007 td 0xfffffe0003fd6720 kdb_enter() at kdb_enter+0x32/frame 0xfffffe000379d9c0 vpanic() at vpanic+0x183/frame 0xfffffe000379da10 panic() at panic+0x43/frame 0xfffffe000379da70 trap_fatal() at trap_fatal+0x409/frame 0xfffffe000379dad0 trap_pfault() at trap_pfault+0x4f/frame 0xfffffe000379db30 calltrap() at calltrap+0x8/frame 0xfffffe000379db30 --- trap 0xc, rip = 0xffffffff80b05c80, rsp = 0xfffffe000379dc00, rbp = 0xfffffe000379dc00 --- vmxnet3_isc_txd_credits_update() at vmxnet3_isc_txd_credits_update+0x20/frame 0xfffffe000379dc00 iflib_fast_intr_rxtx() at iflib_fast_intr_rxtx+0xf7/frame 0xfffffe000379dc60 intr_event_handle() at intr_event_handle+0x123/frame 0xfffffe000379dcd0 intr_execute_handlers() at intr_execute_handlers+0x4a/frame 0xfffffe000379dd00 Xapic_isr1() at Xapic_isr1+0xdc/frame 0xfffffe000379dd00 --- interrupt, rip = 0xffffffff8125b026, rsp = 0xfffffe000379ddd0, rbp = 0xfffffe000379ddd0 --- acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe000379ddd0 acpi_cpu_idle() at acpi_cpu_idle+0x2fe/frame 0xfffffe000379de10 cpu_idle_acpi() at cpu_idle_acpi+0x48/frame 0xfffffe000379de30 cpu_idle() at cpu_idle+0x9e/frame 0xfffffe000379de50 sched_idletd() at sched_idletd+0x4d1/frame 0xfffffe000379def0 fork_exit() at fork_exit+0x7d/frame 0xfffffe000379df30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000379df30 --- trap 0, rip = 0, rsp = 0, rbp = 0 ---We've seen that a few times and looked into it. We submitted a bug fix for it upstream: https://reviews.freebsd.org/D43712
Disabling multi-queue support prevents it if you're hitting it repeatedly.
Increasing the descriptor counts in the tunables will make it happen less frequently. But will still eventually hit it.
Steve
