Firewall Crash Report Generated
-
Hello Everyone,
Found out that my Netgate 5100 running 25.07.1-RELEASE has a crash report and files generated.Any assistance in diagnosing why it crashed would be greatly appreciated. It occurred while out of town - so internet usage was at a minimum, no major configuration changes. Device has been up since it crashed -relatively "stable". snippet of logs:
Crash report begins. Anonymous machine information: amd64 15.0-CURRENT FreeBSD 15.0-CURRENT #0 plus-RELENG_25_07_1-n256513-49844af35a5d: Fri Aug 15 19:21:04 UTC 2025 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-25_07_1-main/obj/amd64/DZizCvOj/var/jenkins/workspace/pfSense-Plus-snapshots-25_07_1-main/sources Crash report details: No PHP errors found. Filename: /var/crash/info.0 Dump header from device: /dev/label/swap0 Architecture: amd64 Architecture Version: 4 Dump Length: 219648 Blocksize: 512 Compression: none Dumptime: 2025-11-20 12:04:20 -0500 Hostname: gateway. Magic: FreeBSD Text Dump Version String: FreeBSD 15.0-CURRENT #0 plus-RELENG_25_07_1-n256513-49844af35a5d: Fri Aug 15 19:21:04 UTC 2025 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-25_07_1-main/obj/amd64/DZizCvOj/ Panic String: page fault Dump Parity: 342753842 Bounds: 0 Dump Status: good Filename: /var/crash/textdump.tar.0 ddb.txt���������������������������������������������������������������������������������������������0600����0�������0�������551535������15107645024� 7121� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������db:0:kdb.enter.default> run pfs db:1:pfs> bt Tracing pid 9912 tid 100435 td 0xfffff80330c2e740 kdb_enter() at kdb_enter+0x33/frame 0xfffffe00d8a80ae0 panic() at panic+0x43/frame 0xfffffe00d8a80b40 trap_fatal() at trap_fatal+0x40b/frame 0xfffffe00d8a80ba0 trap_pfault() at trap_pfault+0x46/frame 0xfffffe00d8a80bf0 calltrap() at calltrap+0x8/frame 0xfffffe00d8a80bf0 --- trap 0xc, rip = 0xffffffff8114abd9, rsp = 0xfffffe00d8a80cc0, rbp = 0xfffffe00d8a80d40 --- softdep_fsync() at softdep_fsync+0x379/frame 0xfffffe00d8a80d40 ffs_fsync() at ffs_fsync+0x61/frame 0xfffffe00d8a80d80 kern_fsync() at kern_fsync+0xe0/frame 0xfffffe00d8a80e00 amd64_syscall() at amd64_syscall+0x115/frame 0xfffffe00d8a80f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00d8a80f30 --- syscall (95, FreeBSD ELF64, fsync), rip = 0x826b69a6a, rsp = 0x8205207e8, rbp = 0x820520800 --- db:1:pfs> show registers cs 0x20 ds 0x3b es 0x3b fs 0x13 gs 0x1b ss 0 rax 0x12 rcx 0x3cf2c11f008d5ab7 rdx 0xffffffff838195ff rbx 0x100 rsp 0xfffffe00d8a809b8 rbp 0xfffffe00d8a80ae0 rsi 0xfffffe00d8a80870 rdi 0xffffffff82750a00 gdb_consdev r8 0x10 r9 0x10 r10 0xf r11 0x10 r12 0 r13 0 r14 0xffffffff8146862e r15 0xfffff80330c2e740 rip 0xffffffff80d4e493 kdb_enter+0x33 rflags 0x86 kdb_enter+0x33: movq $0,0x1d70072(%rip) db:1:pfs> show pcpu cpuid = 1 dynamic pcpu = 0xfffffe009db624c0 curthread = 0xfffff80330c2e740: pid 9912 tid 100435 critnest 1 "pfSense-repoc" curpcb = 0xfffff80330c2ec60 fpcurthread = 0xfffff80330c2e740: pid 9912 "pfSense-repoc" idlethread = 0xfffff801027e9000: tid 100004 "idle: cpu1" self = 0xffffffff83411000 curpmap = 0xfffff8000d5a6a30 tssp = 0xffffffff83411384 rsp0 = 0xfffffe00d8a81000 kcr3 = 0xffffffffffffffff ucr3 = 0xffffffffffffffff scr3 = 0x0 gs32p = 0xffffffff83411404 ldt = 0xffffffff83411444 tss = 0xffffffff83411434 curvnet = 0 db:1:pfs> run lockinfo db:2:lockinfo> show locks No such command; use "help" to list available commands db:2:lockinfo> show alllocks No such command; use "help" to list available commands db:2:lockinfo> show lockedvnods Locked vnodes db:1:pfs> acttrace Tracing command intr pid 12 tid 100044 td 0xfffff80102c09000 (CPU 3) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe00203a9ee0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe00203a9ef0 nmi_handle_intr() at nmi_handle_intr+0x12/frame 0xfffffe00203a9f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe00203a9f20 --- trap 0x13, rip = 0xffffffff81004f2a, rsp = 0xfffffe00d86e3a30, rbp = 0xfffffe00d86e3aa0 --- pf_normalize_ip() at pf_normalize_ip+0xaa/frame 0xfffffe00d86e3aa0 pf_test() at pf_test+0x451/frame 0xfffffe00d86e3d30 pf_check_in() at pf_check_in+0x33/frame 0xfffffe00d86e3d60 pfil_mbuf_in() at pfil_mbuf_in+0x38/frame 0xfffffe00d86e3d90 ip_input() at ip_input+0x3ab/frame 0xfffffe00d86e3df0 swi_net() at swi_net+0x128/frame 0xfffffe00d86e3e60 ithread_loop() at ithread_loop+0x239/frame 0xfffffe00d86e3ef0 fork_exit() at fork_exit+0x7b/frame 0xfffffe00d86e3f30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00d86e3f30 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Tracing command pfSense-repoc pid 9912 tid 100435 td 0xfffff80330c2e740 (CPU 1) kdb_enter() at kdb_enter+0x33/frame 0xfffffe00d8a80ae0 panic() at panic+0x43/frame 0xfffffe00d8a80b40 trap_fatal() at trap_fatal+0x40b/frame 0xfffffe00d8a80ba0 trap_pfault() at trap_pfault+0x46/frame 0xfffffe00d8a80bf0 calltrap() at calltrap+0x8/frame 0xfffffe00d8a80bf0 --- trap 0xc, rip = 0xffffffff8114abd9, rsp = 0xfffffe00d8a80cc0, rbp = 0xfffffe00d8a80d40 --- softdep_fsync() at softdep_fsync+0x379/frame 0xfffffe00d8a80d40 ffs_fsync() at ffs_fsync+0x61/frame 0xfffffe00d8a80d80 kern_fsync() at kern_fsync+0xe0/frame 0xfffffe00d8a80e00 amd64_syscall() at amd64_syscall+0x115/frame 0xfffffe00d8a80f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00d8a80f30 --- syscall (95, FreeBSD ELF64, fsync), rip = 0x826b69a6a, rsp = 0x8205207e8, rbp = 0x820520800 --- Tracing command php-fpm pid 39272 tid 100621 td 0xfffff80330d20000 (CPU 2) cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe002039aee0 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xfffffe002039aef0 nmi_handle_intr() at nmi_handle_intr+0x12/frame 0xfffffe002039af20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe002039af20 --- trap 0x13, rip = 0x63d9c7, rsp = 0x820d5e980, rbp = 0x820d5e980 --- Tracing command php-fpm pid 56227 tid 100575 td 0xfffff80330c85740 (CPU 0) cpustop_handler() at cpustop_handler+0x27/frame 0xffffffff82b2a070 ipi_nmi_handler() at ipi_nmi_handler+0x38/frame 0xffffffff82b2a080 nmi_handle_intr() at nmi_handle_intr+0x12/frame 0xffffffff82b2a0b0 nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff82b2a0b0 --- trap 0x13, rip = 0xffffffff80d7567d, rsp = 0xfffffe00d8aeeba0, rbp = 0xfffffe00d8aeebb0 --- selfdfree() at selfdfree+0x8d/frame 0xfffffe00d8aeebb0 seltdclear() at seltdclear+0x2b/frame 0xfffffe00d8aeebd0 kern_select() at kern_select+0xbb6/frame 0xfffffe00d8aeedc0 sys_select() at sys_select+0x57/frame 0xfffffe00d8aeee00 amd64_syscall() at amd64_syscall+0x115/frame 0xfffffe00d8aeef30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00d8aeef30 --- syscall (93, FreeBSD ELF64, select), rip = 0x829815c6a, rsp = 0x820d5e6c8, rbp = 0x820d5e700 --- db:1:pfs> ps pid ppid pgrp uid state wmesg wchan cmd 22450 22137 479 65534 R pkg-static 22137 21112 479 0 S wait 0xfffffe00c6f375c0 pkg-static 21112 16576 479 0 S select 0xfffff802219528c0 pfSense-repoc-stati 16576 9650 479 0 S wait 0xfffffe00c6f32b00 sh 11684 39272 479 0 R pfSense-repoc 9912 56227 479 0 R CPU 1 pfSense-repoc 9650 9509 479 0 S wait 0xfffffe00c6f1f060 sh 9509 8716 479 0 S wait 0xfffffe00c6f1dae0 lockf 8716 8470 479 0 S wait 0xfffffe00c6f2ab00 sh 8470 1 479 0 S wait 0xfffffe00c6f165a0 sh 90777 88854 479 0 R unbound-control 88854 88593 479 0 S piperd 0xfffffe00e1851cc0 php 88593 88005 479 0 S wait 0xfffffe00c6f325a0 sh 88005 45833 479 0 S wait 0xfffffe00c6e0f5c0 sh 30913 16902 58371 0 S+ nanslp 0xffffffff82aa5a22 sleep 39272 479 479 0 R CPU 2 php-fpm 16902 1 58371 0 S+ wait 0xfffffe00c6f1c000 sh 15348 14787 14787 0 S kqread 0xfffff80326ed7a00 nginx 15064 14787 14787 0 S kqread 0xfffff80006598000 nginx 14787 1 14787 0 Ss pause 0xfffffe00c6f35630 nginx 58371 58122 58371 0 Ss+ ttyin 0xfffff80012d68cb0 sh 58122 60810 58122 0 Ss select 0xfffff800060b3d40 sshd 31810 1 61190 0 S select 0xfffff803cdff46c0 tail_pfb 31013 1 61190 0 S piperd 0xfffffe00e18699e0 php_pfb 30972 1 61190 0 S kqread 0xfffff802bd581c00 tail_pfb 76903 76495 76341 0 S piperd 0xfffffe00e1dc5e60 php 76643 76495 76341 0 S piperd 0xfffffe00e1dc89e0 php 76495 1 76341 0 S kqread 0xfffff803a82c0e00 lighttpd_pfb 98802 1 98802 59 Rs (threaded) unbound 100646 RunQ unbound 862224 S kqread 0xfffff802bde89200 unbound 862225 S kqread 0xfffff8034d407300 unbound 862226 S kqread 0xfffff8001282e700 unbound 52016 51676 50775 0 S nanslp 0xffffffff82aa5a22 sleep 51676 1 50775 0 S wait 0xfffffe00c6f21ae0 sh 11079 0 0 0 DL (threaded) [KTLS] 100525 D - 0xfffff8035ecf2800 [thr_0] 456960 D - 0xfffff8035ecf2880 [thr_1] 456961 D - 0xfffff8035ecf2900 [thr_2] 456962 D - 0xfffff8035ecf2980 [thr_3] 456963 D - 0xffffffff82ad6c58 [reclaim_0] 20505 479 479 0 S (threaded) php-fpm 100169 S accept 0xfffff8000673da10 php-fpm 47977 479 479 0 S (threaded) php-fpm 147939 S accept 0xfffff8000673da10 php-fpm 56227 479 479 0 R (threaded) php-fpm 100575 Run CPU 0 php-fpm 45833 1 479 0 S (threaded) kea-dhcp4 100652 S select 0xfffff802211ea9c0 kea-dhcp4 855535 S uwait 0xfffff8035e080100 kea-dhcp4 855536 S uwait 0xfffff80006efe800 kea-dhcp4 855537 S uwait 0xfffff80221e57300 kea-dhcp4 855538 S uwait 0xfffff803a8724c80 kea-dhcp4 855539 S uwait 0xfffff803cddb5c00 kea-dhcp4 855540 S uwait 0xfffff80006a7ff00 kea-dhcp4 855541 S uwait 0xfffff800048d2c00 kea-dhcp4 855542 S uwait 0xfffff802c6d64880 kea-dhcp4 579 479 479 0 S (threaded) php-fpm 100622 S accept 0xfffff8000673da10 php-fpm 59533 479 479 0 S (threaded) php-fpm 100130 S accept 0xfffff8000673da10 php-fpm 84825 1 84825 0 Ss bpf 0xfffff803cd274000 arpwatch 84375 1 84375 0 Ss bpf 0xfffff800044ae600 arpwatch 84019 1 84019 0 Ss bpf 0xfffff802fcd31c00 arpwatch 83619 1 83619 0 Ss bpf 0xfffff800048a6800 arpwatch 73550 73050 73050 949 S kqread 0xfffff800121d6000 lldpd 73050 1 73050 0 Ss sbwait 0xfffff803a80d2e0c lldpd 66443 1 66443 0 Ss nanslp 0xffffffff82aa5a22 vnstatd 45703 45651 45703 65534 Ss sbwait 0xfffff8000652ba0c darkstat 45651 1 45651 65534 Ss select 0xfffff8035ee8d6c0 darkstat 41437 1 41437 0 Ss select 0xfffff80326a2bcc0 bsnmpd 75565 1 75565 0 Ss (threaded) dpinger 100618 S uwait 0xfffff803cd9d8280 dpinger 682017 S sbwait 0xfffff8000647460c dpinger 682018 S nanslp 0xffffffff82aa5a22 dpinger 682019 S nanslp 0xffffffff82aa5a21 dpinger 682020 S accept 0xfffff80006586e10 dpinger 75115 1 75115 0 Ss (threaded) dpinger 100641 S uwait 0xfffff803cd4aae00 dpinger 682013 S sbwait 0xfffff8000673960c dpinger 682014 S nanslp 0xffffffff82aa5a20 dpinger 682015 S nanslp 0xffffffff82aa5a22 dpinger 682016 S accept 0xfffff80006563210 dpinger 60484 479 479 0 S (threaded) php-fpm 100209 S accept 0xfffff8000673da10 php-fpm 40059 354 40059 0 S+ ttyin 0xfffff800044554b0 sh 1580 135 1580 0 S+ ttyin 0xfffff800047f50b0 sh 354 99479 354 0 S+ wait 0xfffffe00c6e09020 sh 135 97930 135 0 S+ wait 0xfffffe00c6f1e5a0 sh 99479 1 99479 0 Ss+ wait 0xfffffe00c6f24ac0 login 99467 1 99467 0 Ss+ ttyin 0xfffff800061b68b0 getty 99451 1 99451 0 Ss+ ttyin 0xfffff800061b50b0 getty 99124 1 99124 0 Ss+ ttyin 0xfffff800061b58b0 getty 98826 1 98826 0 Ss+ ttyin 0xfffff800061b40b0 getty 98573 1 98573 0 Ss+ ttyin 0xfffff800061b48b0 getty 98253 1 98253 0 Ss+ ttyin 0xfffff800061b30b0 getty 98077 1 98077 0 Ss+ ttyin 0xfffff800061b38b0 getty 97930 1 97930 0 Ss+ wait 0xfffffe00c6e04ac0 login 42366 41926 41926 0 S nanslp 0xffffffff82aa5a22 minicron 41926 1 41926 0 Ss wait 0xfffffe00c6f17060 minicron 41822 41250 41250 0 S nanslp 0xffffffff82aa5a22 minicron 41250 1 41250 0 Ss wait 0xfffffe00c6f235c0 minicron 41230 41025 41025 0 S nanslp 0xffffffff82aa5a20 minicron 41025 1 41025 0 Ss wait 0xfffffe00c6f16040 minicron 40644 40382 40382 0 S nanslp 0xffffffff82aa5a20 minicron 40382 1 40382 0 Ss wait 0xfffffe00c6e065a0 minicron 37605 1 37605 0 Ss (threaded) filterdns 100507 S usem 0xfffff802fcd7ac00 filterdns 100508 S usem 0xfffff80006bb3880 192.168.89 100509 S usem 0xfffff801253d4400 merge-thread 13065 1 13065 0 Ss select 0xfffff8012538d340 syslogd 12777 1 7751 0 S select 0xfffff802fcc7b0c0 syslogd 11823 9130 7591 0 S piperd 0xfffffe00e17beb80 sh 11792 1 11792 0 Ss select 0xfffff801251098c0 sshg-parser 10815 1 10815 0 Ss select 0xfffff800060b35c0 sshg-blocker 9821 1 7591 0 S select 0xfffff80326a2bc40 tail 9130 7882 7591 0 S wait 0xfffffe00c6f0b580 sh 9088 7882 7591 0 S (threaded) sshg-blocker 100173 S piperd 0xfffffe00e1851700 sshg-blocker 100490 S nanslp 0xffffffff82aa5a20 sshg-blocker 8812 7882 7591 0 S piperd 0xfffffe00e17bf140 sshg-parser 8546 7882 7591 0 S kqread 0xfffff803a8871400 tail 7882 7591 7591 0 S wait 0xfffffe00c6f1b5c0 sh 7751 1 7751 0 Ss kqread 0xfffff80326113a00 syslogd 7591 1 7591 0 Ss kqread 0xfffff80326f06d00 daemon 10792 1 10792 0 Ss select 0xfffff80006ee5dc0 miniupnpd 5601 1 5601 0 Ss nanslp 0xffffffff82aa5a20 watchdogd 1047 1 1047 0 Ss (threaded) ntpd 100129 S select 0xfffff8035e74e840 ntpd 40767 1 40767 0 Ss nanslp 0xffffffff82aa5a20 cron 60810 1 60810 0 Ss select 0xfffff800065dd3c0 sshd 97113 1 97113 0 Ss bpf 0xfffff800044ad600 filterlog 75405 1 75405 65 Ss select 0xfffff800060b4dc0 dhclient 63764 1 63764 0 Ss select 0xfffff802c6d642c0 dhclient 62246 1 62246 0 Ss select 0xfffff80006baccc0 dhclient 41872 1 41872 65 Ss select 0xfffff80006b437c0 dhclient 31978 1 31978 0 Ss select 0xfffff801253d4740 dhclient 89909 1 89909 0 Ss select 0xfffff80125252240 dhclient 1202 1 1202 0 Ss select 0xfffff800060b3240 devd 567 565 565 0 S kqread 0xfffff8000653b800 check_reload_status 565 1 565 0 Ss kqread 0xfffff8000690f200 check_reload_status 479 1 479 0 Ss kqread 0xfffff800060bd900 php-fpm 26 0 0 0 DL idle 0xfffffe00c6e0a5a0 [enc_daemon1] 25 0 0 0 DL idle 0xfffffe00c6e0ab00 [enc_daemon0] 24 0 0 0 DL aldslp 0xffffffff82a62a18 [ALQ Daemon] 23 0 0 0 DL syncer 0xffffffff82adf8e8 [syncer] 22 0 0 0 DL vlruwt 0xfffffe00c6e13ac0 [vnlru] 21 0 0 0 DL (threaded) [bufdaemon] 100105 D psleep 0xffffffff82ade908 [bufdaemon] 100106 D - 0xffffffff82602100 [bufspacedaemon-0] 100107 D - 0xffffffff82602380 [bufspacedaemon-1] 100108 D - 0xffffffff82602600 [bufspacedaemon-2] 100109 D - 0xffffffff82602880 [bufspacedaemon-3] 100110 D - 0xffffffff82602b00 [bufspacedaemon-4] 100111 D - 0xffffffff82602d80 [bufspacedaemon-5] 100112 D - 0xffffffff82603000 [bufspacedaemon-6] 100124 D sdflush 0xfffff8000600dce8 [/ worker] 20 0 0 0 DL psleep 0xffffffff82b16678 [vmdaemon] 19 0 0 0 DL (threaded) [pagedaemon] 100103 D psleep 0xffffffff82afcbb8 [dom0] 100113 D launds 0xffffffff82afcbc4 [laundry: dom0] 100114 D umarcl 0xffffffff8117f270 [uma] 18 0 0 0 DL mmcsd d 0xfffff8000442e900 [mmcsd0boot1: mmc/sd] 9 0 0 0 DL mmcsd d 0xfffff8000442ed00 [mmcsd0boot0: mmc/sd] 8 0 0 0 DL mmcsd d 0xfffff800061c4200 [mmcsd0: mmc/sd card] 7 0 0 0 DL - 0xffffffff828a6790 [rand_harvestq] 17 0 0 0 DL tzpoll 0xffffffff8287c4f8 [acpi_thermal] 6 0 0 0 DL pftm 0xffffffff80fd5c10 [pf purge] 16 0 0 0 DL (threaded) [usb] 100078 D - 0xfffffe00dadf8440 [usbus0] 100079 D - 0xfffffe00dadf8498 [usbus0] 100080 D - 0xfffffe00dadf84f0 [usbus0] 100081 D - 0xfffffe00dadf8548 [usbus0] 100082 D - 0xfffffe00dadf85a0 [usbus0] 5 0 0 0 DL - 0xffffffff827f72a0 [busdma] 4 0 0 0 DL (threaded) [cam] 100065 D - 0xffffffff82857240 [doneq0] 100066 D - 0xffffffff828571c0 [async] 100097 D - 0xffffffff82857090 [scanner] 3 0 0 0 DL (threaded) [crypto] 100060 D crypto_ 0xffffffff82afa1c0 [crypto] 100061 D crypto_ 0xfffff800041ea1b0 [crypto returns 0] 100062 D crypto_ 0xfffff800041ea200 [crypto returns 1] 100063 D crypto_ 0xfffff800041ea250 [crypto returns 2] 100064 D crypto_ 0xfffff800041ea2a0 [crypto returns 3] 15 0 0 0 DL seqstat 0xfffff80004623888 [sequencer 00] 14 0 0 0 DL (threaded) [geom] 100051 D - 0xffffffff82a61438 [g_event] 100052 D - 0xffffffff82a61440 [g_up] 100053 D - 0xffffffff82a61448 [g_down] 13 0 0 0 DL (threaded) [ng_queue] 100046 D sleep 0xffffffff8279f250 [ng_queue0] 100047 D sleep 0xffffffff8279f250 [ng_queue1] 100048 D sleep 0xffffffff8279f250 [ng_queue2] 100049 D sleep 0xffffffff8279f250 [ng_queue3] 2 0 0 0 WL (threaded) [clock] 100038 I [clock (0)] 100039 I [clock (1)] 100040 I [clock (2)] 100041 I [clock (3)] 12 0 0 0 RL (threaded) [intr] 100018 I [swi6: task queue] 100019 I [swi6: Giant taskq] 100021 I [swi5: fast taskq] 100042 I [swi1: netisr 0] 100043 I [swi1: netisr 1] 100044 Run CPU 3 [swi1: netisr 2] 100045 I [swi1: netisr 3] 100068 I [irq42: ahci0] 100069 I [irq43: ahci1:ch0] 100070 I [irq44: ahci1:ch1] 100071 I [irq45: ahci1:ch2] 100072 I [irq46: ahci1:ch3] 100073 I [irq47: ahci1:ch4] 100074 I [irq48: ahci1:ch5] 100075 I [irq49: ahci1:ch6] 100076 I [irq50: ahci1:ch7] 100077 I [irq51: xhci0] 100083 I [swi0: uart uart+++] 100084 I [irq16: sdhci_pci0] 100085 I [swi4: apei] 100088 I [swi1: pf send] 100089 I [swi1: pfsync] 100158 I [irq23: ichsmb0] 11 0 0 0 RL (threaded) [idle] 100003 CanRun [idle: cpu0] 100004 CanRun [idle: cpu1] 100005 CanRun [idle: cpu2] 100006 CanRun [idle: cpu3] 1 0 1 0 SLs wait 0xfffffe00c6e02040 [init] 10 0 0 0 DL audit_w 0xffffffff82afa6d0 [audit] 0 0 0 0 DLs (threaded) [kernel] 100000 D parked 0xffffffff83272ff0 [swapper] 100007 D - 0xfffff800041ed400 [softirq_0] 100008 D - 0xfffff800041ed200 [softirq_1] 100009 D - 0xfffff800041ed000 [softirq_2] 100010 D - 0xfffff80004376200 [softirq_3] 100011 D - 0xfffff80004376000 [if_io_tqg_0] 100012 D - 0xfffff80004377d00 [if_io_tqg_1] 100013 D - 0xfffff80004377b00 [if_io_tqg_2] 100014 D - 0xfffff80004377900 [if_io_tqg_3] 100015 D - 0xfffff80004377700 [if_config_tqg_0] 100016 D - 0xfffff80004377500 [pci_hp taskq] 100017 D - 0xfffff80004377300 [kqueue_ctx taskq] 100020 D - 0xfffff80004378a00 [thread taskq] 100022 D - 0xfffff80004378500 [aiod_kick taskq] 100023 D - 0xfffff80004378300 [deferred_unmount ta] 100024 D - 0xfffff80004378100 [inm_free taskq] 100025 D - 0xfffff80004379e00 [in6m_free taskq] 100026 D - 0xfffff80004379c00 [linuxkpi_irq_wq] 100027 D - 0xfffff80004379a00 [linuxkpi_short_wq_0] 100028 D - 0xfffff80004379a00 [linuxkpi_short_wq_1] 100029 D - 0xfffff80004379a00 [linuxkpi_short_wq_2] 100030 D - 0xfffff80004379a00 [linuxkpi_short_wq_3] 100031 D - 0xfffff80004379a00 [linuxkpi_short_wq_4] 100032 D - 0xfffff80004379400 [linuxkpi_long_wq_0] 100033 D - 0xfffff80004379400 [linuxkpi_long_wq_1] 100034 D - 0xfffff80004379400 [linuxkpi_long_wq_2] 100035 D - 0xfffff80004379400 [linuxkpi_long_wq_3] 100036 D - 0xfffff80004379400 [linuxkpi_long_wq_4] 100050 D - 0xfffff80004358d00 [firmware taskq] 100054 D - 0xfffff80004359900 [t4_rst_thr] 100056 D - 0xfffff80004359300 [crypto_0] 100057 D - 0xfffff80004359300 [crypto_1] 100058 D - 0xfffff80004359300 [crypto_2] 100059 D - 0xfffff80004359300 [crypto_3] 100086 D - 0xfffff800048b1400 [ip_mroute_tskq task] 100090 D - 0xfffff800061c4c00 [acpi_task_0] 100091 D - 0xfffff800061c4c00 [acpi_task_1] 100092 D - 0xfffff800061c4c00 [acpi_task_2] 100095 D - 0xfffff800041efa00 [mca taskq] 100096 D - 0xfffff8000435a100 [CAM taskq] 100101 D - 0xfffff800061b0c00 [mlx4] 100102 D - 0xfffff800061b0800 [ipsec_offload] 100250 D - 0xfffff80125223e00 [dummynet] 682136 D - 0xfffff80006ba6b00 [netlink_socket (PID] -
@posix I usually shove these into ChatGPT (or whatever). The LLMs are very good at analyzing dense logs. This was the tl;dr:
"Bottom line
The crash is not caused by rules, PHP, or pfSense GUI misconfig; it is a kernel-mode page fault inside UFS soft-updates code while a pfSense package/repo process was calling fsync().
Most likely causes:
A bug in the UFS softdep path in your 15-CURRENT-based pfSense Plus kernel, and/or
Underlying hardware issues (RAM or eMMC/SD/SSD) triggering that code path.
Recommended actions:
Run memory and storage health checks.
Monitor for recurrence; if it repeats in similar circumstances, treat it as a kernel bug.
Plan a move to ZFS on solid-state storage for long-term robustness, or update pfSense Plus when a newer build is available.
If you’re willing, submit the full textdump.tar.0 to Netgate support so they can confirm and potentially fix it upstream.?
-
Thanks for the insight and direction, I will try with ChatGPT. Sure I don’t mind submitting dump the files and whatever else info to negate. What’s the best path to engage? (I don’t have support contract, power home user).
-
-
That looks like a UFS filesystem issue. Are you running UFS?
If so run an fsck:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html#manual-file-system-check -
Yes, I am running UFS
-
Thanks for jumping in and looking, I will need to look for console cable and get time to reboot into single user mode to run the FSCK, will share output once I am able to. In the mean time I ran it in read only:
/root: fsck -fy / ** /dev/ufsid/61bf5ededf06b426 (NO WRITE) ** Last Mounted on / ** Root file system ** Phase 1 - Check Blocks and Sizes INCORRECT BLOCK COUNT I=4735794 (8 should be 0) CORRECT? no INCORRECT BLOCK COUNT I=4735795 (8 should be 0) CORRECT? no INCORRECT BLOCK COUNT I=5457609 (1536 should be 1472) CORRECT? no INCORRECT BLOCK COUNT I=9871764 (8 should be 0) CORRECT? no INCORRECT BLOCK COUNT I=9871766 (112 should be 0) CORRECT? no ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts UNREF FILE I=9871810 OWNER=root MODE=100666 SIZE=0 MTIME=Nov 20 12:05 2025 CLEAR? no ** Phase 5 - Check Cyl groups FREE BLK COUNT(S) WRONG IN SUPERBLK SALVAGE? no SUMMARY INFORMATION BAD SALVAGE? no BLK(S) MISSING IN BIT MAPS SALVAGE? no 65257 files, 9935692 used, 49478624 free (17632 frags, 6182624 blocks, 0.0% fragmentation) -
Mmm, definitely run that from single user mode then. That's almost certainly what caused the crash.
-
happy new year and thank you all for your responses. I ended up doing a fresh install to 25.11 and migrating to ZFS.
I ran FSCK utility numerous times and it always came back clean.
Uptime 11 Days 09 Hours 27 Minutes 14 Seconds
has been stable since upgrade to 25.11.
Copyright 2026 Rubicon Communications LLC (Netgate). All rights reserved.