Where do I start trying to understand a Kernel Panic
-
2.3.4-RELEASE-p1 (i386)
built on Fri Jul 14 14:53:03 CDT 2017
FreeBSD 10.3-RELEASE-p19Hardware: Soekris net6501-70, 4port Lan 1841 card
I'm getting infrequent Kernel Panics - the first two were during the summer (where I assumed that there was a overheating issue, but the most recent two can't be attributed to the temperature.
I have the following crash dump, but I have no idea where to start in order to isolate the cause (or even determine what module caused the issue. I've also attached the full dump. Can anyone provide some pointers (no pun intended!)
Many thanks.
Crash report begins. Anonymous machine information: i386 10.3-RELEASE-p19 FreeBSD 10.3-RELEASE-p19 #1 76a12c4e6(RELENG_2_3_4): Fri Jul 14 15:03:37 CDT 2017 root@ce23-i386-builder:/builder/pfsense-234/tmp/obj/builder/pfsense-234/tmp/FreeBSD-src/sys/pfSense Crash report details: No PHP errors found. Filename: /var/crash/bounds 1 Filename: /var/crash/info.0 Dump header from device /dev/label/swap0 Architecture: i386 Architecture Version: 1 Dump Length: 140800B (0 MB) Blocksize: 512 Dumptime: Fri Nov 17 09:40:23 2017 Hostname: pfsense.localdomain Magic: FreeBSD Text Dump Version String: FreeBSD 10.3-RELEASE-p19 #1 76a12c4e6(RELENG_2_3_4): Fri Jul 14 15:03:37 CDT 2017 root@ce23-i386-builder:/builder/pfsense-234/tmp/obj/builder/pfsense-234/tmp/FreeBSD-src/sys/pfSense Panic String: double fault Dump Parity: 3586614602 Bounds: 0 Dump Status: good Filename: /var/crash/info.last Dump header from device /dev/label/swap0 Architecture: i386 Architecture Version: 1 Dump Length: 140800B (0 MB) Blocksize: 512 Dumptime: Fri Nov 17 09:40:23 2017 Hostname: pfsense.localdomain Magic: FreeBSD Text Dump Version String: FreeBSD 10.3-RELEASE-p19 #1 76a12c4e6(RELENG_2_3_4): Fri Jul 14 15:03:37 CDT 2017 root@ce23-i386-builder:/builder/pfsense-234/tmp/obj/builder/pfsense-234/tmp/FreeBSD-src/sys/pfSense Panic String: double fault Dump Parity: 3586614602 Bounds: 0 Dump Status: good Filename: /var/crash/textdump.tar.0 ddb.txt06000014000013203527207 7073 ustarrootwheeldb:0:kdb.enter.default> run lockinfo db:1:lockinfo> show locks No such command db:1:locks> show alllocks No such command db:1:alllocks> show lockedvnods Locked vnodes db:0:kdb.enter.default> show pcpu cpuid = 0 dynamic pcpu = 0x56f200 curthread = 0xc7f88000: pid 60886 "sh" curpcb = 0xf18f8d40 fpcurthread = none idlethread = 0xc76fd000: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 db:0:kdb.enter.default> bt Tracing pid 60886 tid 100137 td 0xc7f88000 kdb_enter(c147cc16,c147cc16,c1643ee7,c1fb7994,0,...) at kdb_enter+0x3d/frame 0xc1fb7940 vpanic(c1643ee7,c1fb7994,c1fb7994,c1fb79ac,c12e7feb,...) at vpanic+0x13b/frame 0xc1fb7974 panic(c1643ee7,0,0,0,f18f8b6c,...) at panic+0x1b/frame 0xc1fb7988 dblfault_handler() at dblfault_handler+0xab/frame 0xc1fb7988 --- trap 0x17, eip = 0xc12d2558, esp = 0xf18f8008, ebp = 0xf18f8b6c --- Xpage(c2795bf4,1000,bfc00000,c7e33e04,0,...) at Xpage/frame 0xf18f8b6c vmspace_exit(c7f88000,c7e33c20,1,0,f18f8bd8,...) at vmspace_exit+0xe5/frame 0xf18f8b98 exit1(c7f88000,0,f18f8cdc,c12e86a9,c7f88000,...) at exit1+0x879/frame 0xf18f8c08 sys_sys_exit(c7f88000,f18f8ca8,c7e33c20,f18f8c64,c0d1c042,...) at sys_sys_exit+0x1d/frame 0xf18f8c18 syscall(f18f8ce8) at syscall+0x5c9/frame 0xf18f8cdc Xint0x80_syscall() at Xint0x80_syscall+0x2f/frame 0xf18f8cdc --- syscall (1, FreeBSD ELF32, sys_sys_exit), eip = 0x281ad2f3, esp = 0xbfbfed0c, ebp = 0xbfbfed98 --- db:0:kdb.enter.default> ps pid ppid pgrp uid state wmesg wchan cmd 60886 81474 21 0 RE CPU 0 sh 40355 41051 292 0 S nanslp 0xc1f7aa81 sleep 57388 292 292 0 S accept 0xc7ff31e6 php-fpm 54576 54241 54576 0 S+ ttyin 0xc77c1e70 sh 54241 54084 54241 0 S+ wait 0xca1f5918 sh 54139 2735 54139 0 Ss (threaded) sshlockout_pf 100227 S nanslp 0xc1f7aa80 sshlockout_pf 100203 S piperd 0xc97dd330 sshlockout_pf 54084 1 54084 0 Ss+ wait 0xc7e33308 login 53982 1 1 0 S nanslp 0xc1f7aa81 getty 53681 50140 49356 100 S select 0xc98a1a24 pinger 53313 50140 49356 100 S piperd 0xc7e6c4c8 unlinkd 50140 49356 49356 100 S kqread 0xc8c77100 squid 49356 1 49356 100 Ss wait 0xc7db4918 squid 41051 1 292 0 S wait 0xca55f610 sh 40695 40419 40419 0 S nanslp 0xc1f7aa81 minicron 40419 1 40419 0 Ss wait 0xca55fc20 minicron 40303 39898 39898 0 S nanslp 0xc1f7aa80 minicron 39898 1 39898 0 Ss wait 0xca560000 minicron 39638 39067 39067 0 S nanslp 0xc1f7aa81 minicron 39067 1 39067 0 Ss wait 0xc7e32308 minicron 25060 1 25060 0 Ss (threaded) filterdns 100217 S uwait 0xc8c75280 signal-thread 100216 S uwait 0xca216c00 carinfo.kiev.ua 100215 S uwait 0xca216c80 www.bmwarchiv.de 100214 S uwait 0xca216d00 bmwarchiv.de 100213 S uwait 0xca216d80 v4.ident.me 2735 1 2735 0 Ss select 0xc799a9e4 syslogd 81474 1 21 0 S+ wait 0xca1f7000 sh 78400 1 78400 0 Ss select 0xc7ddd3e4 bsnmpd 77388 77079 77388 0 Ss (threaded) charon 100199 S uwait 0xc8c40a00 charon 100198 S uwait 0xc8646780 charon 100197 S uwait 0xc8646700 charon 100196 S uwait 0xc8646680 charon 100195 S uwait 0xc8c84880 charon 100194 S uwait 0xc8c84900 charon 100193 S uwait 0xc8c84600 charon 100192 S uwait 0xc8c86e00 charon 100191 S uwait 0xc8c86600 charon 100190 S uwait 0xc8c87600 charon 100189 S uwait 0xc8c41500 charon 100188 S select 0xc799a964 charon 100187 S uwait 0xc8c84380 charon 100186 S select 0xc799aa24 charon 100185 S uwait 0xc7dfa680 charon 100184 S uwait 0xc7dfbe80 charon 100130 S sigwait 0xc9722000 charon 77079 1 77079 0 Ss select 0xc98a3224 starter 60758 1 60559 0 S select 0xc7dc1224 radvd 57951 1 57951 1002 Ss select 0xc7dc14e4 dhcpd 52063 1 52063 0 Ss (threaded) ntpd 100129 S select 0xc7dc1724 ntpd 51319 1 51319 0 Ss nanslp 0xc1f7aa80 cron 51199 50944 50944 0 S kqread 0xc8c88080 nginx 51128 50944 50944 0 S kqread 0xc8648a00 nginx 50944 1 50944 0 Ss pause 0xc7db4064 nginx 46213 1 45987 65534 S select 0xc7dc23e4 dnsmasq 45214 1 45214 0 Ss kqread 0xc9700880 dhcpleases 36624 1 36624 0 Ss select 0xc7dc25e4 dhcp6c 33319 1 33319 0 Ss (threaded) dpinger 100178 S accept 0xc7ff203e dpinger 100177 S nanslp 0xc1f7aa80 dpinger 100176 S nanslp 0xc1f7aa80 dpinger 100175 S sbwait 0xc7ff2260 dpinger 100121 S uwait 0xc7dfce00 dpinger 32440 1 32440 0 Ss (threaded) dpinger 100174 S accept 0xc7fa0536 dpinger 100173 S nanslp 0xc1f7aa80 dpinger 100172 S nanslp 0xc1f7aa81 dpinger 100171 S sbwait 0xc7fa0758 dpinger 100160 S uwait 0xc7d91100 dpinger 32087 1 32087 0 Ss (threaded) dpinger 100170 S accept 0xc7fa3bd6 dpinger 100169 S nanslp 0xc1f7aa80 dpinger 100168 S nanslp 0xc1f7aa81 dpinger 100167 S sbwait 0xc7fa3900 dpinger 100155 S uwait 0xc7dfa300 dpinger 20753 1 20753 0 Ss bpf 0xc7e24200 filterlog 12076 1 12076 65 Ss select 0xc7dc3f24 dhclient 7822 1 7822 0 Ss select 0xc7ddb7a4 dhclient 6694 1 6694 0 Ss (threaded) sshlockout_pf 100142 S nanslp 0xc1f7aa81 sshlockout_pf 100141 S uwait 0xc7dfa000 sshlockout_pf 6436 1 6436 0 Ss select 0xc7ddb864 sshd 344 1 344 0 Ss select 0xc7dddda4 devd 333 331 331 0 S kqread 0xc7d8fd00 check_reload_status 331 1 331 0 Ss kqread 0xc7d90200 check_reload_status 292 1 292 0 Ss kqread 0xc7d90100 php-fpm 55 0 0 0 DL mdwait 0xc7d8e800 [md0] 20 0 0 0 DL vlruwt 0xc7db4c20 [vnlru] 19 0 0 0 DL syncer 0xc1f99dc4 [syncer] 18 0 0 0 DL psleep 0xc1f99504 [bufdaemon] 17 0 0 0 DL pgzero 0xc1fa3fa8 [pagezero] 9 0 0 0 DL pollid 0xc1f795f0 [idlepoll] 8 0 0 0 DL psleep 0xc1fa3cc4 [vmdaemon] 7 0 0 0 DL (threaded) [pagedaemon] 100111 D umarcl 0xc1fa38a4 [uma] 100103 D psleep 0xc2027404 [pagedaemon] 6 0 0 0 DL waiting_ 0xc2021d74 [sctp_iterator] 5 0 0 0 DL pftm 0xc0f95180 [pf purge] 16 0 0 0 DL (threaded) [usb] 100072 D - 0xc78b2d64 [usbus7] 100071 D - 0xc78b2d34 [usbus7] 100070 D - 0xc78b2d04 [usbus7] 100069 D - 0xc78b2cd4 [usbus7] 100068 D - 0xc78b2ca4 [usbus7] 100067 D - 0xc78dab8c [usbus6] 100066 D - 0xc78dab5c [usbus6] 100065 D - 0xc78dab2c [usbus6] 100064 D - 0xc78daafc [usbus6] 100063 D - 0xc78daacc [usbus6] 100062 D - 0xc78d2b8c [usbus5] 100061 D - 0xc78d2b5c [usbus5] 100060 D - 0xc78d2b2c [usbus5] 100059 D - 0xc78d2afc [usbus5] 100058 D - 0xc78d2acc [usbus5] 100057 D - 0xc7898b8c [usbus4] 100056 D - 0xc7898b5c [usbus4] 100055 D - 0xc7898b2c [usbus4] 100054 D - 0xc7898afc [usbus4] 100053 D - 0xc7898acc [usbus4] 100048 D - 0xc785ed64 [usbus3] 100047 D - 0xc785ed34 [usbus3] 100046 D - 0xc785ed04 [usbus3] 100045 D - 0xc785ecd4 [usbus3] 100044 D - 0xc785eca4 [usbus3] 100043 D - 0xc788fb8c [usbus2] 100042 D - 0xc788fb5c [usbus2] 100041 D - 0xc788fb2c [usbus2] 100040 D - 0xc788fafc [usbus2] 100039 D - 0xc788facc [usbus2] 100038 D - 0xc7885b8c [usbus1] 100037 D - 0xc7885b5c [usbus1] 100036 D - 0xc7885b2c [usbus1] 100035 D - 0xc7885afc [usbus1] 100034 D - 0xc7885acc [usbus1] 100033 D - 0xc7867b8c [usbus0] 100032 D - 0xc7867b5c [usbus0] 100031 D - 0xc7867b2c [usbus0] 100030 D - 0xc7867afc [usbus0] 100029 D - 0xc7867acc [usbus0] 4 0 0 0 DL (threaded) [cam] 100102 D - 0xc1eb8cac [scanner] 100020 D - 0xc1eb8e00 [doneq0] 15 0 0 0 DL - 0xc1ed5ec8 [rand_harvestq] 3 0 0 0 DL crypto_r 0xc1fa2c28 [crypto returns] 2 0 0 0 DL crypto_w 0xc1fa2b68 [crypto] 14 0 0 0 DL (threaded) [geom] 100013 D - 0xc201cddc [g_down] 100012 D - 0xc201cdd8 [g_up] 100011 D - 0xc201cdd4 [g_event] 13 0 0 0 DL (threaded) [ng_queue] 100010 D sleep 0xc1e9063c [ng_queue1] 100009 D sleep 0xc1e9063c [ng_queue0] 12 0 0 0 WL (threaded) [intr] 100110 I [swi1: netisr 1] 100100 I [swi1: pfsync] 100098 I [swi1: pf send] 100095 I [irq15: ata1] 100094 I [irq14: ata0] 100093 I [irq270: em3:link] 100091 I [irq269: em3:tx0] 100089 I [irq268: em3:rx0] 100088 I [irq267: em2:link] 100086 I [irq266: em2:tx0] 100084 I [irq265: em2:rx0] 100083 I [irq264: em1:link] 100081 I [irq263: em1:tx0] 100079 I [irq262: em1:rx0] 100078 I [irq261: em0:link] 100076 I [irq260: em0:tx0] 100074 I [irq259: em0:rx0] 100073 I [swi0: uart uart+++] 100052 I [irq16: ohci3 ohci4*] 100051 I [irq258: ahci0] 100050 I [irq257: sdhci_pci1] 100049 I [irq256: sdhci_pci0] 100028 I [irq19: ohci0 ohci1*] 100027 I [swi6: Giant taskq] 100025 I [swi6: task queue] 100019 I [swi5: fast taskq] 100008 I [swi4: clock] 100007 I [swi4: clock] 100006 I [swi1: netisr 0] 100005 I [swi3: vm] 11 0 0 0 RL (threaded) [idle] 100004 Run CPU 1 [idle: cpu1] 100003 CanRun [idle: cpu0] 1 0 1 0 SLs wait 0xc76f6308 [init] 10 0 0 0 DL audit_wo 0xc2025cdc [audit] 0 0 0 0 DLs (threaded) [kernel] 100150 D - 0xc8647400 [dummynet] 100101 D - 0xc76f9300 [CAM taskq] 100096 D - 0xc798aa80 [mca taskq] 100092 D - 0xc78dfc80 [em3 txq (cpuid 1)] 100090 D - 0xc78dfd00 [em3 rxq (cpuid 0)] 100087 D - 0xc78f9280 [em2 txq (cpuid 1)] 100085 D - 0xc78f9300 [em2 rxq (cpuid 0)] 100082 D - 0xc78f9b80 [em1 txq (cpuid 1)] 100080 D - 0xc78f9c00 [em1 rxq (cpuid 0)] 100077 D - 0xc78fa900 [em0 txq (cpuid 1)] 100075 D - 0xc78fa980 [em0 rxq (cpuid 0)] 100026 D - 0xc75c8d00 [ffs_trim taskq] 100024 D - 0xc76f9000 [kqueue taskq] 100023 D - 0xc76f9080 [acpi_task_2] 100022 D - 0xc76f9080 [acpi_task_1] 100021 D - 0xc76f9080 [acpi_task_0] 100018 D - 0xc76f9480 [thread] 100014 D - 0xc76fa800 [firmware taskq] 100000 D swapin 0xc201ce54 [swapper] db:0:kdb.enter.default> alltrace [pfSense Panic - 2017-11-17.txt](/public/_imported_attachments_/1/pfSense Panic - 2017-11-17.txt)[/thread]