Help with Kernel Panic Crash dump
-
Hi Guys,
Recently we have been having our pFsense firewall crash and kernel panic. The only changes that have been made are by an upstream partner having issues on their firewall and having to make iPsec changes (we dont know what).
I have managed to get a crash report.I seem to be unable to work out from this what is the issue can anyone help at all?
Crash report begins. Anonymous machine information:
i386
10.3-RELEASE-p5
FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:53:47 CDT 2016 root@ce23-i386-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSenseCrash report details:
Filename: /var/crash/bounds
1Filename: /var/crash/info.0
Dump header from device /dev/ada0s1b
Architecture: i386
Architecture Version: 1
Dump Length: 153088B (0 MB)
Blocksize: 512
Dumptime: Thu Jan 26 15:06:01 2017
Hostname: platform-firewall-bhx.platform.local
Magic: FreeBSD Text Dump
Version String: FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:53:47 CDT 2016
root@ce23-i386-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense
Panic String: double fault
Dump Parity: 641131011
Bounds: 0
Dump Status: goodFilename: /var/crash/info.last
Dump header from device /dev/ada0s1b
Architecture: i386
Architecture Version: 1
Dump Length: 153088B (0 MB)
Blocksize: 512
Dumptime: Thu Jan 26 15:06:01 2017
Hostname: platform-firewall-bhx.platform.local
Magic: FreeBSD Text Dump
Version String: FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:53:47 CDT 2016
root@ce23-i386-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense
Panic String: double fault
Dump Parity: 641131011
Bounds: 0
Dump Status: goodFilename: /var/crash/textdump.tar.0
ddb.txt���������������������������������������������������������������������������������������������0600����0�������0�������140000������13042410131� 7057� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������db:0:kdb.enter.default> run lockinfo
db:1:lockinfo> show locks
No such command
db:1:locks> show alllocks
No such command
db:1:alllocks> show lockedvnods
Locked vnodes
db:0:kdb.enter.default> show pcpu
cpuid = 0
dynamic pcpu = 0x56f000
curthread = 0xc7716000: pid 11 "idle: cpu0"
curpcb = 0xe2b8cd40
fpcurthread = none
idlethread = 0xc7716000: tid 100003 "idle: cpu0"
APIC ID = 0
currentldt = 0x50
db:0:kdb.enter.default> bt
Tracing pid 11 tid 100003 td 0xc7716000
kdb_enter(c147c5b6,c147c5b6,c1643767,c1fb7b9c,0,…) at kdb_enter+0x3d/frame 0xc1fb7b48
vpanic(c1643767,c1fb7b9c,c1fb7b9c,c1fb7bb4,c12e79cb,...) at vpanic+0x13b/frame 0xc1fb7b7c
panic(c1643767,0,0,0,e2b8cc28,...) at panic+0x1b/frame 0xc1fb7b90
dblfault_handler() at dblfault_handler+0xab/frame 0xc1fb7b90
--- trap 0x17, eip = 0xc12d1f38, esp = 0xe2b8c008, ebp = 0xe2b8cc28 ---
Xpage(1,e2b8cc78,c147df53,a3d,6,...) at Xpage/frame 0xe2b8cc28
sched_idletd(0,e2b8cce8,c1f31ce8,0,1a5e,...) at sched_idletd+0x1dd/frame 0xe2b8cca4
fork_exit(c0d3f8b0,0,e2b8cce8) at fork_exit+0xa3/frame 0xe2b8ccd4
fork_trampoline() at fork_trampoline+0x8/frame 0xe2b8ccd4
--- trap 0, eip = 0, esp = 0xe2b8cd20, ebp = 0 ---
db:0:kdb.enter.default> ps
pid ppid pgrp uid state wmesg wchan cmd
12652 12020 21 0 RE CPU 1 tail
12020 45021 21 0 S wait 0xc8059c20 sh
94519 273 273 0 S accept 0xc80b31e6 php-fpm
30605 30565 30605 0 S+ ttyin 0xc794f670 sh
30565 30055 30565 0 S+ wait 0xc8059610 sh
30258 15937 30258 0 Ss (threaded) sshlockout_pf
100213 S nanslp 0xc1f7ac81 sshlockout_pf
100129 S piperd 0xc8ec87f8 sshlockout_pf
30055 1 30055 0 Ss+ wait 0xc8057610 login
29027 1 29027 0 Ss (threaded) filterdns
100212 S uwait 0xc8d72380 signal-thread
100211 S uwait 0xc8d72300 195.191.106.65
100210 S uwait 0xc8dc6300 146.101.60.47
100209 S uwait 0xc8dc6280 5.172.155.141
100208 S uwait 0xc8dc6780 WAN_SWI
19135 18884 18884 0 S nanslp 0xc1f7ac81 minicron
18884 1 18884 0 Ss wait 0xc7e6f918 minicron
18572 18232 18232 0 S nanslp 0xc1f7ac81 minicron
18232 1 18232 0 Ss wait 0xc8f18000 minicron
18098 17838 17838 0 S nanslp 0xc1f7ac81 minicron
17838 1 17838 0 Ss wait 0xc8058610 minicron
15937 1 15937 0 Ss select 0xc7ec4664 syslogd
45021 1 21 0 S+ piperd 0xc9566660 sh
33850 33602 33850 0 Ss (threaded) charon
100190 S uwait 0xc8dc6900 charon
100189 S uwait 0xc8dc6880 charon
100188 S uwait 0xc8dc6800 charon
100187 S uwait 0xc8dc6400 charon
100186 S uwait 0xc8d74e00 charon
100185 S uwait 0xc8d74d80 charon
100184 S uwait 0xc8d74d00 charon
100183 S uwait 0xc8dc8100 charon
100182 S uwait 0xc8dc8080 charon
100181 S uwait 0xc8dc8000 charon
100180 S uwait 0xc8dc7f00 charon
100179 S uwait 0xc8dc7e80 charon
100178 S uwait 0xc8dc7e00 charon
100177 S select 0xc7ec41e4 charon
100176 S uwait 0xc8dc7d00 charon
100175 S select 0xc7ec41a4 charon
100127 S sigwait 0xc8e9a000 charon
33602 1 33602 0 Ss select 0xc7ec4164 starter
28475 1 28475 0 Ss select 0xc7e69064 dhcrelay
25379 1 25100 65534 S select 0xc7e68b24 dnsmasq
22040 1 22040 0 Ss (threaded) ntpd
100143 S select 0xc7ec46e4 ntpd
20887 1 20887 0 Ss nanslp 0xc1f7ac80 cron
20237 19635 19635 0 S kqread 0xc8d75480 nginx
19971 19635 19635 0 S kqread 0xc8d75500 nginx
19635 1 19635 0 Ss pause 0xc8d83674 nginx
15820 1 15820 0 Ss (threaded) dpinger
100164 S accept 0xc804c6de dpinger
100163 S nanslp 0xc1f7ac81 dpinger
100162 S nanslp 0xc1f7ac81 dpinger
100161 S sbwait 0xc804c900 dpinger
100134 S uwait 0xc7ed6200 dpinger
15474 1 15474 0 Ss (threaded) dpinger
100160 S accept 0xc804cbd6 dpinger
100159 S nanslp 0xc1f7ac81 dpinger
100158 S nanslp 0xc1f7ac80 dpinger
100157 S sbwait 0xc804cdf8 dpinger
100136 S uwait 0xc7ed6180 dpinger
15163 1 15163 0 Ss (threaded) dpinger
100156 S accept 0xc80b1d7e dpinger
100155 S nanslp 0xc1f7ac81 dpinger
100154 S nanslp 0xc1f7ac81 dpinger
100153 S sbwait 0xc80b2260 dpinger
100142 S uwait 0xc7e5de80 dpinger
14751 1 14751 0 Ss (threaded) dpinger
100152 S accept 0xc804d38e dpinger
100151 S nanslp 0xc1f7ac80 dpinger
100150 S nanslp 0xc1f7ac80 dpinger
100149 S sbwait 0xc804d5b0 dpinger
100137 S uwait 0xc7ed6100 dpinger
12774 1 12774 0 Ss select 0xc7becca4 xinetd
11711 1 11711 0 Ss bpf 0xc7a5e000 filterlog
316 1 316 0 Ss select 0xc7e69ce4 devd
305 303 303 0 S kqread 0xc7ed5180 check_reload_status
303 1 303 0 Ss kqread 0xc7ec2c00 check_reload_status
273 1 273 0 Ss kqread 0xc7ec2280 php-fpm
56 0 0 0 DL mdwait 0xc7e52000 [md0]
20 0 0 0 DL vlruwt 0xc7e6dc20 [vnlru]
19 0 0 0 DL syncer 0xc1f99fc4 [syncer]
18 0 0 0 DL psleep 0xc1f99704 [bufdaemon]
17 0 0 0 DL pgzero 0xc1fa41a8 [pagezero]
9 0 0 0 DL pollid 0xc1f797f0 [idlepoll]
8 0 0 0 DL psleep 0xc1fa3ec4 [vmdaemon]
7 0 0 0 DL (threaded) [pagedaemon]
100115 D umarcl 0xc1fa3aa4 [uma]
100107 D psleep 0xc2027604 [pagedaemon]
6 0 0 0 DL waiting_ 0xc2021f74 [sctp_iterator]
5 0 0 0 DL pftm 0xc0f94ba0 [pf purge]
16 0 0 0 DL (threaded) [usb]
100096 D - 0xc7a15d64 [usbus4]
100095 D - 0xc7a15d34 [usbus4]
100094 D - 0xc7a15d04 [usbus4]
100093 D - 0xc7a15cd4 [usbus4]
100092 D - 0xc7a15ca4 [usbus4]
100091 D - 0xc79fcddc [usbus3]
100090 D - 0xc79fcdac [usbus3]
100089 D - 0xc79fcd7c [usbus3]
100088 D - 0xc79fcd4c [usbus3]
100087 D - 0xc79fcd1c [usbus3]
100085 D - 0xc79e7ddc [usbus2]
100084 D - 0xc79e7dac [usbus2]
100083 D - 0xc79e7d7c [usbus2]
100082 D - 0xc79e7d4c [usbus2]
100081 D - 0xc79e7d1c [usbus2]
100079 D - 0xc79cfddc [usbus1]
100078 D - 0xc79cfdac [usbus1]
100077 D - 0xc79cfd7c [usbus1]
100076 D - 0xc79cfd4c [usbus1]
100075 D - 0xc79cfd1c [usbus1]
100073 D - 0xc79b1ddc [usbus0]
100072 D - 0xc79b1dac [usbus0]
100071 D - 0xc79b1d7c [usbus0]
100070 D - 0xc79b1d4c [usbus0]
100069 D - 0xc79b1d1c [usbus0]
4 0 0 0 DL (threaded) [cam]
100106 D - 0xc1eb8eac [scanner]
100020 D - 0xc1eb9000 [doneq0]
15 0 0 0 DL - 0xc1ed60c8 [rand_harvestq]
3 0 0 0 DL crypto_r 0xc1fa2e28 [crypto returns]
2 0 0 0 DL crypto_w 0xc1fa2d68 [crypto]
14 0 0 0 DL (threaded) [geom]
100013 D - 0xc201cfd8 [g_down]
100012 D - 0xc201cfd4 [g_up]
100011 D - 0xc201cfd0 [g_event]
13 0 0 0 DL (threaded) [ng_queue]
100010 D sleep 0xc1e9081c [ng_queue1]
100009 D sleep 0xc1e9081c [ng_queue0]
12 0 0 0 WL (threaded) [intr]
100114 I [swi1: netisr 1]
100104 I [swi1: pfsync]
100102 I [swi1: pf send]
100099 I [irq1: atkbd0]
100098 I [swi0: uart]
100097 I [irq14: ata0]
100086 I [irq16: uhci3]
100080 I [irq18: uhci2+]
100074 I [irq19: uhci1]
100068 I [irq23: uhci0 ehci0]
100067 I [irq279: em7:link]
100065 I [irq278: em7:tx0]
100063 I [irq277: em7:rx0]
100062 I [irq276: em6:link]
100060 I [irq275: em6:tx0]
100058 I [irq274: em6:rx0]
100057 I [irq273: em5:link]
100055 I [irq272: em5:tx0]
100053 I [irq271: em5:rx0]
100052 I [irq270: em4:link]
100050 I [irq269: em4:tx0]
100048 I [irq268: em4:rx0]
100047 I [irq267: em3:link]
100045 I [irq266: em3:tx0]
100043 I [irq265: em3:rx0]
100042 I [irq264: em2:link]
100040 I [irq263: em2:tx0]
100038 I [irq262: em2:rx0]
100037 I [irq261: em1:link]
100035 I [irq260: em1:tx0]
100033 I [irq259: em1:rx0]
100032 I [irq258: em0:link]
100030 I [irq257: em0:tx0]
100028 I [irq256: em0:rx0]
100027 I [swi6: Giant taskq]
100025 I [swi6: task queue]
100019 I [swi5: fast taskq]
100008 I [swi4: clock]
100007 I [swi4: clock]
100006 I [swi1: netisr 0]
100005 I [swi3: vm]
11 0 0 0 RL (threaded) [idle]
100004 CanRun [idle: cpu1]
100003 Run CPU 0 [idle: cpu0]
1 0 1 0 SLs wait 0xc770f308 [init]
10 0 0 0 DL audit_wo 0xc2025edc [audit]
0 0 0 0 DLs (threaded) [kernel]
100148 D - 0xc8b74780 [dummynet]
100105 D - 0xc7712300 [CAM taskq]
100100 D - 0xc7a5d380 [mca taskq]
100066 D - 0xc7874e80 [em7 txq (cpuid 1)]
100064 D - 0xc7874f00 [em7 rxq (cpuid 0)]
100061 D - 0xc7905480 [em6 txq (cpuid 1)]
100059 D - 0xc7905500 [em6 rxq (cpuid 0)]
100056 D - 0xc7905a00 [em5 txq (cpuid 1)]
100054 D - 0xc7905a80 [em5 rxq (cpuid 0)]
100051 D - 0xc7906980 [em4 txq (cpuid 1)]
100049 D - 0xc7906a00 [em4 rxq (cpuid 0)]
100046 D - 0xc7907000 [em3 txq (cpuid 1)]
100044 D - 0xc7907080 [em3 rxq (cpuid 0)]
100041 D - 0xc7907600 [em2 txq (cpuid 1)]
100039 D - 0xc7907e00 [em2 rxq (cpuid 0)]
100036 D - 0xc7715300 [em1 txq (cpuid 1)]
100034 D - 0xc7715380 [em1 rxq (cpuid 0)]
100031 D - 0xc7715900 [em0 txq (cpuid 1)]
100029 D - 0xc7715980 [em0 rxq (cpuid 0)]
100026 D - 0xc75c8d00 [ffs_trim taskq]
100024 D - 0xc7712000 [kqueue taskq]
100023 D - 0xc7712080 [acpi_task_2]
100022 D - 0xc7712080 [acpi_task_1]
100021 D - 0xc7712080 [acpi_task_0]
100018 D - 0xc7712480
-
That dump is not complete. Forum post size is limited.
Attach a txt file or post on pastebin please
-
Hi Sorry I missed your post. It did it again today so I have the dump.