Repeated crashs randomly… What steps do I take to diagnose crash logs?
-
I haven't had problems with crashes for years, and over many version of PFsense.
However, lately I have had 3 crashes, at seemingly random times.
It's becoming a problem, so I will post the dump below, however I don't know what I should do with this information to diagnose the issue.
Crash report begins. Anonymous machine information:
i386
8.3-RELEASE-p16
FreeBSD 8.3-RELEASE-p16 #0: Mon Aug 25 08:25:41 EDT 2014 root@pf2_1_1_i386.pfsense.org:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.8Crash report details:
Filename: /var/crash/bounds
1Filename: /var/crash/info.0
Dump header from device /dev/da0s1b
Architecture: i386
Architecture Version: 1
Dump Length: 80384B (0 MB)
Blocksize: 512
Dumptime: Fri Jan 9 11:31:01 2015
Hostname: firewall.sds.local
Magic: FreeBSD Text Dump
Version String: FreeBSD 8.3-RELEASE-p16 #0: Mon Aug 25 08:25:41 EDT 2014
root@pf2_1_1_i386.pfsense.org:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.8
Panic String: double fault
Dump Parity: 607255553
Bounds: 0
Dump Status: goodFilename: /var/crash/textdump.tar.0
ddb.txt���������������������������������������������������������������������������������������������0600����0�������0�������140000������12454001105� 7062� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������db:0:kdb.enter.default> run lockinfo
db:1:lockinfo> show locks
No such command
db:1:locks> show alllocks
No such command
db:1:alllocks> show lockedvnods
Locked vnodes
db:0:kdb.enter.default> show pcpu
cpuid = 2
dynamic pcpu = 0x6067700
curthread = 0xc79945c0: pid 11 "idle: cpu2"
curpcb = 0xc75a6d80
fpcurthread = none
idlethread = 0xc79945c0: tid 100004 "idle: cpu2"
APIC ID = 6
currentldt = 0x50
db:0:kdb.enter.default> bt
Tracing pid 11 tid 100004 td 0xc79945c0
kdb_enter(c0f98d73,c0f98d73,c0fccc1e,c15d02f0,2,…) at kdb_enter+0x3b
panic(c0fccc1e,6,6,0,0,...) at panic+0x102
dblfault_handler() at dblfault_handler+0x9b
--- trap 0x17, eip = 0xc0e750d8, esp = 0xc75a6000, ebp = 0xc75a6c7c ---
Xpage(c75a6c98,c151f8b0,2,0,ffffffff,...) at Xpage
acpi_cpu_idle(1,c75a6ccc,c0f99f18,a08,c79945c0,...) at acpi_cpu_idle+0x1e1
sched_idletd(0,c75a6d28,0,0,0,...) at sched_idletd+0x116
fork_exit(c0ac8fb0,0,c75a6d28) at fork_exit+0x87
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc75a6d60, ebp = 0 ---
db:0:kdb.enter.default> ps
pid ppid pgrp uid state wmesg wchan cmd
34780 57442 28 0 RE CPU 3 sh
64014 62780 62780 0 S nanslp 0xc15571c4 sleep
62780 62064 62780 0 Ss wait 0xc97b8000 sh
62064 62712 62712 0 S piperd 0xc9c38930 cron
68699 1 68451 558 S select 0xc97671a4 avahi-daemon
68223 1 68223 556 Ss select 0xc8e124e4 dbus-daemon
51878 1 51878 0 Ss (threaded) ntpd
100228 S select 0xc94e3964 ntpd
43385 1 43385 0 Ss select 0xc7feb024 openvpn
28660 68109 68109 0 S (threaded) php
100167 S accept 0xc904d1da php
21915 62070 21915 0 Ss (threaded) sshlockout_pf
100293 S nanslp 0xc15571c4 sshlockout_pf
100113 S piperd 0xc8912620 initial thread
21384 1 21384 0 Ss+ ttyin 0xc7c81670 getty
15839 15767 15767 0 S nanslp 0xc15571c4 minicron
15767 1 15767 0 Ss wait 0xc95bb2b0 minicron
15511 15117 15117 0 S nanslp 0xc15571c4 minicron
15117 1 15117 0 Ss wait 0xc8917ac0 minicron
14983 14655 14655 0 S nanslp 0xc15571c4 minicron
14655 1 14655 0 Ss wait 0xc95ba810 minicron
62712 1 62712 0 Ss nanslp 0xc15571c4 cron
62070 1 62070 0 Ss select 0xc8e129e4 syslogd
59396 1 59396 0 Ss select 0xc9460824 powerd
57442 1 28 0 S+ wait 0xc9517810 sh
31758 28859 28859 0 S accept 0xc904eb82 php
31424 28859 28859 0 S accept 0xc904eb82 php
31136 28859 28859 0 S accept 0xc904eb82 php
30868 27621 27621 0 S accept 0xc945f03e php
30581 27621 27621 0 S accept 0xc945f03e php
30276 27621 27621 0 S accept 0xc945f03e php
29455 23730 23730 0 S accept 0xc91889e6 php
29383 23730 23730 0 S accept 0xc91889e6 php
29094 23730 23730 0 S accept 0xc91889e6 php
28859 19940 28859 0 Ss wait 0xc8915ac0 initial thread
27896 21757 21757 0 S accept 0xc9188512 php
27861 21757 21757 0 S accept 0xc9188512 php
27790 21757 21757 0 S accept 0xc9188512 php
27621 20453 27621 0 Ss wait 0xc80512b0 initial thread
24236 21299 21299 0 S accept 0xc9525512 php
24215 21299 21299 0 S accept 0xc9525512 php
23912 21299 21299 0 S accept 0xc9525512 php
23730 19940 23730 0 Ss wait 0xc8915000 initial thread
22346 22084 22084 0 S nanslp 0xc15571c4 minicron
22084 1 22084 0 Ss wait 0xc88652b0 minicron
21757 20453 21757 0 Ss wait 0xc889a560 initial thread
21299 19940 21299 0 Ss wait 0xc80272b0 initial thread
21003 20273 20273 0 S accept 0xc91a7376 php
20672 20273 20273 0 S accept 0xc91a7376 php
20627 20273 20273 0 S accept 0xc91a7376 php
20453 1 20332 0 S kqread 0xc9116680 lighttpd
20273 19940 20273 0 Ss wait 0xc8053810 initial thread
19940 1 19693 0 S kqread 0xc9438680 lighttpd
69913 1 69913 1002 Ss select 0xc7ca0864 dhcpd
69616 1 69529 65534 S select 0xc8e126a4 dnsmasq
68445 60206 60206 0 S accept 0xc894a03e php
68109 59862 68109 0 Ss wait 0xc8053560 initial thread
61605 1 61605 0 Ss kqread 0xc9c8f700 dhcpleases
60206 59862 60206 0 Ss wait 0xc8899560 initial thread
59862 1 59528 0 S kqread 0xc8cc6280 lighttpd
43526 43473 43473 0 S piperd 0xc889eab8 rrdtool
43473 1 43473 0 Ss select 0xc7fc6de4 apinger
38516 1 38516 0 Ss select 0xc891be24 inetd
34388 1 28 0 S+ piperd 0xc8050000 logger
34051 1 28 0 S+ bpf 0xc9051400 tcpdump
21790 1 21790 0 Ss (threaded) mpd5
100160 S select 0xc8e121a4 mpd5
18818 1 18818 65 Ss select 0xc7f9e324 dhclient
10474 1 10474 0 Ss select 0xc7f42924 dhclient
10266 1 10266 0 Ss (threaded) sshlockout_pf
100159 S nanslp 0xc15571c4 sshlockout_pf
100143 S uwait 0xc7fc5d40 initial thread
9984 1 9984 0 Ss select 0xc891b7a4 sshd
273 1 273 0 Ss select 0xc7fc6164 devd
264 262 262 0 S kqread 0xc79dd000 check_reload_status
262 1 262 0 Ss kqread 0xc7fa4200 check_reload_status
69 0 0 0 SL mdwait 0xc7f54000 [md0]
40 0 0 0 SL (threaded) zfskern
100093 D l2arc_fe 0xc81e1b04 [l2arc_feed_thread]
100092 D arc_recl 0xc81d297c [arc_reclaim_thread]
27 0 0 0 SL sdflush 0xc1585b60 [softdepflush]
26 0 0 0 SL vlruwt 0xc7f7b000 [vnlru]
25 0 0 0 SL syncer 0xc156af38 [syncer]
24 0 0 0 SL psleep 0xc156ac68 [bufdaemon]
23 0 0 0 SL pollid 0xc15566fc [idlepoll]
22 0 0 0 SL pgzero 0xc1586814 [pagezero]
21 0 0 0 SL psleep 0xc158643c [vmdaemon]
20 0 0 0 SL psleep 0xc1586404 [pagedaemon]
19 0 0 0 SL ccb_scan 0xc151f5d4 [xpt_thrd]
18 0 0 0 SL pftm 0xc050c700 [pfpurge]
9 0 0 0 SL waiting_ 0xc1572338 [sctp_iterator]
8 0 0 0 SL - 0xc7c8183c [fdc0]
17 0 0 0 SL cooling 0xc7c630d4 [acpi_cooling0]
16 0 0 0 SL tzpoll 0xc152462c [acpi_thermal]
15 0 0 0 SL (threaded) usb
100057 D - 0xc7c4dd34 [usbus2]
100056 D - 0xc7c4dd04 [usbus2]
100055 D - 0xc7c4dcd4 [usbus2]
100054 D - 0xc7c4dca4 [usbus2]
100052 D - 0xc7c32dac [usbus1]
100051 D - 0xc7c32d7c [usbus1]
100050 D - 0xc7c32d4c [usbus1]
100049 D - 0xc7c32d1c [usbus1]
100047 D - 0xc7c1fdac [usbus0]
100046 D - 0xc7c1fd7c [usbus0]
100045 D - 0xc7c1fd4c [usbus0]
100044 D - 0xc7c1fd1c [usbus0]
7 0 0 0 SL idle 0xc7c00084 [ciss_notify0]
14 0 0 0 SL - 0xc1557024 [yarrow]
6 0 0 0 SL crypto_r 0xc158510c [crypto returns]
5 0 0 0 SL crypto_w 0xc15850e8 [crypto]
4 0 0 0 SL - 0xc15547e4 [g_down]
3 0 0 0 SL - 0xc15547e0 [g_up]
2 0 0 0 SL - 0xc15547d8 [g_event]
13 0 0 0 SL (threaded) ng_queue
100016 D sleep 0xc14f2fa0 [ng_queue3]
100015 D sleep 0xc14f2fa0 [ng_queue2]
100014 D sleep 0xc14f2fa0 [ng_queue1]
100013 D sleep 0xc14f2fa0 [ng_queue0]
12 0 0 0 WL (threaded) intr
100079 I [swi1: netisr 1]
100078 I [swi1: netisr 2]
100077 I [swi1: netisr 3]
100064 I [swi0: uart uart]
100063 I [irq12: psm0]
100062 I [irq1: atkbd0]
100059 I [irq15: ata1]
100058 I [irq14: ata0]
100053 I [irq23: ehci0]
100048 I [irq19: uhci1]
100043 I [irq16: uhci0]
100042 I [irq26: bge1]
100041 I [irq25: bge0]
100039 I [irq24: ciss0]
100034 I [irq9: acpi0]
100033 I [swi6: Giant taskq]
100031 I [swi5: +]
100028 I [swi2: cambio]
100024 I [swi6: task queue]
100012 I [swi3: vm]
100011 I [swi4: clock]
100010 I [swi4: clock]
100009 I [swi4: clock]
100008 I [swi4: clock]
100007 I [swi1: netisr 0]
11 0 0 0 RL (threaded) idle
100006 Run CPU 0 [idle: cpu0]
100005 Run CPU 1 [idle: cpu1]
100004 Run CPU 2 [idle: cpu2]
100003 CanRun [idle: cpu3]
1 0 1 0 SLs wait 0xc7992ac0 [init]
10 0 0 0 SL audit_wo 0xc1585500 [audit]
0 0 0 0 RLs (threaded) kernel
100180 RunQ [dummynet]
100091 D - 0xc7f42280 [system_taskq_3]
100090 D - 0xc7f42280 [system_taskq_2]
100089 D - 0xc7f42280 [system_taskq_1]
100088 D - 0xc7f42280 [system_taskq_0]
100038 D - 0xc7be2700 [em3 taskq]
100037 D - 0xc7bd3b40 [em2 taskq]
100036 D - 0xc7bbf800 [em1 taskq]
100035 D - 0xc7bb5d40 [em0 taskq]
100032 D - 0xc7b68d00
-
What did you end up doing to diagnose this?
I'm experiencing similar crashes roughly hourly:
$ grep 'Starting all' serial_console.log Aug 8 22:51:03 php-fpm[5457]: /rc.start_packages: Restarting/Starting all packages. Aug 8 23:07:28 php-fpm[6437]: /rc.start_packages: Restarting/Starting all packages. Aug 9 00:28:44 php-fpm[78164]: /rc.start_packages: Restarting/Starting all packages. Aug 9 01:33:37 php-fpm[6001]: /rc.start_packages: Restarting/Starting all packages. Aug 9 01:49:05 php-fpm[75297]: /rc.start_packages: Restarting/Starting all packages. $and I'm about to post a question about it, but since yours happened 8 months ago (and no one responded, but I don't see anything in your post history about what you did to resolve it), I thought I'd ask if you recall what you did to investigate it (I'm very familiar with debian and very unfamiliar with bsd).
Thanks
-
Post your question as a new post.
I never got to the bottom of this and it seems to have stopped on it's own.However, I have also upgraded twice since this.
-
I never got to the bottom of this and it seems to have stopped on it's own.
However, I have also upgraded twice since this.
This is almost always a confirmation that hardware was going bad; glad you found it.
-
No hardware issues at all here.
I never got to the bottom of it. It remains a mystery.
-
Any use?
https://forum.pfsense.org/index.php?topic=97347.msg542182#msg542182
https://forum.pfsense.org/index.php?topic=95928.msg542034#msg542034
Have you got any syslogs or traffic logs to see what was going on before the crashes, as theres so many possibilities for "random" crashes.
-
I appreciate the links, but I really am not working this any longer.
This thread was several months old when someone revived it.