Crash reporter
-
Hi everyone,
since a few days i often get one of these message on my dashboard:
Unfortunately we have detected a programming bug.
Would you like to submit the programming debug logs to the pfSense developers for inspection?
Please double check the contents to ensure you are comfortable sending this information before clicking Yes.
It seems the whole pfSense is crashing and rebooting after this. While upgrading to the latest version i started with a clean-configuration.
Content of the crash-report:
Crash report begins. Anonymous machine information:
i386
10.1-RELEASE-p15
FreeBSD 10.1-RELEASE-p15 #0 c5ab052(releng/10.1)-dirty: Sat Jul 25 20:25:50 CDT 2015 root@pfs22-i386-builder:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.10Crash report details:
Filename: /var/crash/bounds
2Filename: /var/crash/info.0
Dump header from device /dev/label/swap0
Architecture: i386
Architecture Version: 1
Dump Length: 90112B (0 MB)
Blocksize: 512
Dumptime: Thu Sep 17 22:59:46 2015
Hostname: pfs01.motecscloud.motecs.com
Magic: FreeBSD Text Dump
Version String: FreeBSD 10.1-RELEASE-p15 #0 c5ab052(releng/10.1)-dirty: Sat Jul 25 20:25:50 CDT 2015
root@pfs22-i386-builder:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.10
Panic String:
Dump Parity: 3458590740
Bounds: 0
Dump Status: goodFilename: /var/crash/info.1
Dump header from device /dev/label/swap0
Architecture: i386
Architecture Version: 1
Dump Length: 107520B (0 MB)
Blocksize: 512
Dumptime: Mon Sep 21 03:53:28 2015
Hostname: pfs01.motecscloud.motecs.com
Magic: FreeBSD Text Dump
Version String: FreeBSD 10.1-RELEASE-p15 #0 c5ab052(releng/10.1)-dirty: Sat Jul 25 20:25:50 CDT 2015
root@pfs22-i386-builder:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.10
Panic String:
Dump Parity: 346811412
Bounds: 1
Dump Status: goodFilename: /var/crash/info.last
Dump header from device /dev/label/swap0
Architecture: i386
Architecture Version: 1
Dump Length: 107520B (0 MB)
Blocksize: 512
Dumptime: Mon Sep 21 03:53:28 2015
Hostname: pfs01.motecscloud.motecs.com
Magic: FreeBSD Text Dump
Version String: FreeBSD 10.1-RELEASE-p15 #0 c5ab052(releng/10.1)-dirty: Sat Jul 25 20:25:50 CDT 2015
root@pfs22-i386-builder:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.10
Panic String:
Dump Parity: 346811412
Bounds: 1
Dump Status: goodFilename: /var/crash/textdump.tar.0
ddb.txt06000014000012576624702 7105 ustarrootwheeldb:0:kdb.enter.default> run lockinfo
db:1:lockinfo> show locks
No such command
db:1:locks> show alllocks
No such command
db:1:alllocks> show lockedvnods
Locked vnodes
db:0:kdb.enter.default> show pcpu
cpuid = 0
dynamic pcpu = 0x5bf480
curthread = 0xc7024c40: pid 64796 "syslogd"
curpcb = 0xef2fad60
fpcurthread = none
idlethread = 0xc6700000: tid 100003 "idle: cpu0"
APIC ID = 0
currentldt = 0x50
db:0:kdb.enter.default> bt
Tracing pid 64796 tid 100080 td 0xc7024c40
key_delsp(c792c620,2,c147b09c,16e,c6a07194,…) at key_delsp+0x27/frame 0xef2fa900
_key_freesp(ef2fa94c,c147b09c,1e2,0,0,...) at key_freesp+0xaf/frame 0xef2fa930
ipsec4_checkpolicy(c7e4d600,2,0,ef2faa2c,c6f697e0,...) at ipsec4_checkpolicy+0x7e/frame 0xef2fa970
ip_ipsec_output(ef2faa34,c6f697e0,ef2faa30,ef2faa2c,c7e4d600,...) at ip_ipsec_output+0xa0/frame 0xef2fa9a0
ip_output(c7e4d600,0,0,0,0,...) at ip_output+0xa2f/frame 0xef2faa58
udp_send(c6f334f8,0,c7e4d600,c6dd8e90,0,...) at udp_send+0xb2d/frame 0xef2fab08
sosend_dgram(c6f334f8,c6dd8e90,ef2fab98,0,0,...) at sosend_dgram+0x486/frame 0xef2fab50
kern_sendit(c7024c40,e,ef2fac20,0,0,...) at kern_sendit+0x256/frame 0xef2fabcc
sendit(ef2fac20,0,bfbfcfbf,0,c6dd8e90,...) at sendit+0x12d/frame 0xef2fac08
sys_sendto(c7024c40,ef2facc8,c7024c40,ef2fac60,46,...) at sys_sendto+0x5b/frame 0xef2fac40
syscall(ef2fad08) at syscall+0x48b/frame 0xef2facfc
Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xef2facfc
--- syscall (133, FreeBSD ELF32, sys_sendto), eip = 0x281aac93, esp = 0xbfbfcd9c, ebp = 0xbfbfd44c ---
db:0:kdb.enter.default> ps
pid ppid pgrp uid state wmesg wchan cmd
61383 33657 241 0 R sh
36209 241 241 0 S accept 0xc6f34536 php-fpm
33657 241 241 0 S piperd 0xc6e7db28 php-fpm
32979 50386 22 0 S nanslp 0xc1f25bc8 sleep
94124 64796 94124 0 Ss (threaded) sshlockout_pf
100248 S nanslp 0xc1f25bc8 sshlockout_pf
100073 S piperd 0xc6e7d660 sshlockout_pf
5447 1 5447 0 Ss (threaded) ntpd
100091 S select 0xc6f53da4 ntpd
99734 1 99734 0 Ss (threaded) filterdns
100170 S uwait 0xc7929600 signal-thread
*
100131 S uwait 0xc77c0080 x.motecs.com
88892 1 88709 0 S select 0xc6a06a24 vmtoolsd
81763 1 81763 0 Ss+ ttyin 0xc69e7870 getty
78660 78153 78153 0 S nanslp 0xc1f25bc8 minicron
78153 1 78153 0 Ss wait 0xc793d5e0 minicron
78001 77681 77681 0 S nanslp 0xc1f25bc8 minicron
77681 1 77681 0 Ss wait 0xc78092f0 minicron
77607 77304 77304 0 S nanslp 0xc1f25bc8 minicron
77304 1 77304 0 Ss wait 0xc7807bc0 minicron
75964 1 75964 0 Ss nanslp 0xc1f25bc8 cron
64796 1 64796 0 Rs CPU 0 syslogd
50386 1 22 0 S+ wait 0xc7808000 sh
44595 1 44595 0 Ss (threaded) filterdns
100260 S uwait 0xc7836c00 filterdns
100101 S uwait 0xc6e97280 signal-thread
100100 S uwait 0xc77c1780 voip.spacecamper.de
100099 S uwait 0xc77c1700 filterdns
100098 S uwait 0xc77c1680 x.acsde.de
100097 S uwait 0xc77c1600 filterdns
100096 S uwait 0xc77c1580 filterdns
44513 44170 44513 0 Rs (threaded) charon
100338 S uwait 0xc7836c80 charon
100117 S uwait 0xc7728300 charon
100116 S uwait 0xc7838780 charon
100115 RunQ charon
100114 S uwait 0xc77c0e00 charon
100113 S uwait 0xc77c0180 charon
100112 S uwait 0xc77c0b00 charon
100111 S uwait 0xc77c0a80 charon
100110 S uwait 0xc7727b80 charon
100109 S uwait 0xc7728200 charon
100108 S uwait 0xc7727d80 charon
100107 RunQ charon
100106 S uwait 0xc77c0680 charon
100105 S select 0xc6f53364 charon
100104 S umtxn 0xc6e97380 charon
100103 S accept 0xc6f35536 charon
100102 S uwait 0xc6e26100 charon
100095 S sigwait 0xc782a000 charon
44170 1 44170 0 Ss sbwait 0xc77f0758 starter
26739 1 26739 59 Ss kqread 0xc77c1900 unbound
24543 1 24400 0 S kqread 0xc77c2000 lighttpd
20508 20193 20193 0 S piperd 0xc6e7f000 rrdtool
20193 1 20193 0 Ss select 0xc6f53f64 apinger
19530 1 19530 0 Ss (threaded) sshlockout_pf
100085 S nanslp 0xc1f25bc8 sshlockout_pf
100079 S uwait 0xc6e97480 sshlockout_pf
19378 1 19378 0 Ss select 0xc6f541a4 sshd
16389 1 16389 0 Ss select 0xc6f543a4 inetd
15727 1 15727 0 Ss bpf 0xc6dd4c00 filterlog
269 1 269 0 Ss select 0xc6f561a4 devd
261 257 257 0 S kqread 0xc6e97200 check_reload_status
257 1 257 0 Ss kqread 0xc6e97100 check_reload_status
241 1 241 0 Ss kqread 0xc6e97b00 php-fpm
58 0 0 0 DL mdwait 0xc6e06800 [md0]
21 0 0 0 DL syncer 0xc1f45144 [syncer]
20 0 0 0 DL vlruwt 0xc6dd0bc0 [vnlru]
19 0 0 0 DL psleep 0xc1f44884 [bufdaemon]
18 0 0 0 DL pollid 0xc1f24588 [idlepoll]
17 0 0 0 DL pgzero 0xc20703a0 [pagezero]
16 0 0 0 DL psleep 0xc20700c4 [vmdaemon]
9 0 0 0 DL psleep 0xc20cf904 [pagedaemon]
8 0 0 0 DL waiting 0xc20ca20c [sctp_iterator]
7 0 0 0 DL pftm 0xc0f85580 [pf purge]
6 0 0 0 DL - 0xc69e7a3c [fdc0]
5 0 0 0 DL idle 0xc67fb000 [mpt_recovery0]
4 0 0 0 DL (threaded) [cam]
100059 D - 0xc1e66a28 [scanner]
100021 D - 0xc1e66b80 [doneq0]
3 0 0 0 DL crypto_r 0xc206eff0 [crypto returns]
2 0 0 0 DL crypto_w 0xc206ef30 [crypto]
15 0 0 0 DL - 0xc1e80cd8 [rand_harvestq]
14 0 0 0 DL (threaded) [geom]
100010 D - 0xc20c6268 [g_down]
100009 D - 0xc20c6264 [g_up]
100008 D - 0xc20c6260 [g_event]
13 0 0 0 DL sleep 0xc1e3d418 [ng_queue0]
12 0 0 0 WL (threaded) [intr]
100057 I [swi1: pfsync]
100055 I [swi1: pf send]
100052 I [swi0: uart uart]
100051 I [irq7: ppc0]
100050 I [irq12: psm0]
100049 I [irq1: atkbd0]
100047 I [irq265: vmx9]
100045 I [irq264: vmx8]
100043 I [irq263: vmx7]
100041 I [irq262: vmx6]
100039 I [irq261: vmx5]
100037 I [irq260: vmx4]
100035 I [irq259: vmx3]
100033 I [irq258: vmx2]
100031 I [irq257: vmx1]
100029 I [irq256: vmx0]
100027 I [irq17: mpt0]
100026 I [irq15: ata1]
100025 I [irq14: ata0]
100024 I [swi6: Giant taskq]
100022 I [swi6: task queue]
100016 I [swi5: fast taskq]
100006 I [swi3: vm]
100005 I [swi4: clock]
100004 I [swi1: netisr 0]
11 0 0 0 RL [idle: cpu0]
1 0 1 0 SLs wait 0xc66f92f0 [init]
10 0 0 0 DL audit_wo 0xc20ce208 [audit]
0 0 0 0 DLs (threaded) [kernel]
100058 D - 0xc66fc280 [CAM taskq]
100048 D - 0xc6975600 [vmx9 taskq]
100046 D - 0xc6976080 [vmx8 taskq]
100044 D - 0xc6976880 [vmx7 taskq]
100042 D - 0xc6977400 [vmx6 taskq]
100040 D - 0xc68b3280 [vmx5 taskq]
100038 D - 0xc68b3a00 [vmx4 taskq]
100036 D - 0xc68d5200 [vmx3 taskq]
100034 D - 0xc68d5c00 [vmx2 taskq]
100032 D - 0xc68d6400 [vmx1 taskq]
100030 D - 0xc68d6b80 [vmx0 taskq]
100023 D - 0xc65dee80 [ffs_trim taskq]
100020 D - 0xc66fc300 [acpi_task_2]
100019 D - 0xc66fc300 [acpi_task_1]
100018 D - 0xc66fc300 [acpi_task_0]
100017 D - 0xc66fc480 [kqueue taskq]
100015 D - 0xc66fc600 -
Judging by the backtrace it's crashing in IPsec somehow
-
Could you describe what IPsec configuration you have on the system?
