Pfsense is crashing, need assistance interpreting the crash dump

jbrainard

Once or twice a day, the router crashes and reboots itself. It appears to be entering a kernel panic, but I don't know what the issue might be.

textdump.tar.0

jbrainard

bump

stephenw10

The backtrace there isn't very revealing:

db:0:kdb.enter.default>  bt
Tracing pid 86487 tid 100244 td 0xfffff8002a9ca000
kdb_enter() at kdb_enter+0x37/frame 0xfffffe002eb56210
vpanic() at vpanic+0x197/frame 0xfffffe002eb56260
panic() at panic+0x43/frame 0xfffffe002eb562c0
trap_fatal() at trap_fatal+0x391/frame 0xfffffe002eb56320
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe002eb56370
trap() at trap+0x286/frame 0xfffffe002eb56480
calltrap() at calltrap+0x8/frame 0xfffffe002eb56480
--- trap 0xc, rip = 0xffffffff811e8289, rsp = 0xfffffe002eb56550, rbp = 0xfffffe002eb56560 ---
vm_map_process_deferred() at vm_map_process_deferred+0x49/frame 0xfffffe002eb56560
vm_map_remove() at vm_map_remove+0x119/frame 0xfffffe002eb56590
exec_new_vmspace() at exec_new_vmspace+0x1de/frame 0xfffffe002eb56600
exec_elf64_imgact() at exec_elf64_imgact+0xa8c/frame 0xfffffe002eb566f0
kern_execve() at kern_execve+0x728/frame 0xfffffe002eb56a40
sys_execve() at sys_execve+0x51/frame 0xfffffe002eb56ac0
amd64_syscall() at amd64_syscall+0x387/frame 0xfffffe002eb56bf0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe002eb56bf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x8003d114a, rsp = 0x7fffffffe1f8, rbp = 0x7fffffffe340 ---

Do you have any custom shell scripts on there?

jbrainard

@stephenw10

No sir, nothing crazy. Pretty standard configuration. If it helps, the crashes occur at almost exactly 8 AM it seems. Here's a crash dump from this morning:

textdump.tar (7).0

stephenw10

Hmm, backtrace is similar but not identical:

db:0:kdb.enter.default>  bt
Tracing pid 96484 tid 100231 td 0xfffff8006f46e740
kdb_enter() at kdb_enter+0x37/frame 0xfffffe002eb0b630
vpanic() at vpanic+0x197/frame 0xfffffe002eb0b680
panic() at panic+0x43/frame 0xfffffe002eb0b6e0
trap_fatal() at trap_fatal+0x391/frame 0xfffffe002eb0b740
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe002eb0b790
trap() at trap+0x286/frame 0xfffffe002eb0b8a0
calltrap() at calltrap+0x8/frame 0xfffffe002eb0b8a0
--- trap 0xc, rip = 0xffffffff811eef8e, rsp = 0xfffffe002eb0b970, rbp = 0xfffffe002eb0b9c0 ---
vmspace_fork() at vmspace_fork+0x95e/frame 0xfffffe002eb0b9c0
fork1() at fork1+0x356/frame 0xfffffe002eb0ba60
sys_fork() at sys_fork+0x54/frame 0xfffffe002eb0bac0
amd64_syscall() at amd64_syscall+0x387/frame 0xfffffe002eb0bbf0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe002eb0bbf0
--- syscall (2, FreeBSD ELF64, sys_fork), rip = 0x8003ed8ea, rsp = 0x7fffffffe558, rbp = 0x7fffffffe590 ---

But we see some errors in the message buffer:

<6>pid 75760 (unbound), jid 0, uid 59: exited on signal 11
<6>pid 4613 (awk), jid 0, uid 0: exited on signal 6 (core dumped)

I would guess that is pfBlocker updating except I don't see that running. In which case do you have a large number of custom Unbound values? Host overrides?

Check the crontab for processes running at 8.00. Installing the cron package will show that.

Steve