How to read / analyse a pfSense crashdump

louis2

If pfSense crashes, it does produce a crash textdump containing a lot of information.

The big question is how to read / how to understand that file to get the most out of it. I tried to figure out. Hope that it is correct.

I assume you have to distinguish two situations:

if you have a full development environment or
you do not have that and can just read the text (my situation)

I do not have a development environment, so I can just read the text.

So I was looking for a document describing that, could not find it. Is there such a document !!??

I have attached a sample crash, but let me try to explain some elements out of the textdump the way I interpret them.

curthread = 0xfffff8000439f000: pid 12 tid 100040 "swi1: netisr 2"
I assume that means:
the moment of the crash actual thread running as 12 was executing "swi1: netisr 2"

Tracing pid 12 tid 100040 td 0xfffff8000439f000
I assume the lines below are a backtrace of what the given thread did before it crashed
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00004de4b0
vpanic() at vpanic+0x197/frame 0xfffffe00004de500
panic() at panic+0x43/frame 0xfffffe00004de560
trap_fatal() at trap_fatal+0x391/frame 0xfffffe00004de5c0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00004de610
trap() at trap+0x286/frame 0xfffffe00004de720
calltrap() at calltrap+0x8/frame 0xfffffe00004de720
--- trap 0xc, rip = 0xffffffff80e934f5, rsp = 0xfffffe00004de7f0, rbp = 0xfffffe00004de7f0 ---
if_inc_counter() at if_inc_counter+0x15/frame 0xfffffe00004de7f0
if_simloop() at if_simloop+0xd1/frame 0xfffffe00004de830
pim_input() at pim_input+0x409/frame 0xfffffe00004de890
encap_input() at encap_input+0xd1/frame 0xfffffe00004de900
encap4_input() at encap4_input+0x28/frame 0xfffffe00004de930
ip_input() at ip_input+0x168/frame 0xfffffe00004de9e0
swi_net() at swi_net+0x12b/frame 0xfffffe00004dea50
ithread_loop() at ithread_loop+0x23c/frame 0xfffffe00004deab0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00004deaf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00004deaf0

Within the lines above I marked one line starting with pim_input()
What I assume is that that is the startpoint of the trouble. PIMD is asking for some information from the underlying FreeBSD-layers

A bit above that line I notice that an trap is called "--- trap 0xc, rip"

And I think that is "the start of the crash". That trap is not handled correctly and/or that trap is not called with proper parameters

Then there are

a lot of lines, what seems "an advanced memory dump" to me.
followed by backtraces from other processes running at the time of the crash.
followed by information about os-settings and interfaces
followed by detailed configuration info
followed by boot log
followed by pfsense startup log
followed by a summary of the crash

I think that is it.

As said in the beginning, I did try to understand the log. I could not find formal documentation and I do not guarantee that it is correct or complete.

So please correct me or add useful comments, links with more info etc.

Louis

textdump.tar(20).0