pfSense Crash diagnose
-
Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 06
fault virtual address = 0x28
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80ec01fe
stack pointer = 0x28:0xfffffe00747f2950
frame pointer = 0x28:0xfffffe00747f2980
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 12 (swi4: clock (0))
trap number = 12
panic: page fault
cpuid = 3
time = 1629423140
KDB: enter: panic
panic.txt0600001214107603044 7130 ustarrootwheelpage faultversion.txt0600006714107603044 7535 ustarrootwheelFreeBSD 12.2-STABLE fd0f54f44b5c(RELENG_2_5_0) pfSense -
Can we see the full crash report? We need to see at least the backtrace. The console output leading up to it is usually helpful too.
Steve
-
@stephenw10 thank you for your willingness to help!
I have these filesdrwxr-xr-x 2 aschmi users 4.0K Aug 20 09:09 .
drwxr-xr-x 8 aschmi users 12K Aug 19 19:42 ..
-rw------- 1 aschmi users 6.2K Aug 19 18:47 config.txt
-rw------- 1 aschmi users 48K Aug 19 18:47 ddb.txt
-rw------- 1 aschmi users 83K Aug 19 18:47 msgbuf.txt
-rw------- 1 aschmi users 10 Aug 19 18:47 panic.txt
-rw-r--r-- 1 aschmi users 141K Aug 20 09:08 textdump.tar
-rw------- 1 aschmi users 55 Aug 19 18:47 version.txtWich one can it help? How do I upload them here?
Thanks!!! -
The msgbuf and ddb files are where I would normally check. That is normally in the compressed crash file though which you should be able to upload.
It does contain a lot of details from you network though, WAN IP address etc.Steve
-
ddb.txt msgbuf.txt
Thank you so much! -
OK, so the important part of that is this:
db:0:kdb.enter.default> show pcpu cpuid = 2 dynamic pcpu = 0xfffffe0080d71380 curthread = 0xfffff80005740000: pid 12 tid 100035 "swi4: clock (0)" curpcb = 0xfffff800057405a0 fpcurthread = none idlethread = 0xfffff80005622000: tid 100005 "idle: cpu2" curpmap = 0xffffffff8368d5a8 tssp = 0xffffffff837176f0 commontssp = 0xffffffff837176f0 rsp0 = 0xfffffe00747f2cc0 kcr3 = 0x80000000040cb002 ucr3 = 0xffffffffffffffff scr3 = 0x10b690d52 gs32p = 0xffffffff8371df08 ldt = 0xffffffff8371df48 tss = 0xffffffff8371df38 tlb gen = 130953 curvnet = 0 db:0:kdb.enter.default> bt Tracing pid 12 tid 100035 td 0xfffff80005740000 kdb_enter() at kdb_enter+0x37/frame 0xfffffe00747f2610 vpanic() at vpanic+0x197/frame 0xfffffe00747f2660 panic() at panic+0x43/frame 0xfffffe00747f26c0 trap_fatal() at trap_fatal+0x391/frame 0xfffffe00747f2720 trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00747f2770 trap() at trap+0x286/frame 0xfffffe00747f2880 calltrap() at calltrap+0x8/frame 0xfffffe00747f2880 --- trap 0xc, rip = 0xffffffff80ec01fe, rsp = 0xfffffe00747f2950, rbp = 0xfffffe00747f2980 --- ether_8021q_frame() at ether_8021q_frame+0x2e/frame 0xfffffe00747f2980 vlan_transmit() at vlan_transmit+0xc8/frame 0xfffffe00747f29f0 vlan_altq_start() at vlan_altq_start+0xb4/frame 0xfffffe00747f2a20 cbqrestart() at cbqrestart+0x64/frame 0xfffffe00747f2a50 rmc_restart() at rmc_restart+0x6f/frame 0xfffffe00747f2a80 softclock_call_cc() at softclock_call_cc+0x141/frame 0xfffffe00747f2b30 softclock() at softclock+0x79/frame 0xfffffe00747f2b50 ithread_loop() at ithread_loop+0x23c/frame 0xfffffe00747f2bb0 fork_exit() at fork_exit+0x7e/frame 0xfffffe00747f2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00747f2bf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 ---It looks like on that last boot where us then crashed you has added a new VLAN interface:
vlan3: changing name to 'igb1.2'
Is there traffic shaping on that VLAN? Using CBQ?
Is it different to the other VLANs?This looks pretty much exactly like this:
https://forum.netgate.com/topic/165827/random-crash
That was resolved removing the shaping from the VLAN.Steve
-
Yeah, looks to be this: https://redmine.pfsense.org/issues/11470
-
@stephenw10 Hi yes I do have CBQ traffic, in all the VLAN interfaces. The vlan3 is for "guest" so has a lower speed than the others.
This seems to be crashing when I'm heavily using that VLAN (on some guest machines that I was running Windows updates on)
Should I remove completely the traffic shapping, or is the problem with just CBQ?
Good eye I can still not see how you discovered the issue with CBQ and VLANS but great catch!
Adrian -
The last lines in the backtrace before it panics point to it, read in reverse order:
ether_8021q_frame() at ether_8021q_frame+0x2e/frame 0xfffffe00747f2980 vlan_transmit() at vlan_transmit+0xc8/frame 0xfffffe00747f29f0 vlan_altq_start() at vlan_altq_start+0xb4/frame 0xfffffe00747f2a20 cbqrestart() at cbqrestart+0x64/frame 0xfffffe00747f2a50That bug report confirms it.
Can you share any details of how you have it configured?
I'm trying to replicate it here but failing. It's stubbornly not crashing!Steve
-
@stephenw10 Hi Stephen how do I send you the config file in a safe way? What do I need to sanitize? I do have radius on my server, should I remove that part of the xml? And what about the admin password is that stored on the xml backup?
I have one WAN interface, and one intel with 4 nics for the internal part, and is an old Dell desktop (those mini ones) that I can give you more specs, 16Gb of RAM. I had a ssd that failed before, but now it was working with a normal drive (but again failing)
The interesting thing is that this was only failing when that network that was being shaped was under heavy usage (probably reaching the limit of CBQ?)
Thanks -
We probably don't need the full config but I can PM you something.
Just the details of VLAN in question and the shaper queues that were defined on it.
Steve
-
@stephenw10 Sorry I was sure I sent you the config! Just to let you know that I reenabled QOS on that interface, and had a crash again... here is the config (what I think should be relevant)
(attached tried to post here but got rejected by anti spam)config.xmlThank you so much for your time!!! And let me know If I need to send you anything else
-
igb1.2 is not in that config. Which interface did you enable it on to cause the crash?
Using the same queues that are on the other interfaces?
Steve
-
@stephenw10 Sorry I tried to copy/paste but didn't work too well. Here is my actual config
I have QOS disabled now so it doesn't crash
Thanks again!
-
I was able to get the config OK it's just that it doesn't have the VLAN igb1.2 in it and it looks like that is what caused the crash.
Can I assume you had that configured the same as the other VLANs? Same traffic shaping queues on it?Steve
-
@stephenw10 Good you noticed and yes I was using the VLAN2 on igb1 (second interface on a 4 port 1Gbps interface) but not on my normal interfaces
I can send you the whole .XML if you tell me how to sanitize not to send usernames/passwords if that will help
-
What I have should be enough if the queues were the same.
However you can get the sanitised config as part of the status file by going directly to:
[your firewall IP]/status.phpNothing special about igb1? igb0 is also a port on the same card?
Steve
-
@stephenw10 status_output.tgz
There you have it. Yes igbo, 1, 2 and 3 is just one card (supposedly intel but bought on ebay)
I was trying to use igb1.2 as a bridge with igb0.2, but crashes happened before I started playing with that configuration -
OK cool, so you had not added the bridge when it crashed?
-
@stephenw10 No I didn't have it in the initial crashes (the bridge). Well almost sure. This is my home so this is the only place where I don't document changes (oops) -
When it was failing badly (I guess more when I was trying to add a bridge to do some port mirroring as well), I did a change of hardware.
Then it seemed okay, but just for a while.
The only consistent result (and no crashes) I have is when I have shaping completely off (with the old or new hardware)Should I remove the VLAN on igb1 and try again to add shaping to see what happens?
