pfSense Crash diagnose

aschmi

Hi this is Adrian, and I installed many pfSense box (netgate)
Now I have a little problem at home and is driving my nuts. My pfSense (free version) started to randomly reboot. I already changed the disk because previous one (solid state) was giving errors, but now even with that, it does randomly reboots.
Here is my crashdump
Help please!
Thank you so much in advance

aschmi

Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 06
fault virtual address = 0x28
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80ec01fe
stack pointer = 0x28:0xfffffe00747f2950
frame pointer = 0x28:0xfffffe00747f2980
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 12 (swi4: clock (0))
trap number = 12
panic: page fault
cpuid = 3
time = 1629423140
KDB: enter: panic
panic.txt0600001214107603044 7130 ustarrootwheelpage faultversion.txt0600006714107603044 7535 ustarrootwheelFreeBSD 12.2-STABLE fd0f54f44b5c(RELENG_2_5_0) pfSense

stephenw10

Can we see the full crash report? We need to see at least the backtrace. The console output leading up to it is usually helpful too.

Steve

aschmi

@stephenw10 thank you for your willingness to help!
I have these files

drwxr-xr-x 2 aschmi users 4.0K Aug 20 09:09 .
drwxr-xr-x 8 aschmi users 12K Aug 19 19:42 ..
-rw------- 1 aschmi users 6.2K Aug 19 18:47 config.txt
-rw------- 1 aschmi users 48K Aug 19 18:47 ddb.txt
-rw------- 1 aschmi users 83K Aug 19 18:47 msgbuf.txt
-rw------- 1 aschmi users 10 Aug 19 18:47 panic.txt
-rw-r--r-- 1 aschmi users 141K Aug 20 09:08 textdump.tar
-rw------- 1 aschmi users 55 Aug 19 18:47 version.txt

Wich one can it help? How do I upload them here?
Thanks!!!

stephenw10

The msgbuf and ddb files are where I would normally check. That is normally in the compressed crash file though which you should be able to upload.
It does contain a lot of details from you network though, WAN IP address etc.

Steve

aschmi

ddb.txt msgbuf.txt
Thank you so much!

stephenw10

OK, so the important part of that is this:

db:0:kdb.enter.default>  show pcpu
cpuid        = 2
dynamic pcpu = 0xfffffe0080d71380
curthread    = 0xfffff80005740000: pid 12 tid 100035 "swi4: clock (0)"
curpcb       = 0xfffff800057405a0
fpcurthread  = none
idlethread   = 0xfffff80005622000: tid 100005 "idle: cpu2"
curpmap      = 0xffffffff8368d5a8
tssp         = 0xffffffff837176f0
commontssp   = 0xffffffff837176f0
rsp0         = 0xfffffe00747f2cc0
kcr3         = 0x80000000040cb002
ucr3         = 0xffffffffffffffff
scr3         = 0x10b690d52
gs32p        = 0xffffffff8371df08
ldt          = 0xffffffff8371df48
tss          = 0xffffffff8371df38
tlb gen      = 130953
curvnet      = 0
db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100035 td 0xfffff80005740000
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00747f2610
vpanic() at vpanic+0x197/frame 0xfffffe00747f2660
panic() at panic+0x43/frame 0xfffffe00747f26c0
trap_fatal() at trap_fatal+0x391/frame 0xfffffe00747f2720
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00747f2770
trap() at trap+0x286/frame 0xfffffe00747f2880
calltrap() at calltrap+0x8/frame 0xfffffe00747f2880
--- trap 0xc, rip = 0xffffffff80ec01fe, rsp = 0xfffffe00747f2950, rbp = 0xfffffe00747f2980 ---
ether_8021q_frame() at ether_8021q_frame+0x2e/frame 0xfffffe00747f2980
vlan_transmit() at vlan_transmit+0xc8/frame 0xfffffe00747f29f0
vlan_altq_start() at vlan_altq_start+0xb4/frame 0xfffffe00747f2a20
cbqrestart() at cbqrestart+0x64/frame 0xfffffe00747f2a50
rmc_restart() at rmc_restart+0x6f/frame 0xfffffe00747f2a80
softclock_call_cc() at softclock_call_cc+0x141/frame 0xfffffe00747f2b30
softclock() at softclock+0x79/frame 0xfffffe00747f2b50
ithread_loop() at ithread_loop+0x23c/frame 0xfffffe00747f2bb0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00747f2bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00747f2bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

It looks like on that last boot where us then crashed you has added a new VLAN interface:
vlan3: changing name to 'igb1.2'
Is there traffic shaping on that VLAN? Using CBQ?
Is it different to the other VLANs?

This looks pretty much exactly like this:
https://forum.netgate.com/topic/165827/random-crash
That was resolved removing the shaping from the VLAN.

Steve

stephenw10

Yeah, looks to be this: https://redmine.pfsense.org/issues/11470

aschmi

@stephenw10 Hi yes I do have CBQ traffic, in all the VLAN interfaces. The vlan3 is for "guest" so has a lower speed than the others.
This seems to be crashing when I'm heavily using that VLAN (on some guest machines that I was running Windows updates on)
Should I remove completely the traffic shapping, or is the problem with just CBQ?
Good eye I can still not see how you discovered the issue with CBQ and VLANS but great catch!
Adrian

stephenw10

The last lines in the backtrace before it panics point to it, read in reverse order:

ether_8021q_frame() at ether_8021q_frame+0x2e/frame 0xfffffe00747f2980
vlan_transmit() at vlan_transmit+0xc8/frame 0xfffffe00747f29f0
vlan_altq_start() at vlan_altq_start+0xb4/frame 0xfffffe00747f2a20
cbqrestart() at cbqrestart+0x64/frame 0xfffffe00747f2a50

That bug report confirms it.

Can you share any details of how you have it configured?
I'm trying to replicate it here but failing. It's stubbornly not crashing!

Steve

aschmi

@stephenw10 Hi Stephen how do I send you the config file in a safe way? What do I need to sanitize? I do have radius on my server, should I remove that part of the xml? And what about the admin password is that stored on the xml backup?
I have one WAN interface, and one intel with 4 nics for the internal part, and is an old Dell desktop (those mini ones) that I can give you more specs, 16Gb of RAM. I had a ssd that failed before, but now it was working with a normal drive (but again failing)
The interesting thing is that this was only failing when that network that was being shaped was under heavy usage (probably reaching the limit of CBQ?)
Thanks

stephenw10

We probably don't need the full config but I can PM you something.

Just the details of VLAN in question and the shaper queues that were defined on it.

Steve

aschmi

@stephenw10 Sorry I was sure I sent you the config! Just to let you know that I reenabled QOS on that interface, and had a crash again... here is the config (what I think should be relevant)
(attached tried to post here but got rejected by anti spam)config.xml

Thank you so much for your time!!! And let me know If I need to send you anything else

stephenw10

igb1.2 is not in that config. Which interface did you enable it on to cause the crash?

Using the same queues that are on the other interfaces?

Steve

aschmi

@stephenw10 Sorry I tried to copy/paste but didn't work too well. Here is my actual config

I have QOS disabled now so it doesn't crash

Thanks again!

stephenw10

I was able to get the config OK it's just that it doesn't have the VLAN igb1.2 in it and it looks like that is what caused the crash.
Can I assume you had that configured the same as the other VLANs? Same traffic shaping queues on it?

Steve

aschmi

@stephenw10 Good you noticed and yes I was using the VLAN2 on igb1 (second interface on a 4 port 1Gbps interface) but not on my normal interfaces

I can send you the whole .XML if you tell me how to sanitize not to send usernames/passwords if that will help

stephenw10

What I have should be enough if the queues were the same.

However you can get the sanitised config as part of the status file by going directly to:
[your firewall IP]/status.php

Nothing special about igb1? igb0 is also a port on the same card?

Steve

aschmi

@stephenw10 status_output.tgz
There you have it. Yes igbo, 1, 2 and 3 is just one card (supposedly intel but bought on ebay)
I was trying to use igb1.2 as a bridge with igb0.2, but crashes happened before I started playing with that configuration

stephenw10

OK cool, so you had not added the bridge when it crashed?