pfSense Crash diagnose

stephenw10

OK, so the important part of that is this:

db:0:kdb.enter.default>  show pcpu
cpuid        = 2
dynamic pcpu = 0xfffffe0080d71380
curthread    = 0xfffff80005740000: pid 12 tid 100035 "swi4: clock (0)"
curpcb       = 0xfffff800057405a0
fpcurthread  = none
idlethread   = 0xfffff80005622000: tid 100005 "idle: cpu2"
curpmap      = 0xffffffff8368d5a8
tssp         = 0xffffffff837176f0
commontssp   = 0xffffffff837176f0
rsp0         = 0xfffffe00747f2cc0
kcr3         = 0x80000000040cb002
ucr3         = 0xffffffffffffffff
scr3         = 0x10b690d52
gs32p        = 0xffffffff8371df08
ldt          = 0xffffffff8371df48
tss          = 0xffffffff8371df38
tlb gen      = 130953
curvnet      = 0
db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100035 td 0xfffff80005740000
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00747f2610
vpanic() at vpanic+0x197/frame 0xfffffe00747f2660
panic() at panic+0x43/frame 0xfffffe00747f26c0
trap_fatal() at trap_fatal+0x391/frame 0xfffffe00747f2720
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00747f2770
trap() at trap+0x286/frame 0xfffffe00747f2880
calltrap() at calltrap+0x8/frame 0xfffffe00747f2880
--- trap 0xc, rip = 0xffffffff80ec01fe, rsp = 0xfffffe00747f2950, rbp = 0xfffffe00747f2980 ---
ether_8021q_frame() at ether_8021q_frame+0x2e/frame 0xfffffe00747f2980
vlan_transmit() at vlan_transmit+0xc8/frame 0xfffffe00747f29f0
vlan_altq_start() at vlan_altq_start+0xb4/frame 0xfffffe00747f2a20
cbqrestart() at cbqrestart+0x64/frame 0xfffffe00747f2a50
rmc_restart() at rmc_restart+0x6f/frame 0xfffffe00747f2a80
softclock_call_cc() at softclock_call_cc+0x141/frame 0xfffffe00747f2b30
softclock() at softclock+0x79/frame 0xfffffe00747f2b50
ithread_loop() at ithread_loop+0x23c/frame 0xfffffe00747f2bb0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00747f2bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00747f2bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

It looks like on that last boot where us then crashed you has added a new VLAN interface:
vlan3: changing name to 'igb1.2'
Is there traffic shaping on that VLAN? Using CBQ?
Is it different to the other VLANs?

This looks pretty much exactly like this:
https://forum.netgate.com/topic/165827/random-crash
That was resolved removing the shaping from the VLAN.

Steve

stephenw10

Yeah, looks to be this: https://redmine.pfsense.org/issues/11470

aschmi

@stephenw10 Hi yes I do have CBQ traffic, in all the VLAN interfaces. The vlan3 is for "guest" so has a lower speed than the others.
This seems to be crashing when I'm heavily using that VLAN (on some guest machines that I was running Windows updates on)
Should I remove completely the traffic shapping, or is the problem with just CBQ?
Good eye I can still not see how you discovered the issue with CBQ and VLANS but great catch!
Adrian

stephenw10

The last lines in the backtrace before it panics point to it, read in reverse order:

ether_8021q_frame() at ether_8021q_frame+0x2e/frame 0xfffffe00747f2980
vlan_transmit() at vlan_transmit+0xc8/frame 0xfffffe00747f29f0
vlan_altq_start() at vlan_altq_start+0xb4/frame 0xfffffe00747f2a20
cbqrestart() at cbqrestart+0x64/frame 0xfffffe00747f2a50

That bug report confirms it.

Can you share any details of how you have it configured?
I'm trying to replicate it here but failing. It's stubbornly not crashing!

Steve

aschmi

@stephenw10 Hi Stephen how do I send you the config file in a safe way? What do I need to sanitize? I do have radius on my server, should I remove that part of the xml? And what about the admin password is that stored on the xml backup?
I have one WAN interface, and one intel with 4 nics for the internal part, and is an old Dell desktop (those mini ones) that I can give you more specs, 16Gb of RAM. I had a ssd that failed before, but now it was working with a normal drive (but again failing)
The interesting thing is that this was only failing when that network that was being shaped was under heavy usage (probably reaching the limit of CBQ?)
Thanks

stephenw10

We probably don't need the full config but I can PM you something.

Just the details of VLAN in question and the shaper queues that were defined on it.

Steve

aschmi

@stephenw10 Sorry I was sure I sent you the config! Just to let you know that I reenabled QOS on that interface, and had a crash again... here is the config (what I think should be relevant)
(attached tried to post here but got rejected by anti spam)config.xml

Thank you so much for your time!!! And let me know If I need to send you anything else

stephenw10

igb1.2 is not in that config. Which interface did you enable it on to cause the crash?

Using the same queues that are on the other interfaces?

Steve

aschmi

@stephenw10 Sorry I tried to copy/paste but didn't work too well. Here is my actual config

I have QOS disabled now so it doesn't crash

Thanks again!

stephenw10

I was able to get the config OK it's just that it doesn't have the VLAN igb1.2 in it and it looks like that is what caused the crash.
Can I assume you had that configured the same as the other VLANs? Same traffic shaping queues on it?

Steve

aschmi

@stephenw10 Good you noticed and yes I was using the VLAN2 on igb1 (second interface on a 4 port 1Gbps interface) but not on my normal interfaces

I can send you the whole .XML if you tell me how to sanitize not to send usernames/passwords if that will help

stephenw10

What I have should be enough if the queues were the same.

However you can get the sanitised config as part of the status file by going directly to:
[your firewall IP]/status.php

Nothing special about igb1? igb0 is also a port on the same card?

Steve

aschmi

@stephenw10 status_output.tgz
There you have it. Yes igbo, 1, 2 and 3 is just one card (supposedly intel but bought on ebay)
I was trying to use igb1.2 as a bridge with igb0.2, but crashes happened before I started playing with that configuration

stephenw10

OK cool, so you had not added the bridge when it crashed?

aschmi

@stephenw10 No I didn't have it in the initial crashes (the bridge). Well almost sure. This is my home so this is the only place where I don't document changes (oops) -
When it was failing badly (I guess more when I was trying to add a bridge to do some port mirroring as well), I did a change of hardware.
Then it seemed okay, but just for a while.
The only consistent result (and no crashes) I have is when I have shaping completely off (with the old or new hardware)

Should I remove the VLAN on igb1 and try again to add shaping to see what happens?

stephenw10

Let me try to replicate it with the values I have first.