Diagnosing a PFSense Crash Report

solis

Hi All,

I am facing consistence crash report please help me to diagnose this crash main root cause I will be grateful to you. attaching my textdump.0 and info.0 file. thankyou in advance
info.0
textdump.tar.0

Gertjan

@solis

The interrupt handler for the network interface 'bce0' failed.

My advise : "remove some package load"
Like : Squid - vncstat - bandwidth.

What is "sqp_monitor" ?

stephenw10

Yes, something in bce:

db:0:kdb.enter.default>  show pcpu
cpuid        = 6
dynamic pcpu = 0xfffffe0080e65380
curthread    = 0xfffff808337a3740: pid 12 tid 100184 "irq256: bce0"
curpcb       = 0xfffff808337a3ce0
fpcurthread  = none
idlethread   = 0xfffff8000408b000: tid 100009 "idle: cpu6"
curpmap      = 0xffffffff8368d5a8
tssp         = 0xffffffff83717890
commontssp   = 0xffffffff83717890
rsp0         = 0xfffffe00ba05ae00
kcr3         = 0x8000000003d03002
ucr3         = 0xffffffffffffffff
scr3         = 0x9c11b3ee5
gs32p        = 0xffffffff8371e0a8
ldt          = 0xffffffff8371e0e8
tss          = 0xffffffff8371e0d8
tlb gen      = 446828
curvnet      = 0
db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100184 td 0xfffff808337a3740
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00ba05a8d0
vpanic() at vpanic+0x197/frame 0xfffffe00ba05a920
panic() at panic+0x43/frame 0xfffffe00ba05a980
trap_fatal() at trap_fatal+0x391/frame 0xfffffe00ba05a9e0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00ba05aa30
trap() at trap+0x286/frame 0xfffffe00ba05ab40
calltrap() at calltrap+0x8/frame 0xfffffe00ba05ab40
--- trap 0xc, rip = 0xffffffff805d97cb, rsp = 0xfffffe00ba05ac10, rbp = 0xfffffe00ba05ac90 ---
bce_intr() at bce_intr+0x48b/frame 0xfffffe00ba05ac90
ithread_loop() at ithread_loop+0x23c/frame 0xfffffe00ba05acf0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00ba05ad30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ba05ad30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Is it always in bce0? Can you swap the NIC assignments?

Steve

solis

@stephenw10 Currently my bce0 is my lan port are you asking me to swap it with some other port?? but i dnt have more port available in my server.

Gertjan

@solis said in Diagnosing a PFSense Crash Report:

to swap

No need for an available NIC.
You have four NIC's, bce0 to bce3.

swap means : change places - port WAN and LAN. Or assign LAN to an OPTx port, and that OPTx port to LAN You can assign them these ports the same way as you did during installing pfSense.

stephenw10

Yeah, that. If the crashes are frequent and identical then does swapping the NIC cause the fault to move to the new LAN NIC or remain on bce0 on the new interface?

Steve

solis

@gertjan i have only 4 lan ports bce3 is also a stand by port for backup internet i have no slot available to switch it with

Gertjan

@solis

Well, it's solved then : make your bce3 do what you bce0 now does.
Make bce0 your stand by / spare port.
That called swapping.

Remember :
bce0 doesn't need to be LAN WAN or OPTx
bce1 doesn't need to be LAN WAN or OPTx
bce2 doesn't need to be LAN WAN or OPTx
bce3 doesn't need to be LAN WAN or OPTx
LAN can be port 3 - WAN 2, spare 1 and 0 doing nothing.

solis

@gertjan brother i have 2 wans ready, 1 wan standby and 1 lan
all ports are full are you telling me crash are happening due to port bce0 ?

Gertjan

@solis said in Diagnosing a PFSense Crash Report:

are happening due to port bce0 ?

Exact.
So, - again - swap 'bce0' with 'bcex'. If the problem stays on 'bce0' you know that bce0 interface is bad.
If the issue shown before on bce0 is now on bcex, you know - pretty sure - that the NIC is ok.

solis

@gertjan [textdump.tar.1] i am still getting crashes

Gertjan

@solis said in Diagnosing a PFSense Crash Report:

[textdump.tar.1] i am still getting crashes

Where ? What interface ? Did you swap ?

What is :
[textdump.tar.1]

viktor_g

It could be a bce(4) bug:
https://bugs.freebsd.org/bugzilla/buglist.cgi?quicksearch=%5Bbce%5D

stephenw10

Mmm, looks like the attachment failed.

Does the crash follow the hardware or is it still on the same pfSense interface?

Steve

solis

@stephenw10 no it stay on pfsense

solis

@solis textdump.tar.0 check it i change the interface with bce2 but i dnt know maybe this crash report is old

stephenw10

Shows:

db:0:kdb.enter.default>  show pcpu
cpuid        = 2
dynamic pcpu = 0xfffffe0080e3d380
curthread    = 0xfffff808337bd740: pid 12 tid 100185 "irq257: bce1"
curpcb       = 0xfffff808337bdce0
fpcurthread  = none
idlethread   = 0xfffff80004084000: tid 100005 "idle: cpu2"
curpmap      = 0xffffffff8368d5a8
tssp         = 0xffffffff837176f0
commontssp   = 0xffffffff837176f0
rsp0         = 0xfffffe00ba05fe00
kcr3         = 0x8000000003d03002
ucr3         = 0xffffffffffffffff
scr3         = 0x8d1f6f8ae
gs32p        = 0xffffffff8371df08
ldt          = 0xffffffff8371df48
tss          = 0xffffffff8371df38
tlb gen      = 10148
curvnet      = 0
db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100185 td 0xfffff808337bd740
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00ba05f8d0
vpanic() at vpanic+0x197/frame 0xfffffe00ba05f920
panic() at panic+0x43/frame 0xfffffe00ba05f980
trap_fatal() at trap_fatal+0x391/frame 0xfffffe00ba05f9e0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00ba05fa30
trap() at trap+0x286/frame 0xfffffe00ba05fb40
calltrap() at calltrap+0x8/frame 0xfffffe00ba05fb40
--- trap 0xc, rip = 0xffffffff805d97cb, rsp = 0xfffffe00ba05fc10, rbp = 0xfffffe00ba05fc90 ---
bce_intr() at bce_intr+0x48b/frame 0xfffffe00ba05fc90
ithread_loop() at ithread_loop+0x23c/frame 0xfffffe00ba05fcf0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00ba05fd30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00ba05fd30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

current process		= 12 (irq257: bce1)
trap number		= 12
panic: page fault
cpuid = 2
time = 1643621859
KDB: enter: panic

1643621859 is Monday, 31 January 2022 09:37:39

So showing in bce1. But you swapped bce0 with bce2?

<118>WAN  -> bce1
<118>LAN  -> bce2
<118>OPT1 -> bce3
<118>OPT2 -> bce0

It looks like you might have re-assgined all of them....

solis

@stephenw10 i did swap lan on bce2 but today my device crash again when i check server display was on boot when boot complete configuration was deleted please help me recent crash report textdump.tar.0 info.0 please help me if possible should i change server or should a create vm server and install on it

solis

@stephenw10 textdump.tar.0 info.0 please help latest crash report

Gertjan

@solis

Are the bce0-3 network interfaces on a PCI card ? if so, you could change the entire card form some other quad NIC Intel based card.

@solis said in Diagnosing a PFSense Crash Report:

recent crash report textdump.tar.0 info.0

Well, you can see it yourself now : line 11 :

curthread    = 0xfffff808337bf740: pid 12 tid 100185 "irq257: bce1"

IRQ handling for bce1 crashed.