Crash Report

hassling

@stephenw10
Sorry for the confusion, yes it is the same system and yes and disabled darkstat. Multiple crashes occur and continues.

stephenw10

Ok, can you upload any further crash reports to the Nextcloud link?

Or just give us the backtraces here to compare?

hassling

@patient0 I did. Still happening. I bought this Topton pfSense router from AliExpress two months ago.

hassling

@stephenw10 I'll post backtraces later as I am not home. Thank you

hassling

@stephenw10 uploaded.

Konstanti

@hassling

This line
current process = 7 (pf purge)
indicates that the error occurs at the pfSense boot stage when initializing the PF kernel module.
I would recommend to install the system again first

It is possible that there is some kind of problem at the hardware level

hassling

@Konstanti The last two differnet codes happened after I put an external USB fan (plugged to device the wall) under the router as per seller recommendation to dispense heat. Recorded temperature at 4 cores reach 62 Celsius though Pfsense occasionally throughs 70 Celsius.

Correlation vs causation? I am just trying to exclude.

stephenw10

Did the fan bring down the CPU temps? On an N100 that's normally passively cooled any fan should have a dramatic effect. If it doesn't I would check the CPU is correctly attached to the heasink/case, has thermal paste etc.

There are 4 distinct crashes there. One each in darkstat and netstat and all the others in pfctl and pfpurge which are very similar:

db:0:kdb.enter.default>  show pcpu
cpuid        = 3
dynamic pcpu = 0xfffffe009d4fff80
curthread    = 0xfffffe0020552740: pid 7 tid 100113 critnest 1 "pf purge"
curpcb       = 0xfffffe0020552c60
fpcurthread  = none
idlethread   = 0xfffffe002047fe40: tid 100006 "idle: cpu3"
self         = 0xffffffff84013000
curpmap      = 0xffffffff83020ab0
tssp         = 0xffffffff84013384
rsp0         = 0xfffffe00c5809000
kcr3         = 0xffffffffffffffff
ucr3         = 0xffffffffffffffff
scr3         = 0x0
gs32p        = 0xffffffff84013404
ldt          = 0xffffffff84013444
tss          = 0xffffffff84013434
curvnet      = 0xfffff80001240440
db:0:kdb.enter.default>  bt
Tracing pid 7 tid 100113 td 0xfffffe0020552740
kdb_enter() at kdb_enter+0x32/frame 0xfffffe00c5808b40
vpanic() at vpanic+0x163/frame 0xfffffe00c5808c70
panic() at panic+0x43/frame 0xfffffe00c5808cd0
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe00c5808d30
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00c5808d90
calltrap() at calltrap+0x8/frame 0xfffffe00c5808d90
--- trap 0xc, rip = 0xffffffff80fa3cfb, rsp = 0xfffffe00c5808e60, rbp = 0xfffffe00c5808e80 ---
pf_state_expires() at pf_state_expires+0xb/frame 0xfffffe00c5808e80
pf_purge_expired_states() at pf_purge_expired_states+0xd5/frame 0xfffffe00c5808ec0
pf_purge_thread() at pf_purge_thread+0x13b/frame 0xfffffe00c5808ef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe00c5808f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c5808f30
--- trap 0xe8e0e8e0, rip = 0x3cd03cdd431d431, rsp = 0x186b186b6af46af4, rbp = 0x5fc25fc2d476d476 ---

db:0:kdb.enter.default>  show pcpu
cpuid        = 1
dynamic pcpu = 0xfffffe009d4e1f80
curthread    = 0xfffffe00fdaee3a0: pid 36100 tid 100648 critnest 1 "pfctl"
curpcb       = 0xfffffe00fdaee8c0
fpcurthread  = 0xfffffe00fdaee3a0: pid 36100 "pfctl"
idlethread   = 0xfffffe0020480c80: tid 100004 "idle: cpu1"
self         = 0xffffffff84011000
curpmap      = 0xfffff800095f9ad0
tssp         = 0xffffffff84011384
rsp0         = 0xfffffe00fcdb3000
kcr3         = 0xffffffffffffffff
ucr3         = 0xffffffffffffffff
scr3         = 0x0
gs32p        = 0xffffffff84011404
ldt          = 0xffffffff84011444
tss          = 0xffffffff84011434
curvnet      = 0xfffff80001240440
db:0:kdb.enter.default>  bt
Tracing pid 36100 tid 100648 td 0xfffffe00fdaee3a0
kdb_enter() at kdb_enter+0x32/frame 0xfffffe00fcdb23e0
vpanic() at vpanic+0x163/frame 0xfffffe00fcdb2510
panic() at panic+0x43/frame 0xfffffe00fcdb2570
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe00fcdb25d0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00fcdb2630
calltrap() at calltrap+0x8/frame 0xfffffe00fcdb2630
--- trap 0xc, rip = 0xffffffff80fc1b62, rsp = 0xfffffe00fcdb2700, rbp = 0xfffffe00fcdb2be0 ---
pfioctl() at pfioctl+0x2ca2/frame 0xfffffe00fcdb2be0
devfs_ioctl() at devfs_ioctl+0xcc/frame 0xfffffe00fcdb2c30
vn_ioctl() at vn_ioctl+0xcf/frame 0xfffffe00fcdb2ca0
devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe00fcdb2cc0
kern_ioctl() at kern_ioctl+0x255/frame 0xfffffe00fcdb2d30
sys_ioctl() at sys_ioctl+0x123/frame 0xfffffe00fcdb2e00
amd64_syscall() at amd64_syscall+0x109/frame 0xfffffe00fcdb2f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00fcdb2f30
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x929772e79ca, rsp = 0x929740778a8, rbp = 0x92974077920 ---

All of those crashes are after bootup completes.

Nothing else significant is shown. How long does it take to crash after booting?

stephenw10

Similar to this: https://redmine.pfsense.org/issues/13417

hassling

@stephenw10 random between 1-2 days. However, when I removed the fan the system was stable for 4 days. I put the fan and connected it throught a USB on wall outlet within two days two crashes and froze once.

Now I removed the fan completely and I am watching. Next I'll check the thermal paste and update.

Any chance how clear darkstat remnants files or other packages (I already removed them)?

hassling

@stephenw10 Yes, the fan brought the temperature down to 40 Celsius.

Anything I can do to clear darkstats/ netstat remnants cache files?

stephenw10

darkstat was the 1st crash like 20days ago (1719551956). The netstat crash was the 5th of July (1720211173).

Hard to imagine the fan caused a crash. Especially if it wasn't powered from the device itself.

hassling

@stephenw10 A little update since last crash July 17th, no more crashes after removing the external (or USB) fan.

I am now inclined to believe that trying to cool the processor is the culprit for the crash. The processor seems to be very sensitive to temp change.

Hope this will help someone else. I'll post here if anything changes.

Thank you for your insight.

stephenw10

Hmm, odd. Hard to imagine that it doesn't like running cooler!