Pfsense crashing daily on protectli vault

kindlywasp

Hi there,

I am running pfsense community edition version 2.8.1-RELEASE on a protectli vault VP2410 configured with 8GB of crucial ram and an SSD for the OS.

I am running a few plugins such as pfblocker, wireguard and a few others.

I have been running pfsense for about a year, however recently I appear to be getting daily crashes. I notice it as my internet goes down for about 5 minutes and comes back after pfsense has recovered.

I’m not sure what is causing this, but I have included the latest crash report log here: pfsense crash.txt

Any help would be appreciated, thanks.

stephenw10

Backtrace 1:

db:1:pfs> bt
Tracing pid 11 tid 100004 td 0xfffff8026f7ef000
kdb_enter() at kdb_enter+0x33/frame 0xfffffe008c4199b0
panic() at panic+0x43/frame 0xfffffe008c419a10
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe008c419a70
trap_pfault() at trap_pfault+0x46/frame 0xfffffe008c419ac0
calltrap() at calltrap+0x8/frame 0xfffffe008c419ac0
--- trap 0xc, rip = 0xffffffff80d15b68, rsp = 0xfffffe008c419b90, rbp = 0xfffffe008c419bf0 ---
callout_process() at callout_process+0x138/frame 0xfffffe008c419bf0
handleevents() at handleevents+0x186/frame 0xfffffe008c419c30
timercb() at timercb+0x236/frame 0xfffffe008c419c80
lapic_handle_timer() at lapic_handle_timer+0xab/frame 0xfffffe008c419ca0
Xtimerint() at Xtimerint+0xb1/frame 0xfffffe008c419ca0
--- interrupt, rip = 0xffffffff812ab776, rsp = 0xfffffe008c419d70, rbp = 0xfffffe008c419d70 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe008c419d70
acpi_cpu_idle() at acpi_cpu_idle+0x2cd/frame 0xfffffe008c419db0
cpu_idle_acpi() at cpu_idle_acpi+0x46/frame 0xfffffe008c419dd0
cpu_idle() at cpu_idle+0x9d/frame 0xfffffe008c419df0
sched_idletd() at sched_idletd+0x546/frame 0xfffffe008c419ef0
fork_exit() at fork_exit+0x7b/frame 0xfffffe008c419f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008c419f30
--- trap 0x3e023f12, rip = 0xf958f859e948e948, rsp = 0x9162906381629062, rbp = 0xc910d900d810c910 -

Backtrace 2:

db:1:pfs> bt
Tracing pid 11 tid 100005 td 0xfffff8026f7ee740
kdb_enter() at kdb_enter+0x33/frame 0xfffffe008c4149b0
panic() at panic+0x43/frame 0xfffffe008c414a10
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe008c414a70
trap_pfault() at trap_pfault+0x46/frame 0xfffffe008c414ac0
calltrap() at calltrap+0x8/frame 0xfffffe008c414ac0
--- trap 0xc, rip = 0xffffffff80d15b68, rsp = 0xfffffe008c414b90, rbp = 0xfffffe008c414bf0 ---
callout_process() at callout_process+0x138/frame 0xfffffe008c414bf0
handleevents() at handleevents+0x186/frame 0xfffffe008c414c30
timercb() at timercb+0x236/frame 0xfffffe008c414c80
lapic_handle_timer() at lapic_handle_timer+0xab/frame 0xfffffe008c414ca0
Xtimerint() at Xtimerint+0xb1/frame 0xfffffe008c414ca0
--- interrupt, rip = 0xffffffff812ab776, rsp = 0xfffffe008c414d70, rbp = 0xfffffe008c414d70 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe008c414d70
acpi_cpu_idle() at acpi_cpu_idle+0x2cd/frame 0xfffffe008c414db0
cpu_idle_acpi() at cpu_idle_acpi+0x46/frame 0xfffffe008c414dd0
cpu_idle() at cpu_idle+0x9d/frame 0xfffffe008c414df0
sched_idletd() at sched_idletd+0x546/frame 0xfffffe008c414ef0
fork_exit() at fork_exit+0x7b/frame 0xfffffe008c414f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe008c414f30
--- trap 0x118d009c, rip = 0x54a554b555b554a4, rsp = 0x2b4e3b4f2a5f2a4e, rbp = 0xb094b195b185b185 ---

Pretty much identical backtraces so probably something in software.

It could be this: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=291294

We are working on something similar. Let me check....

Surprising to see it in 2.8.1 though.

kindlywasp

@stephenw10 Thanks for getting back to me. It doesn't seem to have done it today, but is there anything I can do such as disabling pf blocker that would make any difference, or is this a bug with the OS itself?

Thanks.

stephenw10

If it's that bug it looks like something in kernel. Waiting to hear.

kindlywasp

@stephenw10 No worries, thanks for looking into this.