pfSense has detected a crash report or programming bug.

bimmerdriver

When I upgraded to 2.7.0.a.20221104.0600, pfSense detected a crash report. The problem still exists in 2.7.0.a.20221107.0600.

Here is the panic string: softdep_waitidle: work added after flush

I attached info.0. I can upload textdump.tar.0 if required.

info.0

stephenw10

@bimmerdriver said in pfSense has detected a crash report or programming bug.:

The problem still exists in 2.7.0.a.20221107.0600.

You see it regenerated if you remove it you mean?

Is it created at boot? At upgrade between snapshots?

bimmerdriver

@stephenw10 said in pfSense has detected a crash report or programming bug.:

@bimmerdriver said in pfSense has detected a crash report or programming bug.:

The problem still exists in 2.7.0.a.20221107.0600.

You see it regenerated if you remove it you mean?

Is it created at boot? At upgrade between snapshots?

Every time I reboot the system, there is a crash report at the top of the dashboard. It started when I updated to 2.7.0.a.20221104.0600 and has remained up to 2.7.0.a.20221107.0600. I see there is another update available this morning. I will report back if there is any change.

bimmerdriver

I updated this morning and there is still a crash report after booting.

Should I attach the crash report?

stephenw10

Yes upload it we can check it.

bimmerdriver

@stephenw10 The files are attached.

info.0 textdump.tar.0

stephenw10

Hmm, this looks like a filesystem error while shutting down:

Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop... 
Syncing disks, vnodes remaining... 5 2 0 0 done
All buffers synced.
panic: softdep_waitidle: work added after flush
cpuid = 0
time = 1667574154
KDB: enter: panic

db:0:kdb.enter.default>  bt
Tracing pid 59106 tid 100170 td 0xfffffe0051c26740
kdb_enter() at kdb_enter+0x32/frame 0xfffffe006cb4aac0
vpanic() at vpanic+0x182/frame 0xfffffe006cb4ab10
panic() at panic+0x43/frame 0xfffffe006cb4ab70
softdep_flushfiles() at softdep_flushfiles+0x475/frame 0xfffffe006cb4ac10
ffs_unmount() at ffs_unmount+0x6a/frame 0xfffffe006cb4ac80
dounmount() at dounmount+0x7c3/frame 0xfffffe006cb4acf0
vfs_unmountall() at vfs_unmountall+0x6a/frame 0xfffffe006cb4ad20
bufshutdown() at bufshutdown+0x34e/frame 0xfffffe006cb4ad70
kern_reboot() at kern_reboot+0x734/frame 0xfffffe006cb4adb0
sys_reboot() at sys_reboot+0x382/frame 0xfffffe006cb4ae00
amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe006cb4af30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe006cb4af30
--- syscall (55, FreeBSD ELF64, sys_reboot), rip = 0x6408ed62d8a, rsp = 0x6408ccfc418, rbp = 0x6408ccfc590 ---

Has this system ever crashed out for any other reason?

The first thing I would do here is run a filesystem check:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html#manual-filesystem-check

Steve

bimmerdriver

@stephenw10 said in pfSense has detected a crash report or programming bug.:

Hmm, this looks like a filesystem error while shutting down:

Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop... 
Syncing disks, vnodes remaining... 5 2 0 0 done
All buffers synced.
panic: softdep_waitidle: work added after flush
cpuid = 0
time = 1667574154
KDB: enter: panic

db:0:kdb.enter.default>  bt
Tracing pid 59106 tid 100170 td 0xfffffe0051c26740
kdb_enter() at kdb_enter+0x32/frame 0xfffffe006cb4aac0
vpanic() at vpanic+0x182/frame 0xfffffe006cb4ab10
panic() at panic+0x43/frame 0xfffffe006cb4ab70
softdep_flushfiles() at softdep_flushfiles+0x475/frame 0xfffffe006cb4ac10
ffs_unmount() at ffs_unmount+0x6a/frame 0xfffffe006cb4ac80
dounmount() at dounmount+0x7c3/frame 0xfffffe006cb4acf0
vfs_unmountall() at vfs_unmountall+0x6a/frame 0xfffffe006cb4ad20
bufshutdown() at bufshutdown+0x34e/frame 0xfffffe006cb4ad70
kern_reboot() at kern_reboot+0x734/frame 0xfffffe006cb4adb0
sys_reboot() at sys_reboot+0x382/frame 0xfffffe006cb4ae00
amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe006cb4af30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe006cb4af30
--- syscall (55, FreeBSD ELF64, sys_reboot), rip = 0x6408ed62d8a, rsp = 0x6408ccfc418, rbp = 0x6408ccfc590 ---

Has this system ever crashed out for any other reason?

The first thing I would do here is run a filesystem check:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html#manual-filesystem-check

Steve

Thank you for your reply. I'm happy to report that your suggestion to fsck the file system seems to have fixed the problem. I had to do a couple of reboot / fsck. It reported the filesystem was clean after running it the first time, but it still had the error, so I did it again. After that, the error was gone. I did it a third time and it remained okay. I have no idea how this happened. The system did not crash, at least not that I noticed.

Thanks again.

stephenw10

Cool. That's an odd place to see a panic from the filesystem. Let us know if it returns.