page fault kernel panics after 2.5.2 upgrade

[Schoolofhardknocks]

I had something similar that happened to me and it happened during the boot sequence, so I had to reinstall pfsense altogether because I couldn't finish booting or restore a recent configuration, but the dump was more...

Tracing pid 431 tid 100111 td 0xfffff800055f2000
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00005a4620
vpanic() at vpanic+0x197/frame 0xfffffe00005a4670
panic() at panic+0x43/frame 0xfffffe00005a46d0
ffs_valloc() at ffs_valloc+0x8f3/frame 0xfffffe00005a4760
ufs_makeinode() at ufs_makeinode+0xa3/frame 0xfffffe00005a48f0
ufs_create() at ufs_create+0x34/frame 0xfffffe00005a4910
VOP_CREATE_APV() at VOP_CREATE_APV+0x75/frame 0xfffffe00005a4940
vn_open_cred() at vn_open_cred+0x2d9/frame 0xfffffe00005a4a90
kern_openat() at kern_openat+0x213/frame 0xfffffe00005a4c00
amd64_syscall() at amd64_syscall+0x387/frame 0xfffffe00005a4d30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00005a4d30
--- syscall (5, FreeBSD ELF64, sys_open), rip = 0x800b34e0a, rsp = 0x7fffffffd168, rbp = 0x7fffffffd1a0 ---

Then it proceeded with...

Tracing command sleep pid 96166 tid 100128 td 0xfffff800056c7740
sched_switch() at sched_switch+0x630/frame 0xfffffe00005f9a00
mi_switch() at mi_switch+0xd4/frame 0xfffffe00005f9a30
sleepq_catch_signals() at sleepq_catch_signals+0x403/frame 0xfffffe00005f9a80
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe00005f9ac0
_sleep() at _sleep+0x1b3/frame 0xfffffe00005f9b40
kern_clock_nanosleep() at kern_clock_nanosleep+0x1d2/frame 0xfffffe00005f9bc0
sys_nanosleep() at sys_nanosleep+0x3b/frame 0xfffffe00005f9c00
amd64_syscall() at amd64_syscall+0x387/frame 0xfffffe00005f9d30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00005f9d30
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x80038b6aa, rsp = 0x7fffffffec18, rbp = 0x7fffffffec60 ---

It repeated these same sleep system calls for a long time and then forced a reboot.
After reinstalling everything the crash was occurring within the pfSense webConfigurator. Its still happening, not sure why, but its not rebooting my box anymore..at least.

[Schoolofhardknocks]

@schoolofhardknocks

I also did some testing and when I reroot the device from WebConfigurator, it triggers the same crash dump. I see it launching on my COM.

[stephenw10]

@schoolofhardknocks said in page fault kernel panics after 2.5.2 upgrade:

panic() at panic+0x43/frame 0xfffffe00005a46d0
ffs_valloc() at ffs_valloc+0x8f3/frame 0xfffffe00005a4760
ufs_makeinode() at ufs_makeinode+0xa3/frame 0xfffffe00005a48f0
ufs_create() at ufs_create+0x34/frame 0xfffffe00005a4910

That's a filesystem error in UFS. You can probably recover by running a filesystem check:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html#manual-filesystem-check

Steve

[MrPete]

@stephenw10 @Schoolofhardknocks fsck is great for some filesystems... but if you have ZFS, a different set of tools is used. (Or, am I being dumb, and the fact that it is a UFS panic proves fsck is needed? ;) )

Expanding on the ZFS note on the linked page:

An advantage of ZFS: you can do the equivalent of fsck while up and running!

I do:

• zpool status (shows name of your pool(s), any past errors, and current scrub status)
• (assuming no HW error - use smartctl -x <device> to check the raw drives!***
• zpool scrub <poolname> to do a live scrub
• zpool clear to clear old errors (again, assuming it is not a HW error!)

*** The documentation describes SMART tools... not clear to me if the most extensive info is made available in the gui. smartctl -x shows quite a bit more... was added to the smart system a few years ago.

[stephenw10]

@mrpete said in page fault kernel panics after 2.5.2 upgrade:

the fact that it is a UFS panic proves fsck is needed?

Yes, that. You would not see that panic if ZFS was used.

Steve