Pfsense CE 2.7.0 to 2.7.1 big failure

[mcpfl]

I went ahead and installed the update from Pfsense 2.7.0 to 2.7.1 from the GUI and it all seemed to be OK until the firewall rebooted, the update failed to finish and the console was stuck in panic mode. I was able to rebuild by installing CE 2.7.0 and restoring the configuration from the backup. I see now there is a dump file with a ton of information, this is part of it.
Any advice on what went wrong, the hardware I am running is a Topton 12th gen N100 box with a 250GB SSD drive and 8GB of DDR5 ram

Dump header from device: /dev/label/swap0
Architecture: amd64
Architecture Version: 4
Dump Length: 157696
Blocksize: 512
Compression: none
Dumptime: 2023-11-22 12:31:18 +0000
Hostname:
Magic: FreeBSD Text Dump
Version String: FreeBSD 14.0-CURRENT amd64 1400094 #1 RELENG_2_7_1-n255918-774957be06d: Wed Nov 15 17:41:06 UTC 2023
root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_1-main/obj/amd64/GScwGwy
Panic String: Failed to free pagedep 0xfffff8000fdd1200
Dump Parity: 2441639242
Bounds: 0
Dump Status: good

[stephenw10]

If you can upload the crash report here I can look at it: https://nc.netgate.com/nextcloud/index.php/s/Crnjoeyrx3QWwR2

Steve

[mcpfl]

@stephenw10
Done, I dropped both the dump file and the info file

Thanks a lot

[stephenw10]

Backtrace:

db:0:kdb.enter.default>  bt
Tracing pid 305 tid 100157 td 0xfffffe0090091560
kdb_enter() at kdb_enter+0x32/frame 0xfffffe00848497d0
vpanic() at vpanic+0x163/frame 0xfffffe0084849900
panic() at panic+0x43/frame 0xfffffe0084849960
trunc_dependencies() at trunc_dependencies+0xbfe/frame 0xfffffe0084849a00
softdep_setup_freeblocks() at softdep_setup_freeblocks+0x843/frame 0xfffffe0084849a80
ffs_truncate() at ffs_truncate+0x6be/frame 0xfffffe0084849bd0
ufs_inactive() at ufs_inactive+0x273/frame 0xfffffe0084849c10
vinactivef() at vinactivef+0xde/frame 0xfffffe0084849c50
vput_final() at vput_final+0x1af/frame 0xfffffe0084849ca0
kern_frmdirat() at kern_frmdirat+0x278/frame 0xfffffe0084849e00
amd64_syscall() at amd64_syscall+0x109/frame 0xfffffe0084849f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0084849f30
--- syscall (137, FreeBSD ELF64, rmdir), rip = 0x8cc3655beea, rsp = 0x8cc35e7b9d8, rbp = 0x8cc35e7ba10 ---

Panic:

/: bad dir ino 21153794 at offset 1536: mangled entry
/: bad dir ino 21153794 at offset 0: mangled entry
/: bad dir ino 21153794 at offset 512: mangled entry
/: bad dir ino 21153794 at offset 1024: mangled entry
<118>===> Removing static obsoleted files
panic: Failed to free pagedep 0xfffff8000fdd1200
cpuid = 0
time = 1700656278
KDB: enter: panic

That's a UFS filesystem corruption issue.
Not clear why since it looks to have been shutdown correctly in the logs. It it did lose power at any point with an unclean shutdown that could have caused it.

Steve

[mcpfl]

@stephenw10

Any advise to run the update again? Does it look ok now to proceed. Can I run a command or utility to ensure the file system is free of any corruption?

[stephenw10]

It should be fine because you reinstalled which completely recreates the filesystem clean.
But to be sure you could run a manual filesystem check:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html#manual-filesystem-check

Steve

[mcpfl]

@stephenw10
I thank you for your help and great customer service

[pppfsense]

@mcpfl

Move away from UFS.

I did that as soon as ZFS came out as I was TIRED of having to fsck the "Firewall", every time the power went out.
(and/or the UPS was weak, or the communication to the UPS usb had failed).

Reinstall on different system/media and use ZFS instead.