Fresh install of 2.7 on new hardware then restorng setting from my old hardware running 2.5.2. PHP errors, system crashes and reboots, very unstable

stephenw10

If you have that crash report you can upload the file dump here: https://nc.netgate.com/nextcloud/index.php/s/oHkedWmPJQFWTRw

mc866

@stephenw10 I've uploaded a textdump file

stephenw10

This looks like a filesystem error:

<118>Bootup complete
UFS /dev/ufsid/65148cf762e575fb (/) cylinder checksum failed: cg 207, cgp: 0xb01051 != bp: 0xae7eac13
UFS /dev/ufsid/65148cf762e575fb (/) cylinder checksum failed: cg 207, cgp: 0x4ade4a9d != bp: 0xae7eac13
/: inode 15624960: check-hash failed
/: inode 15624960: check-hash failed
/: inode 15624960: check-hash failed
UFS /dev/ufsid/65148cf762e575fb (/) cg 152: bad magic number 0x0 should be 0x90255
panic: softdep_deallocate_dependencies: dangling deps
cpuid = 0
time = 1695846014
KDB: enter: panic

That could be a bad drive. But the first thing to do is just run a few FSCK loops:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html

Steve

mc866

@stephenw10 I ran a few fsck loops and keep getting system marked dirty.

I also ran a reboot from the console (#5) with F and that seems to come back clean but when I run fsck after that I seem to get errors that come back such as bad magic number & unexpected soft update inconsistency.

I also booted into a live instance of ubuntu and ran fsck there as well as check file system from the disk utility and it check out as OK.

stephenw10

It could potentially be a bad drive then. But make sure you run fsck at least 5 times from single user mode in pfSense to be sure.

mc866

@stephenw10 Yes I ran it 5 times, a couple of times now LOL.

Seems like the results are very inconsistent.

Does a dirty drive mean it's bad or has errors?

stephenw10

If the filesystem cannot be cleaned or keeps becoming corrupt then, yes that could be a bad drive. But I have seen the same thing on older hardware with simply a lose connector.

mc866

@stephenw10

So I purchased a new nvme drive to test that out. I'm getting immediate errors again even with a fresh install of 2.7 and then a restore with no installed packages

I've added new dump files to the nextcloud share

Also noticed this in the logs and finally captured it.
As I stated I removed the packages from the XML but this is showing up as though the system is trying to pull down info as part of pfblocker

Oct 13 15:06:19 php-fpm 86183 /rc.update_urltables: : ERROR: could not update pfB_PRI1_v4 content from https://127.0.0.1:443/pfblockerng/pfblockerng.php?pfb=pfB_PRI1_v4

stephenw10

That pfBlocker can sometimes happen at boot before the lists have been fetched. It's not normally an issue.

Those crash reports all show filesystem issues still. It could be an issue with the board firmware. Is it running the latest BIOS?

Do you see those errors before you restore the old config?

Steve

mc866

@stephenw10

No errors on the old machine at all.

I've purchased another machine to try/test and just did a fresh install and system restore.

Just got this error on the new-new hardware. Looks like it's similar to the other machine

Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 RELENG_2_7_0-n255866-686c8d3c1f0: Wed Jun 28 04:21:19 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/obj/amd64/LwYAddCr/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/sources/FreeBSD-src-REL

Crash report details:

PHP Errors:
[13-Oct-2023 16:37:00 America/Chicago] PHP Fatal error: Uncaught TypeError: Unsupported operand types: string / int in /etc/inc/util.inc:2409
Stack trace:
#0 /etc/inc/pfsense-utils.inc(1902): get_memory()
#1 /usr/local/www/includes/functions.inc.php(104): pfsense_default_state_size()
#2 /usr/local/www/includes/functions.inc.php(35): get_pfstate()
#3 /usr/local/www/getstats.php(40): get_stats(Array)
#4 {main}
thrown in /etc/inc/util.inc on line 2409

No FreeBSD crash data found.

stephenw10

That's this: https://redmine.pfsense.org/issues/14648

Which is nothing to do with the drive. It's usually a one time error if you see it.

mc866

@stephenw10
Newest one

Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 RELENG_2_7_0-n255866-686c8d3c1f0: Wed Jun 28 04:21:19 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/obj/amd64/LwYAddCr/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/sources/FreeBSD-src-REL

Crash report details:

PHP Errors:
[13-Oct-2023 17:11:36 America/Chicago] PHP Fatal error: Uncaught DivisionByZeroError: Division by zero in /usr/local/www/includes/functions.inc.php:212
Stack trace:
#0 /usr/local/www/includes/functions.inc.php(33): mem_usage()
#1 /usr/local/www/getstats.php(40): get_stats(Array)
#2 {main}
thrown in /usr/local/www/includes/functions.inc.php on line 212

No FreeBSD crash data found.

stephenw10

@mc866 said in Fresh install of 2.7 on new hardware then restorng setting from my old hardware running 2.5.2. PHP errors, system crashes and reboots, very unstable:

functions.inc.php:212

That's the same bug. Did you try the patch on the bug report?

mc866

@stephenw10

Found it, that said how does one apply a patch?

Couldn’t seem to figure it out.

stephenw10

You can use the System Patches package, just use the commit ID in the patch.

https://docs.netgate.com/pfsense/en/latest/development/system-patches.html

mc866

@stephenw10

Awesome thanks!