XG-1541 broken after upgrade to 22.05

[getcom]

We have a problem with an older Netgate XG-1541. This box had pfSense 2.4.4 running.
First upgrade step to 2.4.5_p1 was successfully.
Next upgrade to 21.05 seems to be normal or we have overseen an error message.
Result was that the system version was still 2.4.5_p1 after reboot. After that we could select only 22.05 or 23.01. We decided to upgrade to 22.05 which had some PHP errors while installing the packages. Now 22.05 is running but without internet or LAN connection.
In the attached pictures you can see the error messages after first boot.
We asked the support for a download link for 22.05 and 23.01.

In the meanwhile until the Netgate support provides the image files, we could try to install from local cache with --no-repo-update.

Before you ask: The troubleshooting docs did not help here.

Here are the error messages after first boot.

Maybe somebody knows which php package is missing or broken?

[rcoleman-netgate]

@getcom Usually a reboot will clear up most PHP errors.

There are a load of system patches for 23.01 related to PHP issues.

https://docs.netgate.com/pfsense/en/latest/development/system-patches.html

We suggest you add all the recommended patches and reboot.

[getcom]

@rcoleman-netgate
As you can see in the Screenshot there is a fatal errors with a call to an undefined function.
This is a version missmatch or something similar. This cannot be fixed with a reboot.

How to add patches without GUI and why for version 23.01?
"...Now 22.05 is running but without internet or LAN connection..."
What we have is a console. We can download necessary packages on another system and store it on a USB flash drive and then mount it on the pfSense box.
There are no other possibilities at the moment, except new installation for which I still need the images from support. I already sent the serial number with the request to partner@netgate.com.
Is there a partner login to a download area for all our Netgate devices for all pfSense versions and we do not know anything about? If not this would be great in the near future.

[rcoleman-netgate]

@getcom said in XG-1541 broken after upgrade to 22.05:

How to add patches without GUI and why for version 23.01?
"...Now 22.05 is running but without internet or LAN connection..."

With no internet connection the only solution here is to install from the image you were sent by us in TAC.

We fulfill requests in <15 minutes most of the time. What's the ticket # for your request? You should have received an email with one immediately after opening the ticket to the requesting address.

If you did not go here to request the image you will not be receiving a ticket number and email with the image. https://go.netgate.com

[getcom]

@rcoleman-netgate I always wrote an email to partner@netgate.com in the past which never was a problem. My understanding is that TAC is for end users with support contract and not for partners.
Just check your emails, where you should find it.

[rcoleman-netgate]

@getcom Software images are a TAC fulfillment. Or you can use the Partner Portal to download images.

[getcom]

@rcoleman-netgate
Ok., let me check.
Additionally here are the last uncompleted installer log files until reboot.
Maybe this is interesting for the developers:

pfSense-Last-Log.txt

[SteveITS]

@getcom FWIW most times I've had issues with upgrades (out of not many) are when skipping several versions. I have had it work though. Usually Netgate says to just reinstall new instead of attempting it.

Images are in the partner portal, though not always very soon after release. (I don't always look but not all models had 23.01 a few weeks after release)

In theory pfSense is supposed to offer the current and previous versions so should have offered 23.01 and 22.05. If not it's not seeing the current info. In the past I have had it offer prior versions before and upgraded that way. If doing that I'd try to install the oldest, e.g. pick Previous Stable branch. Also, as suggested in the upgrade guide uninstall packages first. Otherwise the upgrade is going to try to reinstall the packages. Not quite sure what happens if "current" is selected and after upgraded to 22.05 "current" is actually 23.01 but if it pulls later packages, that is Bad.

[getcom]

@steveits
Hi Steve, thank you. I just downloaded both available versions for 1541, which is 22.05 and 23.01 and will start to install from scratch.
This is the third time in seven years that I have to reinstall. This is ok. for me.

[getcom]

@steveits
I could successfully install this system after wiping the ZFS meta data with dd (first and last 512k) and restored the last config.

Now I get this system crash report:

*Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #0 plus-RELENG_23_01-n256037-6e914874a5e: Fri Feb 10 20:30:29 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/obj/amd64/VDZvZksF/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBS

Crash report details:

PHP Errors:
[26-Apr-2023 23:25:14 Europe/Berlin] PHP Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/www/pkg_edit.php:675
Stack trace:
#0 {main}
thrown in /usr/local/www/pkg_edit.php on line 675
[26-Apr-2023 23:26:18 Europe/Berlin] PHP Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/www/pkg_edit.php:675
Stack trace:
#0 {main}
thrown in /usr/local/www/pkg_edit.php on line 675

No FreeBSD crash data found.*

Additionally I cannot start pfblockerNG config. I got this error:

Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/www/pkg_edit.php:675 Stack trace: #0 {main} thrown in /usr/local/www/pkg_edit.php on line 675 PHP ERROR: Type: 1, File: /usr/local/www/pkg_edit.php, Line: 675, Message: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/www/pkg_edit.php:675 Stack trace: #0 {main} thrown

Any idea?

[SteveITS]

@getcom Three seems high for one router, but if you're a partner maybe you mean across all routers? I do not remember the OS versions off the top of my head but 2.4 to 23.01 is probably multiple FreeBSD versions.

re: the PHP error, does it recur after restarting pfSense again? I seem to recall some Netgate posts about ignoring PHP errors on the first restart, or especially during the upgrade since those are expected as PHP is upgraded and packages are reinstalled.

"Uncaught TypeError: Cannot access offset of type string on string" is showing up a lot on the forums lately due to the change to PHP 8, and old code that doesn't work in PHP 8. If you install the System Patches package there are quite a lot of patches available already.

[getcom]

Restarting or reinstalling pfblockerNG does not change anything.
After deleting pfblockerNG and installing pfblockerNG-devel the error messages related to pfblockerNG are gone.
As I know the code base of both versions should be identical now.
Why I got this behavior with the pfblockerNG version and not with the pfblockerNG-devel version?
I assume that if the system is upgrated, up & running, every package which will then be automatically or manually installed will have the correct PHP version, or I'm wrong?

@steveits : "Three times" means overall of all routers in seven years beginning with pfSense 2.2.6, but surprisingly it is the second time for this system...
Does it makes sense to install every available patch in "System => Patches", or only if I identify a current problem in the description of a specific patch?

[SteveITS]

@getcom Up to you. Usually I’ve just installed as needed but there are a huge number for 23.01 and a shiny new apply all button so…

[stephenw10]

Yes, all of those patches are in 23.05 and well tested. I would also just apply all of them.

[getcom]

@stephenw10
Today I checked this box again because of the 23.05 update and got this:

There was no hardware change on this box which means I have no clue why the Netgate ID should have been changed.
I assume it has not been changed and this a bug. Can you fix this in your database?

[stephenw10]

Please send me the NDI and serial from that box in chat and I'll look into it.

[getcom]

@stephenw10
Thank you.

[rcoleman-netgate]

@stephenw10 said in XG-1541 broken after upgrade to 22.05:

Please send me the NDI and serial from that box in chat and I'll look into it.

Would it be too much to ask you to drop a link to these posts (our devices with new NDIs that need Repo access) to the internal tracking doc? 10337 IIRC.

[stephenw10]

@rcoleman-netgate Done

[rcoleman-netgate]

@stephenw10 Thank you. I know I have eyes on it now... but every entry helps.