Problems after Netgate 1100 Can't Update from 23.05.1 to 23.09

stephen.betts

I have just upgraded one device a Netgate 1100 to 23.09
It mostly seems fine, the IPsec tunnel cam straigt up

The system login page still says

23.05.1-RELEASE (arm64)
built on Wed Jun 28 03:57:42 UTC 2023
FreeBSD 14.0-CURRENT
Version 23.09 is available.

however when i check the Upgrade page again

Current Base System : 23.09
Latest Base System : 23.09
Status : Up to date.

from shell
cat /etc/version
23.05.1-RELEASE

Questions :
Am I successfully upgraded or not ?
Has something related to the main web page not updated correctly ?
Did some files get missed or not upgrade ?

alot of files seem to have a date of 2023-11-02

What other shell commands can i run ?

also i tried ssh into the console to do some checks, but it would not let me logon
said

-sh: /etc/rc.initial: not found
Connection to pfsense.example.com closed.

fortunately I happened to have a second device, and was able to connect a raspberry pi to the serial interface and teminal in to the bad device
I managed to scp a copy off the second good pfsense device that and upload it via the web interface, and move it to the correct location then "chmod +x /etc/rc.initial"
for execution

Now i can ssh in again

stephenw10

Try running at the command line: pkg-static info -x pfsense

See which pkgs have been upgraded.

Steve

stephen.betts

@stephenw10

pkg-static info -x pfsense

pfSense-23.05.1
pfSense-Status_Monitoring-php82-1.8_3
pfSense-base-23.09
pfSense-boot-23.09
pfSense-default-config-serial-23.05.1
pfSense-kernel-pfSense-23.09
pfSense-pkg-acme-0.7.4
pfSense-repo-23.05.1
pfSense-repoc-20230912
pfSense-u-boot-1100-20220428
pfSense-u-boot-2100-20210930_1
pfSense-u-boot-env-20230123
pfSense-upgrade-1.2_6
php82-pfSense-module-0.95

stephenw10

Do you know if it rebooted after the first upgrade attempt? The uptime on the dashboard should show that.

There may be an upgrade log in /conf showing any errors.

Otherwise you can probably upgrade the remaining pkgs from the command line using:
pkg-static upgrade

I would do that from the real command line not the GUI command prompt.

Steve

stephen.betts

@stephenw10

Main screen now shows updated

23.09-RELEASE (arm64)
built on Wed Nov 1 8:56:00 NZDT 2023
FreeBSD 14.0-CURRENT

However have this message at the top of the page

Netgate pfSense Plus has detected a crash report or programming bug. Click here for more information.

Crash report begins. Anonymous machine information:

arm64
14.0-CURRENT
FreeBSD 14.0-CURRENT aarch64 1400094 #0 plus-RELENG_23_09-n256163-2763857e770: Wed Nov 1 21:17:52 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_09-main/obj/aarch64/OKyCuLux/var/jenkins/workspace/pfSense-Plus-snapshots-23_09-m

Crash report details:
got Netgate pfSense Plus has detected a crash report or programming bug. Click here for more information.

PHP Errors:
[09-Nov-2023 06:24:43 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20220829/intl.so (Shared object "libicuio.so.72" not found, required by "intl.so"), /usr/local/lib/php/20220829/intl.so.so (Cannot open "/usr/local/lib/php/20220829/intl.so.so")) in Unknown on line 0
[09-Nov-2023 06:24:55 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20220829/intl.so (Shared object "libicuio.so.72" not found, required by "intl.so"), /usr/local/lib/php/20220829/intl.so.so (Cannot open "/usr/local/lib/php/20220829/intl.so.so")) in Unknown on line 0
[09-Nov-2023 06:26:08 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20220829/intl.so (Shared object "libicuio.so.72" not found, required by "intl.so"), /usr/local/lib/php/20220829/intl.so.so (Cannot open "/usr/local/lib/php/20220829/intl.so.so")) in Unknown on line 0

No FreeBSD crash data found.

ctowne

@stephen-betts

Just putting this out there in case anyone else has this same issue. I was in a similar situation, Netgate 1100 upgrading from 23.05.01 to 23.09 left my pfsense in a partially upgraded state. It was reporting upgraded to 23.09 but /etc/version was still showing 23.05.1. Running pkg-static upgrade twice completed the install for me. I never rebooted until I was sure all packages were updated. I received a similar crash report after rebooting with the following php errors.

HP Errors:
[09-Nov-2023 06:28:27 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20220829/intl.so (Shared object "libicuio.so.72" not found, required by "intl.so"), /usr/local/lib/php/20220829/intl.so.so (Cannot open "/usr/local/lib/php/20220829/intl.so.so")) in Unknown on line 0

stephen.betts

@ctowne

I upgraded my second device and had no problems at all.

Previous 23.05.01 update had trashed the second device and I had to reinstall using the tac image so I assume it was "clean"

both devices seem to be fine now and report the correct versions, the ipsec tunnel was still working regardless of the state of the packlages

I would have to say it took alot longer than i expected , possible approaching 30 minutes
I did capture a partial log using putty / serial connection to the second device, everything I saw upgraded

I captured this message at the end of the install which may be informative to some , i saw another post about problems with openvpn after upgrade .

Message from boost-libs-1.82.0_1:
--
You have built the Boost library with thread support.
Don't forget to add -pthread to your linker options when
linking your code.
You may need to manually remove /usr/local/etc/strongswan.conf if it is no longer needed.
=====
Message from strongswan-5.9.11_1:
--
The default strongSwan configuration interface have been updated to vici.
To use the stroke interface by default either compile the port without the vici option or
set 'strongswan_interface="stroke"' in your rc.conf file.
=====
Message from php82-gmp-8.2.11:
--
This file has been added to automatically load the installed extension:
/usr/local/etc/php/ext-20-gmp.ini
=====
Message from openvpn-2.6.5:
--
Note that OpenVPN now configures a separate user and group "openvpn",
which should be used instead of the NFS user "nobody"
when an unprivileged user account is desired.
It is advisable to review existing configuration files and
to consider adding/changing user openvpn and group openvpn.
=====
Message from dhcpcd-10.0.2:
--
The default FreeBSD kernel does not allow userland to provide IPv6
Prefix Routes when the kernel is handling Router Advertisements.
The default dhcpcd configuration will disable the kernel from handling
Router Advertisements.
See http://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194485 for a patch.

stephenw10

That OpenVPN message is expected. The new pkg prints that to the console when it's upgraded.

Those PHP errors are probably from before the upgrade was completed. As long as you're not seeing those after upgrading all pkgs and rebooting that shouldn't be an issue.

hoopy

@ctowne uh oh :-(

I tried to login via ssh - login worked, but no sudo.
Then I ran pkg-static info -x pfsense via the "Diagnostics / Command Prompt" UI and got the same output as @stephen-betts
pkg-static upgrade however, first asked for confirmation "Continue? [y]" - which I couldn't respond to via the UI.
I tried again with pkg-static upgrade -y (hoping it would automatically answer "yes" to such prompts - it didn't)
but it did time out and I got a "bad gateway" error.

and now I seem to have entered the valley of doom :-(

All done via {pfsense-host}/diag_command.php

pkg-static info -x pfsense

pfSense-23.05.1
pfSense-Status_Monitoring-php82-1.8_3
pfSense-base-23.09
pfSense-boot-23.09
pfSense-composer-deps-0.1
pfSense-default-config-serial-23.05.1
pfSense-kernel-pfSense-23.09
pfSense-pkg-aws-wizard-0.10
pfSense-pkg-ipsec-profile-wizard-1.1_1
pfSense-repo-23.09
pfSense-repoc-20230912
pfSense-u-boot-1100-20220428
pfSense-u-boot-2100-20210930_1
pfSense-u-boot-env-20230123
pfSense-upgrade-1.2_6
php82-pfSense-module-0.95

pkg-static upgrade

Updating pfSense-core repository catalogue...
pkg-static: An error occured while fetching package
pkg-static: An error occured while fetching package
repository pfSense-core has no meta file, using default settings
pkg-static: An error occured while fetching package
pkg-static: An error occured while fetching package
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
pkg-static: An error occured while fetching package
pkg-static: An error occured while fetching package
repository pfSense has no meta file, using default settings
pkg-static: An error occured while fetching package
pkg-static: An error occured while fetching package
Unable to update repository pfSense
Error updating repositories!

... but the SG1100 IS still running and I am still connected to the net... for now.

And to make it all that much more fun, I'm going on holidays on Monday (which is a good thing!)

Thanks for helping!

stephenw10

You should be able to run that with -y there. But it would be much better to use the real command line. You can't ssh in as admin?

hoopy

@stephenw10 :-(

I think it's dead Jim :-(

Years ago I set up a second user and was able to do everything that way, including ssh key etc. i.e. never use "admin/admin"

I don't have an ssh key for admin - and when I try to ssh as admin I just get "admin@{pfsense host}: Permission denied (publickey)"

Now the front-end is dead too - I got the login screen but now I tried to login as admin and just get "502 bad gateway" since then (it's been about 30 minutes now).

But I can still ssh in with my "other" (non-admin) user.
Just, no sudo and su just replies with "su: Sorry"

I recently did a full re-install (my SG1100 was one of the ones that needed the full firmware upgrade) and still have my configuration backup but I deleted the install image (since it all went so well at the time )

So, "contact support, get a fresh re-install image"?
I'm thinking that's beginning to sound like my only option... sigh

stephenw10

If you installed ZFS you can probably just roll back the BE.

hoopy

@stephenw10 hhmm....

as non-admin... (as I said, ssh is the only thing I have left)

% bectl list
BE                          Active Mountpoint Space Created
auto-default-20230910233855 -      -          693M  2023-09-10 23:38
auto-default-20231108132804 -      -          855M  2023-11-08 13:28
default                     NRT    /          3.46G 2023-02-10 22:00

I just read that a snapshot is made just before an upgrade (https://docs.netgate.com/pfsense/en/latest/backup/zfsbe/gui.html)

So... I could do a bectl activate auto-default-20231108132804 and reboot?
I only found docs for the GUI or "interrupting the boot process" (via JTAG/USB?)

stephenw10

Yup exactly that. Or you can select the BE from the loader menu by pressing option 8, then:

+---- Welcome to Netgate pfSense Plus ----+      __________________________  
|                                         |     /                       ___\ 
|  1. Back to main menu [Backspace]       |    |                      /`     
|  2. Active: zfs:pfSense/ROOT/default (1 of 3)|                     /    :-|
|  3. bootfs: zfs:pfSense/ROOT/default    |    |      _________  ___/    /_ |
|                                         |    |    /` ____   / /__    ___/ |
|                                         |    |   /  /   /  /    /   /     |
|                                         |    |  /  /___/  /    /   /      |
|                                         |    | /   ______/    /   /  _    |
|                                         |    |/   /          /   / _| |_  |
|                                         |        /          /___/ |_   _| |
|                                         |       /                   |_|   |
|                                         |      /_________________________/ 
+-----------------------------------------+

hoopy

@stephenw10 ok - here goes...

if I don't come back, I'm offline ;-)

hoopy

@stephenw10

short story: after several failed attempts at booting from old BE backups, I'm back on default and everything seems to have cleared up.

long story:

from the old BE images it boots, but the GUI is dead:

Both backup images showed an unresponsive menu and a red error message:

"local/www/head.inc, Line: 535, Message: Uncaught TypeError: count: Argument #1 ($value) must be of type Countable|array, bool given in /us/local/www/head.inc:535 Stack trace: #0 /us/local/www/index. php(309): include0) #1 (main) thrown

I then switched back to the default image and I can log in again (as admin).
pkg-static seems to think it's all OK, at least pkg-static info -x pfsense says I'm on 23.09-RELEASE and pkg-static upgrade also says I'm up-to-date.

So to sum up: booting from the BE images did not work, but after a few round trips, I seem to be back on track after booting into the default environment again.

Anyway, thanks for your help

Time for a holiday, no more updates until after I get back

ctowne

@hoopy said in Problems after Netgate 1100 Can't Update from 23.05.1 to 23.09:

pkg-static upgrade

I forgot to mention, I don't have SSH enabled, and that I connected to the console to run pkg-static upgrade. I hope that didn't contribute to any extra work. I did attempt to enable SSH when in was in the half updated state but it would not work (I was also missing /etc/rc.initial). After fully updating with pkg-static upgrade the file is now there. Next chance I'll get I'll enable SSH and try to login.

https://docs.netgate.com/pfsense/en/latest/solutions/sg-1100/connect-to-console.html