Problems after Netgate 1100 Can't Update from 23.05.1 to 23.09

stephen.betts

I upgraded my second device and had no problems at all.

Previous 23.05.01 update had trashed the second device and I had to reinstall using the tac image so I assume it was "clean"

both devices seem to be fine now and report the correct versions, the ipsec tunnel was still working regardless of the state of the packlages

I would have to say it took alot longer than i expected , possible approaching 30 minutes
I did capture a partial log using putty / serial connection to the second device, everything I saw upgraded

I captured this message at the end of the install which may be informative to some , i saw another post about problems with openvpn after upgrade .

Message from boost-libs-1.82.0_1:
--
You have built the Boost library with thread support.
Don't forget to add -pthread to your linker options when
linking your code.
You may need to manually remove /usr/local/etc/strongswan.conf if it is no longer needed.
=====
Message from strongswan-5.9.11_1:
--
The default strongSwan configuration interface have been updated to vici.
To use the stroke interface by default either compile the port without the vici option or
set 'strongswan_interface="stroke"' in your rc.conf file.
=====
Message from php82-gmp-8.2.11:
--
This file has been added to automatically load the installed extension:
/usr/local/etc/php/ext-20-gmp.ini
=====
Message from openvpn-2.6.5:
--
Note that OpenVPN now configures a separate user and group "openvpn",
which should be used instead of the NFS user "nobody"
when an unprivileged user account is desired.
It is advisable to review existing configuration files and
to consider adding/changing user openvpn and group openvpn.
=====
Message from dhcpcd-10.0.2:
--
The default FreeBSD kernel does not allow userland to provide IPv6
Prefix Routes when the kernel is handling Router Advertisements.
The default dhcpcd configuration will disable the kernel from handling
Router Advertisements.
See http://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194485 for a patch.

stephenw10

That OpenVPN message is expected. The new pkg prints that to the console when it's upgraded.

Those PHP errors are probably from before the upgrade was completed. As long as you're not seeing those after upgrading all pkgs and rebooting that shouldn't be an issue.

hoopy

@ctowne uh oh :-(

I tried to login via ssh - login worked, but no sudo.
Then I ran pkg-static info -x pfsense via the "Diagnostics / Command Prompt" UI and got the same output as @stephen-betts
pkg-static upgrade however, first asked for confirmation "Continue? [y]" - which I couldn't respond to via the UI.
I tried again with pkg-static upgrade -y (hoping it would automatically answer "yes" to such prompts - it didn't)
but it did time out and I got a "bad gateway" error.

and now I seem to have entered the valley of doom :-(

All done via {pfsense-host}/diag_command.php

pkg-static info -x pfsense

pfSense-23.05.1
pfSense-Status_Monitoring-php82-1.8_3
pfSense-base-23.09
pfSense-boot-23.09
pfSense-composer-deps-0.1
pfSense-default-config-serial-23.05.1
pfSense-kernel-pfSense-23.09
pfSense-pkg-aws-wizard-0.10
pfSense-pkg-ipsec-profile-wizard-1.1_1
pfSense-repo-23.09
pfSense-repoc-20230912
pfSense-u-boot-1100-20220428
pfSense-u-boot-2100-20210930_1
pfSense-u-boot-env-20230123
pfSense-upgrade-1.2_6
php82-pfSense-module-0.95

pkg-static upgrade

Updating pfSense-core repository catalogue...
pkg-static: An error occured while fetching package
pkg-static: An error occured while fetching package
repository pfSense-core has no meta file, using default settings
pkg-static: An error occured while fetching package
pkg-static: An error occured while fetching package
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
pkg-static: An error occured while fetching package
pkg-static: An error occured while fetching package
repository pfSense has no meta file, using default settings
pkg-static: An error occured while fetching package
pkg-static: An error occured while fetching package
Unable to update repository pfSense
Error updating repositories!

... but the SG1100 IS still running and I am still connected to the net... for now.

And to make it all that much more fun, I'm going on holidays on Monday (which is a good thing!)

Thanks for helping!

stephenw10

You should be able to run that with -y there. But it would be much better to use the real command line. You can't ssh in as admin?

hoopy

@stephenw10 :-(

I think it's dead Jim :-(

Years ago I set up a second user and was able to do everything that way, including ssh key etc. i.e. never use "admin/admin"

I don't have an ssh key for admin - and when I try to ssh as admin I just get "admin@{pfsense host}: Permission denied (publickey)"

Now the front-end is dead too - I got the login screen but now I tried to login as admin and just get "502 bad gateway" since then (it's been about 30 minutes now).

But I can still ssh in with my "other" (non-admin) user.
Just, no sudo and su just replies with "su: Sorry"

I recently did a full re-install (my SG1100 was one of the ones that needed the full firmware upgrade) and still have my configuration backup but I deleted the install image (since it all went so well at the time )

So, "contact support, get a fresh re-install image"?
I'm thinking that's beginning to sound like my only option... sigh

stephenw10

If you installed ZFS you can probably just roll back the BE.

hoopy

@stephenw10 hhmm....

as non-admin... (as I said, ssh is the only thing I have left)

% bectl list
BE                          Active Mountpoint Space Created
auto-default-20230910233855 -      -          693M  2023-09-10 23:38
auto-default-20231108132804 -      -          855M  2023-11-08 13:28
default                     NRT    /          3.46G 2023-02-10 22:00

I just read that a snapshot is made just before an upgrade (https://docs.netgate.com/pfsense/en/latest/backup/zfsbe/gui.html)

So... I could do a bectl activate auto-default-20231108132804 and reboot?
I only found docs for the GUI or "interrupting the boot process" (via JTAG/USB?)

stephenw10

Yup exactly that. Or you can select the BE from the loader menu by pressing option 8, then:

+---- Welcome to Netgate pfSense Plus ----+      __________________________  
|                                         |     /                       ___\ 
|  1. Back to main menu [Backspace]       |    |                      /`     
|  2. Active: zfs:pfSense/ROOT/default (1 of 3)|                     /    :-|
|  3. bootfs: zfs:pfSense/ROOT/default    |    |      _________  ___/    /_ |
|                                         |    |    /` ____   / /__    ___/ |
|                                         |    |   /  /   /  /    /   /     |
|                                         |    |  /  /___/  /    /   /      |
|                                         |    | /   ______/    /   /  _    |
|                                         |    |/   /          /   / _| |_  |
|                                         |        /          /___/ |_   _| |
|                                         |       /                   |_|   |
|                                         |      /_________________________/ 
+-----------------------------------------+

hoopy

@stephenw10 ok - here goes...

if I don't come back, I'm offline ;-)

hoopy

@stephenw10

short story: after several failed attempts at booting from old BE backups, I'm back on default and everything seems to have cleared up.

long story:

from the old BE images it boots, but the GUI is dead:

Both backup images showed an unresponsive menu and a red error message:

"local/www/head.inc, Line: 535, Message: Uncaught TypeError: count: Argument #1 ($value) must be of type Countable|array, bool given in /us/local/www/head.inc:535 Stack trace: #0 /us/local/www/index. php(309): include0) #1 (main) thrown

I then switched back to the default image and I can log in again (as admin).
pkg-static seems to think it's all OK, at least pkg-static info -x pfsense says I'm on 23.09-RELEASE and pkg-static upgrade also says I'm up-to-date.

So to sum up: booting from the BE images did not work, but after a few round trips, I seem to be back on track after booting into the default environment again.

Anyway, thanks for your help

Time for a holiday, no more updates until after I get back

ctowne

@hoopy said in Problems after Netgate 1100 Can't Update from 23.05.1 to 23.09:

pkg-static upgrade

I forgot to mention, I don't have SSH enabled, and that I connected to the console to run pkg-static upgrade. I hope that didn't contribute to any extra work. I did attempt to enable SSH when in was in the half updated state but it would not work (I was also missing /etc/rc.initial). After fully updating with pkg-static upgrade the file is now there. Next chance I'll get I'll enable SSH and try to login.

https://docs.netgate.com/pfsense/en/latest/solutions/sg-1100/connect-to-console.html