23.09.1 update failed. Now it won't restart. What now?

NGUSER6947

Per my prior post, I could not get past the 23.01.1 update. Then I noticed clients were having issues getting out.

I restarted the device several times (soft restart). No luck, same PHP error (though only one showed up).

I tried this, per the troubleshooting, from a shell:
pkg-static clean -ay; pkg-static install -fy pkg pfSense-repo pfSense-upgrade

Still nothing, so I did a Halt, waited a bit, then powered it down and finally powered it back up. Nothing, and I cannot ping it. Power light is on (green circle, solid).

I'm connected to the console now. Watching it boot, I get to this point:
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Updating motd:.
Updating /var/run/os-release done.
Clearing /tmp (X related).
Creating and/or trimming log files.
Starting syslogd.
Mounting late filesystems:.
Starting cron.
/etc/rc: WARNING: $scponlyc_enable is not set properly - see rc.conf(5).
/etc/rc: WARNING: $sshguard_enable is not set properly - see rc.conf(5).
Starting background file system checks in 60 seconds.

then this, over and over.

FreeBSD/arm64 (Amnesiac) (ttyu0)

-sh: /etc/rc.initial: not found

What now?

stephenw10

If it's a ZFS install you can roll back to the last good BE from a the boot loader menu.

If you have a config backup you can just install 24.03 clean and restore the config into it. Get the recovery image here: https://www.netgate.com/tac-support-request

NGUSER6947

@stephenw10

It is ZFS. But I can't ping it hence no web GUI).

I do have console connectivity, should I pause it at the pfSense menu and select one of the boot options?

I did download the installer package. I can leave it plugged into my cable modem and load that way, correct? I made a config backup from 23.09.1, will that restore into the current version?

SteveITS

@NGUSER6947 In general config files are forward compatible so you can restore onto 23.09.1 or later.

You can install either 24.03 or 23.09.1 from the net installer.

There is a command line boot environment restore if you want to revert.

NGUSER6947

@SteveITS

Thank you so much. I'll revert using the command line and at least get the system stable.

Thanks again.

NGUSER6947

So, I did the command line reboot. Ended up with this:



*** Welcome to Netgate pfSense Plus 23.05.1-RELEASE (arm64) on pfSense ***

 Current Boot Environment:  auto-default-20240507135936
    Next Boot Environment:  default

 WAN (wan)       -> mvneta0.4090 -> v4/DHCP4: 69.250.117.181/21
                                  v6/DHCP6: 2001:558:6020:118:a45b:4420:39da:5334/128
 LAN (lan)       -> mvneta0.4091 -> v4: 192.168.1.1/24
                                  v6/t6: 2601:147:4580:ef23:f2ad:4eff:fe19:2ba2/64
 OPT (opt1)      -> mvneta0.4092 -> v4: 192.168.2.1/24

 0) Logout (SSH only)                  9) pfTop
 1) Assign Interfaces                 10) Filter Logs
 2) Set interface(s) IP address       11) Restart webConfigurator
 3) Reset webConfigurator password    12) PHP shell + Netgate pfSense Plus tools
 4) Reset to factory defaults         13) Update from console
 5) Reboot system                     14) Enable Secure Shell (sshd)
 6) Halt system                       15) Restore recent configuration
 7) Ping host                         16) Restart PHP-FPM
 8) Shell

Enter an option:

Where do I go/what do I do now?

I tried restarting, but I just go back to this:

add host 127.0.0.1: gateway lo0 fib 0: route already in table
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Updating motd:.
Updating /var/run/os-release done.
Clearing /tmp (X related).
Creating and/or trimming log files.
Starting syslogd.
Mounting late filesystems:.
Starting cron.
/etc/rc: WARNING: $scponlyc_enable is not set properly - see rc.conf(5).
/etc/rc: WARNING: $sshguard_enable is not set properly - see rc.conf(5).
Starting background file system checks in 60 seconds.

Wed Dec 31 19:00:25 EST 1969

FreeBSD/arm64 (Amnesiac) (ttyu0)

-sh: /etc/rc.initial: not found

FreeBSD/arm64 (Amnesiac) (ttyu0)

-sh: /etc/rc.initial: not found

FreeBSD/arm64 (Amnesiac) (ttyu0)

-sh: /etc/rc.initial: not found

FreeBSD/arm64 (Amnesiac) (ttyu0)

-sh: /etc/rc.initial: not found

NGUSER6947

Ok, got it running under 23.05.1. Clients coming back online!

stephenw10

You should be able to upgrade from 23.05.1 to 23.09.1 without issue as long as there is sufficient drive space.

NGUSER6947

@stephenw10
So I did try the upgrade from 23.05.1 to 23.09.1, and it was successful this time.

Following that I did the upgrade to 24.03, which was also successful.

Thanks for your assistance!

stephenw10

I'd still guess that it ran out of space. The library errors you were seeing are indicative of an update that started but failed to complete.

Yes, you can reload the spare 1100 from the existing LAN. You will need to change it's LAN to a different subnet to prevent a conflict there though.

NGUSER6947

@stephenw10
To clarify, change the LAN to another subnet and I do also need to change it's (the one I am reloading) static IP to an unused IP correct?

stephenw10

The spare 1100 will need a WAN IP that is in your current LAN subnet. I would just use the default for that which sets the WAN as DHCP. It will pull a lease from your existing dhcp server and should be able to connect out.