Netgate 2100 - Update from 23.05.1 to 23.09: Firewall will not boot

ironmaiden2222

Hello!

I tried updating my 2100 from 23.05.1 to 23.09 from the UI.

The update was kinda weird, it being stuck blinking yellow but saying it was updated to 23.09.
However then I saw the package cache was at 95% storage or something similar.

Mistake 1

So I ran pkg-static clean -ay
I guess this was a mistake. I just would not respond.

Mistake 2

I was just stuck. Could not reach the UI nor ssh into it.
So I unplugged it, forcing it to reboot.

I still cannot reach it, and now I have plugged in to the serial port running screen like this guide says.
https://docs.netgate.com/pfsense/en/latest/solutions/netgate-2100/connect-to-console.html#launch-a-terminal-program

Current state

All I can see now when running sudo screen <console-port> 115200 is:

-sh: /etc/rc.initial: not found
FreeBSD/arm64 (Amnesiac) (ttyu0)

Now what?

So comes the million dollar question: What do I do now?
Any tips on what to do next?

jimp

If you were using ZFS, you should be able to select a previous version from the loader menu to roll back and start from there:

https://docs.netgate.com/pfsense/en/latest/backup/zfsbe/loader.html

If that doesn't work, you can always contact TAC for installation media to reinstall 23.09 cleanly.

SteveITS

@ironmaiden2222 What Jim said but for future reference did you wait 10-15 minutes for it to finish? “Square slowly flashes orange“ is upgrade in progress…

ironmaiden2222

@jimp Yes I created a ticket at TAC and got the firmware. Will try a to reinstall 23.09 cleanly.

@SteveITS yes I waited hours, it was just stuck while blinking orange.

MikeHalsey

I knew I should have read these forums before upgrading as my Netgate appliance got borked too. This is the second time it's happened in the short period I've owned it. Why can't Netgate release reliable firmware upgrades FFS!? You've got the hardware to test it on?

So I had to do a reinstall to get it working again. I'm not going to upgrade again for a while, but for a security appliance doesn't that defeat the object? Come on people, pull your finger out.

stephenw10

How did it fail for you? I assume was also a 2100?

ironmaiden2222

I finally managed to reinstall 23.09 cleanly I contacted TAC and got the firmware.

I was following the guide at https://docs.netgate.com/pfsense/en/latest/solutions/netgate-2100/reinstall-pfsense.html

One thing I really struggled with during this was the terminal output from the USB Console Port.

There is documentation on how to do this:
https://docs.netgate.com/pfsense/en/latest/solutions/netgate-2100/connect-to-console.html

I am on a Mac and tried using both regular Terminal and iTerm to connect to the USB Console Port.
Running sudo screen <console-port> 115200. I got partial outputs and it was basically impossible to give commands like run usbrecovery.

What worked best for me was installing Tabby (https://tabby.sh) and using that as the terminal.
It has built-in support for serial terminal. For anyone doing this on a Mac I would definitely recommend this

jimp

@ironmaiden2222 said in Netgate 2100 - Update from 23.05.1 to 23.09: Firewall will not boot:

I am on a Mac and tried using both regular Terminal and iTerm to connect to the USB Console Port.
Running sudo screen <console-port> 115200. I got partial outputs and it was basically impossible to give commands like run usbrecovery.

Curious that you saw partial output -- that's usually what happens if two separate processes are attached to the serial port at the same time. They'll each see ~half the output and input won't work properly. Maybe you'd attached a screen to the terminal already and detached (not quit)? Hard to say.

What worked best for me was installing Tabby (https://tabby.sh) and using that as the terminal.
It has built-in support for serial terminal. For anyone doing this on a Mac I would definitely recommend this

Tabby is a solid choice. It's cross-platform as well, so you can get approximately the same experience on Linux, Mac, and Windows. It has a couple problems that prevent me from using it primarily over others like SecureCRT but it's better than PuTTY for most things.

ironmaiden2222

@jimp said in Netgate 2100 - Update from 23.05.1 to 23.09: Firewall will not boot:

Curious that you saw partial output -- that's usually what happens if two separate processes are attached to the serial port at the same time. They'll each see ~half the output and input won't work properly. Maybe you'd attached a screen to the terminal already and detached (not quit)? Hard to say.

That sounds very plausible. I might have attached multiple processes like you say.

Maybe the guide at https://docs.netgate.com/pfsense/en/latest/solutions/netgate-2100/connect-to-console.html should warn against this?

I feel like it's a pitfall for people like me that are not very familiar with the sceen command.