Brand New SG-2100 bricked after update

mpcjames

I just had a brand new out of the box SG-2100 die after the update from 22.05 to 23.01.
The steps completed were as follows:

Brand new SG-100 unit purchased from UK distributor.
Ran initial setup wizard to configure it for my customer (only changed LAN subnet, DNS name, and DNS servers).
Completed the Initial Setup Wizard and was presented with Dashboard and working network connections.
Dashboard shows current firmware as 22.05.1 and that version 23.01 is available.
Started the upgrade and the unit rebooted after approx 10mins.
Waited another 30mins it still didn't boot.
No lights on front, but there are activity lights on rear ethernet ports.
I connected a USB console cable into putty and when pressing Enter, am getting an output stating "TIM-1.0"

The full output is:

TIM-1.0
>h
h/? - print this help screen
r yyyyyyyy - read register/memory at address yyyyyyyy in hex
w yyyyyyyy zzzzzzzz - write zzzzzzzz to address yyyyyyyy in hex
j yyyyyyyy - jump to address yyyyyyyy in hex
x y - change the boot mode, where y is in hex
a - UART control passed to AP CPU ROM
c - UART control passed to CM3 CPU ROM
>

If I power off the unit. Wait. And then power the unit back on it'll return to the TIM-1.0 message.
Still no lights on the front of the unit.

I have opened a ticket with the distributor we purchased through in the UK since the unit is brand new.
Luckily, I have another brand new SG-2100 unit in stock, so I will attempt to configure that one from scratch and see how I get on.

stephenw10

Did you open a ticket with us about that? Do you have the ticket number?

What you see there is the uart prompt. You should only ever see that if uboot cannot run for some reason. Which should never happen.

Steve

mpcjames

@stephenw10
Hi Stephen.
I haven't opened a ticket direct with Netgate as I wasn't sure if I could because I don't have the added Tac support .
Could you direct me to where I should lodge it, please?

Thanks
James

SteveITS

@mpcjames https://www.netgate.com/tac-support-request

All new units have a base level of support. https://www.netgate.com/support

stephenw10

You don't need an active support sub for an issue like that. All Netgate hardware is covered by TAC Lite anyaway. Open one here:
https://www.netgate.com/tac-support-request

Steve

mpcjames

@stephenw10 @SteveITS Thank you both. I will open that request first thing in the morning and see how I get on.

Gertjan

@mpcjames said in Brand New SG-2100 bricked after update:

so I will attempt to configure that one from scratch and see how I get on.

Before you even touch a RJ45 LAN / WAN / OPT port : hook up the console. Make it work.
Use, for example Putty, select the used COM port, have the console log to a file.

While you have the access over console, update from there.
Because you can now see what happens,
nothing happens (and all goes well).

mpcjames

@Gertjan
I carried out exactly the same steps as before with my second unit last night (although this time, after I went through the initial setup wizard I did perform a reboot before running the upgrade to 23.01) and the second unit worked fine with no issues.
After the 23.01 I rebooted again and was able to run the 23.05 update, and this also ran fine.

I can only assume it's something faulty with this original unit.

Appreciate what you say regarding the COM port, but I've never had to use it on any appliances in the past (except for when something crashes unexpectedly - usually due to power cuts). For the most part, to perform general updates and configuration etc, everything has always worked fine via the web GUI.

stephenw10

We are trying to replicate this failure internally now but so far have only been to do so by deliberately overwriting uboot which should obviously never happen.
Digging continues....

mpcjames

@stephenw10
I've got another 2 new SG-2100 units in stock which I need to setup and configure for different clients next week, therefore I can repeat the same steps on those ones as well and see how they perform.
I assume at some point you guys will come back on the support ticket with some instructions on how to recover this faulty one out of the state its in currently (not urgent for me right now)?

SteveITS

@mpcjames said in Brand New SG-2100 bricked after update:

After the 23.01 I rebooted again and was able to run the 23.05 update

That's actually a bit interesting also because it should have offered the current version 23.05 to begin with. Once I upgraded a router which was on a rather old version, it offered the not-latest, and I let it try, and it barfed terribly with it sort of half installing two versions. If pfSense was only offering the prior version I would either:

go into System/Update/Settings, change to Prior, wait a bit, change to Current, or
change it to Prior, upgrade to Prior, then afterwards change to Current and upgrade

FWIW we have a bunch of 2100s in service at clients and haven't had a problem updating those over the years.

Gertjan

@mpcjames said in Brand New SG-2100 bricked after update:

Appreciate what you say regarding the COM port, but I've never had to use it on any appliances in the past (except for when something crashes unexpectedly - usually due to power cuts).

Your past is what ? Last week ? ;)
(I guess I'll make my entire live these make so called null-modem cables - 3 wire connection)

COM ports (PC name of the device) or better known as RS232 is a easy simple and ancient communication port, long before even USB was invented.
It's an easy access for .... everything like koffee machine, elevators, etc etc etc. Basically : every device that has a CPU, but no screen keyboard.

There is one on a Netgate device, because : what happens when the network driver (NIC) has an issue, and doesn't load ? None of the network work ports would work. No more GUI ! What next ? Return the device ?
Or what about a bad disk/ SSD ? Same thing.

With this COM port (the cable will convert to USB, because PCs these days don't have COM ports any more) you have access to the device at 'bios' level, and you can 'do' thing in influence the boot process.

This interface is not optional. If you don't have it, you need it badly.
It's also the other way : if you have it, you don't need it.

mpcjames

@Gertjan
"Past" being 8+ years running pfsense on APU hardware and/or virtualised systems, and 2+ years running on official Netgate devices.

mpcjames

@SteveITS

Yes, I did think it was odd that it came out of the factory as 22.05.1 and then only offered 23.01 as "available" upgrade.
Both the initial "crashed" unit, and the second unit only offered 23.01. But with the second unit, it then offered 23.05 after the update to 23.01 and a reboot.

If I get time later I'll see what this third new unit offers.

Gertjan

@mpcjames said in Brand New SG-2100 bricked after update:

"Past" being 8+ years running pfsense on APU

You've been somewhat lucky maybe.
I was running pfSense myself on a downscaled Dell server : so the com port wasn't needed as it had a VGA + USB Keyboard. These always work, even when all the NICs are down.

A Netgate device doesn't have a VGA chip .... or HDMI/VGA plug, doesn't have a 'screen'.
That's where the com port kicks in.

stephenw10

23.01 is the version that introduced dynamic repos. Everything before 23.01 has to be upgraded via that in order to get the dynamic repo pkgs and then be able to see later versions.

Yes, when we have a practical solution to this we will come back to you on the ticket.

Steve

mpcjames

@Gertjan
Yeah, I know what it's for, I've just never needed to really use it.
Fingers crossed this one was just a glitch and I don't have to use it for every install as I do like the GUI.

mpcjames

An update for anyone else who many encounter this,...

Netgate support suggested (on my ticket) that I RMA the device to the distributor for replacement since it's under warranty (it's only 3 weeks old at this point).

I have been trying to get hold of my UK Distributor (Telappliant) on calls to them on 6th June, 8th June & today 12th June, but all they can tell me is that PFsense products are handled by a different support department and I can't get a response from them. I've got a ticket number, but that's meaning-less.

Anyone have any recommendations for a better UK disty?

Before they were taken over by Telappliant, they were called AmicaTech and their support was excellent (i.e. you could pick up the phone and speak with someone very knowledgeable within a matter of minutes). Now it's Telappliant, it's useless. As yet they haven't actually provided any "support" and this is a relatively straight forward case of raising the RMA and replacing the faulty item (especially since I've given them a official Netgate ticket ref telling me to do just that).

etnoy

Hey, just want to let you know that I had the same issue and was able to solve it last week with the help of Netgate TAC. I put some details to my solution in my reddit post: https://www.reddit.com/r/PFSENSE/comments/13y6o6c/netgate_2100_dead_after_upgrade/

In short, you don't need to RMA but you can fix it almost yourself.

stephenw10

Yes, it's possible to reflash uboot to it from a Linux client. It's not a procedure we would ever expect a user to have to do but you can do it if you need to.

If you want to go that route just reply on your ticket and we can get instructions to you.