SG-1000 essentially bricked with latest update

[TauCeti]

So I attempted an update to the latest 2.4 firmware almost exactly 3 hours and 30 minutes ago and it seems to have made the device completely unrespsonsive. The previous version I had was: 2.4.0.b.20170409.1957

It's a remote unit, and have been working through another person to get it going again. We managed to connect to the serial console but it's not possible to login. When attempting to enter login details we get error message:

getty[393]: /usr/bin/login: Exec format error getty[416]: getty: unknown gettytab entry 'default' getty[416]: getty: unknown gettytab entry 'al.3wire' getty[416]: getty: unknown gettytab entry 'al.3wire'

It does display the login prompt however it says:
lo_<random unicode="" char=""></random>_i_<random unicode="" char=""></random>_:_<random unicode="" char=""></random>_
(photo of this is attached)

There was zero abnormal behavior before hand except for:

1. First update attempt failed from webGUI
2. Second attempt, which was through SSH, also failed and seemed to get stuck in a loop. It was complaining about being "unable to lock . But it is the first file it always tries to lock.
3. Third attempt, which was also through SSH, completed, rebooted and from that point the device was completely unresponsive.

Attached is:

1. A horrible photo of the serial console output after a reboot.
   both
2. Login attempts

I'm just about to try a reflash.

[jimp]

Not seeing that here on either of my SG-1000 units. From the looks of it, there are files missing or corrupted causing it to fail in various ways.

Probably the original upgrade was actually still running (the GUI sometimes loses track of it and claims it failed when it did not – https://redmine.pfsense.org/issues/7334 ), and the subsequent attempts may have contributed to the problem.

Reflashing it should work though.

[TauCeti]

Yup a reflash fixed the problem.

Going through the information in that ticket and reflecting on what I was doing I think you are bang on.

One detail which I had forgotten about was that the second update attempt (the which was complaining about being unable to lock the file) I ctrl-C'ed (which was stupid in hindsight). I started the third attempt asap, and it was probably able to get the file locks as the GUI update had probably finished with those files and it was going through the motions to reboot, probably about the time the command-line update was editing those files -_-

So nothing exploding, just a wetware problem ^_^

Thanks for your help again jimp.

[mike98]

This has happened to me, not exactly the same issue but bricked after update. I'm trying to restore os per pfsense portal instructions and can't do that, not sure what to do next.
This is what it ends with when trying to restore with USB drive ( I don't have a micro sd card right now, I was wondering if that would change anything?)
How do I proceed to fix this?
Edit: add pic of where warning starts

[jimp]

Here is how to reinstall: https://www.netgate.com/docs/sg-1000/faq.html#how-do-i-restore-the-firewall-os-firmware

There are two methods listed there, try the first one and then if it doesn't work, try the second.

[mike98]

Ok I try to get micro sd today, screen shots are of method 1.

Edit: OK method 2 ended the exact same.

What can I do here? I'm thinking I gotta contact support. I can get to a loader> prompt during boot, but don't know what to do.

Edit 2: Found this at very beginning of boot up

Ok after looking at this I think I didn't dd correctly to sd card. Which I didn't, re dd'd now all is well. Sorry amateur hour, thanks for help n patients.

FIXED

[deadmalc]

Just did an upgrade and it looks like my SG-1000 is toast.
The upgrade looked like it went fine.
All booting looked OK.
There's no explanation why, it stated that it had been cleanly unmounted and stuck at:

Trying to mount root from ufs:/dev/ufsid/58480ee2a77beec6 [rw,noatime]…
WARNING: / was not properly dismounted
WARNING: /: mount pending error: blocks 71552 files 400
warning: no time-of-day clock registered, system time will not be set accurately

After awhile it came up with "random: unblocking device"
I left it for 10 minutes and still nothing, so I assume it wasn't doing an insane fsck.

Re-flashing , hopefully it's not a hardware fault.

Re-flash procedure seemed to run fine, I can't verify and get it backup and running at the moment.
One for the weekend.
But it looks like the upgrade goosed it.

[jimp]

Probably not the upgrade but perhaps some latent filesystem corruption from a previous unclean reboot/shutdown.

I've got two SG-1000s and I update them every few days and abuse them in various ways, they haven't ever died on me yet.

[LostInIgnorance]

At first I didn't want to report it as I thought it was a 1 off, but I am adding to this thread as I am experiencing the same exact issue and as we speak, I have a unit that refuses to boot.  This is the second time I have had to rebuild the SG-1000 in two and a half weeks.

JimP - What can I do to assist in the log building/debugging of this issue?

[jimp]

Contact our support team directly and give us as much information as possible (exact steps to reproduce, console output, error messages, etc)

[deadmalc]

@Jimp - I suspect something got bricked a couple of months back, after an upgrade I kept getting problems (an error message like PF was jammed). The only thing that seemed to fix it was a power off and restart.
Anyway, after re-flashing it seems OK - as yet it seems stable.
I'm not going to restore from a backup, as it's possible the config has something dodgy in it too.
I'm slowly adding services again.