6100 Max - Bricked - Won't boot after upgrade
-
Has anyone had any luck (or a similar situation) with resolving a Netgate 6100 Max becoming bricked after upgrading firmware through the GUI?
Background: I bought the device in April 2022, received it in May 2022, and just got around to turning it on and getting it updated--unfortunately it's out of warranty, and after the firmware update all 3 LEDs have remained steady white, with no change when powered up. The reason it took me so long to get this configured was after I received it I found there was a BSD bug preventing it from communicating with my Frontier ONT that tags traffic with a vlan0 and was recently resolved in an BSD stack update.
What I did:
- Upgraded firmware through GUI on local LAN, after initial setup (Running setup wizard, establishing initial connectivity to get the device updated).
Symptoms:
- All LEDs on 6100 max are illuminated white/steady (no blinking).
- Reset and Powecycle buttons do not work.
- No CLI/Console output
- No connectivity via ports
- The device has remained stored in a low humidity environment, packaged, and powered off for the past year.
- The device has never been dropped, damaged, or in harsh conditions.
What I've tried:
- Accessing via CLI over USB - Drivers loaded onto two different machines, with multiple known-good USB cables: Console session remains blank.
- Removing CMOS battery for 15 minutes.
- Reseating M.2 NVME drive.
- Opening ticket, burning agent provided image onto USB drive, plugging it in and rebooting 6100 while attempting Console access.
Still no luck. Not sure If I have a COVID "Lemon" or what, but the device was a splurge for me and I'm not comfortable with a $1000 appliance with a low MTTF not being serviceable. If anyone has any tips or knows of any jumpers on the board that can help unbrick, I am all ears. This device has not been powered on for more than probably a day and remained in the box for the past year. I consider it "brand new". Since it is out of warranty, Netgate is unwilling to provide any support other than the recommendations I've already attempted (above). I'm open to all recommendations--if anyone knows of any type of advanced troubleshooting/reset procedure, trade in, hardware servicing, board replacement, CSR program, etc., I am all ears.
-
When you say firmware do you mean a new pfSense version or what's offered by 'Netgate Firmware Upgrade'?
-
I did the firmware update through the GUI--I didn't do anything through the USB ports until the device frozen and I contacted Netgate support and they provided me with an image to burn to the USB.
-
But do you mean a pfSense version upgrade?
Or a BIOS/blinkboot upgrade which is what the 'Netgate Firmware Upgrade' menu offers?
A failed BIOS update could result it the box no longer POSTing but a pfSense upgrade should never do that.
Where you able to connect to the serial console before running the upgrade?
Steve
-
@stephenw10 I didn't use the console at all before I did the upgrade -- I was connected to the GUI via local subnet/IP from my PC. I first updated pfSense to the current version, then once it was complete, I updated the firmware. The GUI lost connectivity during the upgrade (expected) but when it said "waiting for it to come back online" with a countdown, the unit never responded -- this is when I walked away from it for over an hour, and returned to the LEDs being frozen on. It has to have something to do the the BIOS--I've tried removing the battery for a few hours, reseating SSD, basically everything. I tested the PSU with a multimeter, and that is in working order.
-
Hmm, OK. Do you have the ticket number you opened with us?
-
@stephenw10 I opened 2036643612 -- they provided the USB image, but without it POSTing I can't get anywhere, I bought the device in May 2022 and this was my real first hands on with it, it's out of warranty but I'm not sold on the idea that the "device is junk". I was waiting for the VLAN0 tagging to get fixed in freeBSD to work with my Frontier ONT (without needing to netgraph) which is why I was trying to get pfSense and the firmware updated before I configured it and put it into my "home prod".
-
Ok, let me see what the options are to recover from a bad firmware flash. I've never had to do that myself. Assuming it's not a component failure.
-
@stephenw10 Thank you very much -- I appreciate the help!
-
Probably be tomorrow before I can find out.
-
@stephenw10 I ordered a 4100 MAX yesterday and am interested in the outcome of his thread.
Ted
-
Make sure you open the box whilst it's still in warranty.
But that should have the most recent BIOS version already anyway.
-
@stephenw10 ...and check the bios version before doing anything!
Ted
-
I will add that if this proves to be a bad bios update it will be the first one I'm aware of.
-
I just submitted this post today about my 6100 possibly being bricked, it was interesting to see yours from just a few days ago:
https://forum.netgate.com/topic/183781/is-my-netgate-6100-bricked-3-lights-flashing-blue-blue-green
The light patterns seem different than yours (mine are blinking - see the gif in my post) and it just happened this afternoon. I bought mine in October 2011 and have been using it consistently since then without any significant downtime. There didnt seem to be any power incidents (outages, surges) today and the 6100 is connected to power through a UPS anyway
-
I'm interested in the outcome as well. My 6100 Base bricked as well last week. I get the stand by light and it won't output to serial or boot from USB. I worked with Support and they said it was hardware failure and nothing further could be done. To me it sounds more like firmware.
-
@deneon holy crap thats 3 6100s bricked last week by people who happen to read these forums? Is that an unusual amount of reports of these devices dropping dead in a given week?
I thought the fact that mine was unreachable and no CLI/Console accessible was strange and then I saw this post and I thought it was a coincidence, but a third person posting about a hardware failure last week feels like a pattern
-
@deneon said in 6100 Max - Bricked - Won't boot after upgrade:
I get the stand by light and it won't output to serial or boot from USB
What exactly do you see after connecting the power?
-
Sorry to bump an old thread.
I held off upgrading my 6100 that I've had for 2 years or so.
Today I tried to upgrade to 23.05 or so . It updated by some packages were not working. Like haproxy.
Showed they needed an update. They wouldn't tho .
So I noticed the next update was to 23.09.. it would not allow to attempt to update.I restarted the 6100 and now it won't start.
Is this all related and if so what can I do to fix.
I've tried resetting..
Fortunately I backed up my config just before the first update.
-
How is it failing at boot? You may be able to roll back the ZFS BE and try again.
However if you have the config I would just install 24.03 clean and restore the config (which you can do during the install).
