SG-2220 Boot Issue (No Console)

Carboxyl · Jul 29, 2017, 2:51 PM

Hi there,

I have an SG-2220 running up-to-date pfSense. Today I noticed low speed over the WAN. I rebooted the device from the web GUI to see if that would help. The SG-2220 didn't restart properly and seemed to get stuck. I waited for a bit, but nothing improved. I tried physically power cycling the device and still no luck.

At that point I realised something odd was going on. Here are the symptoms:

After every power cycle the status light lights up orange, and both Ethernet interfaces' green LEDs all go on. (So an orange status LED and all four green Ethernet LEDs go on solidly).
Doing a hardware factory reset (holding the reset button through a power-up for at least a minute) does nothing. The LEDs respond in exactly the same way.
There is no sign of the console. I tried PuTTY on Windows, screen on Linux, different PCs, with the correct settings (8N1 115,200 bps), etc.
Leaving the device on for an extended period, or off for an extended period does nothing.

Any ideas?

ivor · Jul 30, 2017, 9:10 AM

Hi,

Please contact our support at https://customercare.netgate.com

Please reference this thread.

Thanks!

Carboxyl · Aug 1, 2017, 11:27 AM

Thanks, Ivor. I'll keep the forum posted on any resolution.

Carboxyl · Aug 1, 2017, 11:20 PM

I contacted support and got told to use the console to recover. :P

After explaining the situation a second time, I got a "well your warranty's up, so sucks to be you" sort of response. (I'm paraphrasing, of course.)

I'm concerned about committing more money to the 300 or so already spent, just to be told it's dead.

Is there no way for an end user to try to re-write the flash storage, without the console? (Is there, perhaps, a JTAG header inside the device, or something like that?)

ivor · Aug 2, 2017, 9:04 AM

@Carboxyl:

I contacted support and got told to use the console to recover. :P

After explaining the situation a second time, I got a "well your warranty's up, so sucks to be you" sort of response. (I'm paraphrasing, of course.)

I'm concerned about committing more money to the 300 or so already spent, just to be told it's dead.

Is there no way for an end user to try to re-write the flash storage, without the console? (Is there, perhaps, a JTAG header inside the device, or something like that?)

Well I can understand the warranty part, it's just how it works. However, accessing console is a valid suggestion. You don't need a JTAG header, there's USB console port in the front of the device. It's the easiest and quickest way to figure out what's happening.

Can you please confirm you attempted to connect to console like described here: https://www.netgate.com/docs/sg-2220/connect-to-console.html

Very important, make sure you have console drivers installed https://www.netgate.com/docs/sg-2220/connect-to-console.html#advanced-configuration

If you followed those instructions correctly, you should have access to the console.

Try that and please report back. Most likely you will have to reinstall pfSense, which is pretty simple.

Carboxyl · Sep 7, 2017, 1:06 AM

OK. I've spent a few hours now going through every procedure you've suggested in detail.

I've confirmed that:

I have the correct CP210x USB to UART Bridge VCP driver installed correctly.
I have used the correct settings in PuTTY, including trying 115,200, 34,800 and 9,600 baud.
I have the cables plugged in correctly.

There is absolutely no output in PuTTY when I power up the SG-2220. Nothing.

As for the LEDs on the front of the device, the STATUS LED goes on solid orange and all four of the Ethernet port LEDs go solid green. Nothing blinks. There is no visible activity… Which is a clue. I don't think the clock is running. More on that in a moment.

I have also created a pfSense image on a USB drive (using the install image for the Netgate ADI platform) and tried powering up the device with the USB drive plugged in. The result is exactly the same. Absolutely no output in PuTTY and all LEDs stuck on solidly. (I tried this same procedure with an old PC using the AMD64 platform and it worked correctly on the old PC, so I know I'm getting this procedure right.)

I did a really deep dive on this problem and I can now see that this is almost certainly a hardware fault, and its symptoms exactly match the Intel Atom C2xxx LPC problem described in a few places:

https://www.netgate.com/blog/clock-signal-component-issue.html
https://www.theregister.co.uk/2017/02/06/cisco_intel_decline_to_link_product_warning_to_faulty_chip/
https://forum.pfsense.org/index.php?topic=125105.msg691024#msg691024
https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/atom-c2000-family-spec-update.pdf (AVR54)

Per the Netgate commitment in the first link, I assume I can get this device repaired or replaced. I'll report back on Netgate's response.

ivor · Aug 19, 2017, 9:11 PM

I was just informed that the RMA was approved. I'm glad we confirmed the issue. Please coordinate with our support the rest of RMA process.

Thank you!

Carboxyl · Sep 7, 2017, 1:04 AM

I'm very happy to say that Netgate replaced the motherboard in my device and it's working like new! :D

ivor · Sep 7, 2017, 10:13 AM

Thanks for the follow up! :)

dennypage · Sep 16, 2017, 4:23 PM

My SG-2220 just failed with the same symptoms. Reaching out to support.

shull · Nov 3, 2017, 4:29 AM

My SG-2220 just died with the same symptoms. One minute everything was fine, then it just stopped and everything was lit up like a Christmas tree. I cannot communicate with the box using the console cable. I will be contacting Netgate to see what my options are.

dennypage · Nov 3, 2017, 6:25 AM

Sorry, forgot to do a follow up post earlier…

As with the original poster, Netgate resolved my issue by replacing the board.

bcruze · Nov 3, 2017, 12:04 PM

i have the same box. did you perform the firmware update to fix the processor bug?

if i remember right because of the bug didn't they extended the warranty of this device as well?

i do NOT look forward to the day i have to replace this box. it has been 100% reliable until i did the 2.4 beta upgrades and it disabled the network ports…

the 2 port minnowboard will probably be my next replacement device from Netgate when that day comes

browner87 · Dec 9, 2017, 7:34 PM

My 2220 just died the exact same way. I'm really hopeful that support can replace it and do so asap. Now I need to scramble to find something I can put pfSense on so I have internet until they do…

grink · Dec 9, 2017, 8:02 PM

My SG-2220 just died today too, around 12:00 PM EDT. Solid green network LEDS and solid orange LED. No console output or anything. I wonder if the date had anything to do with it? I'm going to contact support as well.

flxkid · Dec 11, 2017, 6:10 PM

Mine just died in exactly the same manner as well….argh! Hopefully support can help out here

pfsensex2it.com · Dec 16, 2017, 9:19 PM

Count one more SG-2220 that just died in exactly the same way. They appear to have a serious issue that is not just an isolated occurrence or two.

ivor · Dec 17, 2017, 1:47 PM

Contact our support. C2000 series has a known issue and we offer an extended warranty because of it. Please contact our support so they can diagnose. https://customercare.netgate.com/

bcruze · Dec 18, 2017, 1:06 PM

just curious for those that have failed. before they failed obviously. did you apply the netgate coreboot upgrades?

CTGriffin · May 18, 2018, 1:52 AM

I have the same problem. My SG-2220 served well since 2015 and I am sorry to see it bricked in this state. I have ordered a new SG-1000 to replace it. I am curious if I can use my saved XML configuration for my various VPN setups or if I will need to manually reconfigure from scratch? No biggie either way as I know what needs to get done, but the saved XML from the SG-2220 would be a time saver.

Thank you for developing such a cool product!

–Christopher