Intel Atom C2xxx LPC failures

[VAMike]

@gcu_greyarea:

I fail to see why a small branch office should have a HA Router Setup.

Because it's an incredibly cheap way to ensure that an office full of people doesn't go idle because there's a network problem. I'd also recommend a backup internet connection if it's at all possible. I guess if salaries are really low in your area then the extra couple hundred bucks isn't worth it, but in most places nowadays a day without internet is going to cause a lot more than a couple hundred bucks worth of loss.

You seem to think that the C2xxx errata is the only thing that can possibly go wrong with a firewall, but most people would be sad if their office went offline for some period of time because (to use an example I've seen personally) their $5 wall wart failed. An HA configuration prevents multiple failure modes.

[gcu_greyarea]

Of course, there are many things that possibly can and will go wrong.
That's why you'd address the obvious and known issues first.

Looks like the C3000 has been announced…. which will most likely replace my C2000 kit. I'm sure the C3000 will have its own quirks, but hopefully intel learned from the clock issue.

I wouldn't want a "reworked" C2000 board anyway (if the soldering is done by a human).
Preference would be the intel fix - or - an entirely new CPU (i.e. C3000)

I decided that next time I will "Roll my Own" Hardware, so that I can blame myself if anything goes wrong. I still like the fanless ADI/Netgate kit, but I lost confidence in the company.

The advantage of the Netgate appliances is that they have no moving parts, don't require assembly and can be deployed quickly. But they are certainly not really that cheap, considering you can get more powerful hardware for the same money.
And despite the generous warranty extension I do not like the fact that Netgate won't proactively replace the systems for affected customers.
Let alone - continues to sell faulty units.

[Derelict]

Wasn't necessarily talking about HA but, since the term critical was used in the context of the unlikely event of an LPC component issue, there should be at least a spare on the shelf. If it's a critical router.

[chpalmer]

@Derelict:

since the term critical was used in the context of the unlikely event of an LPC component issue, there should be at least a spare on the shelf. If it's a critical router.

That is even true if your that afraid of your wife..  ;D

I usually just offer taking mine out to dinner and all is good!    ;)

[garyd9]

For those with supermicro boards who were approved for, and submitted the cross-ship agreement, have any of you had any news from supermicro since returning the cross-ship agreement?

I got the impression from that document (in particular, the part stating "*The replacement will be shipped by FedEx/UPS Standard Overnight for domestic destinations…") that they try to get those cross shipments out and delivered quickly.  If that's the case, at least some of you would have received a replacement board by now...

(I sent my form back yesterday, but it's been over 24 hours and I haven't seen anything back from them.  Not a tracking number, or anything else.)  I'm starting to wonder if they've actually shipped out ANY replacement boards yet (that have the issue resolved.)

That leads me to wonder if the "platform level change" that Intel claims works around the problem requires manufacturing a new revision of motherboards, or if it's something they can do as a "repair."  If the former, I'm wondering if Supermicro is just going to wait for a new stepping from Intel instead of bringing up new tooling/etc to produce new boards.

THAT, in turn, makes me wonder if the supermicro boards that pfsense got as "advance replacements" are actually boards with the problem resolved, or if they are just more with  the same issue, but held in reserve to quickly service failed devices.  (I don't think that tidbit of info was mentioned.)  If the latter, it would certainly explain why pfsense hasn't commented on if "current" stock has the issue already resolved or not.  The answer MIGHT be that current (and even current replacement) stock doesn't resolve the issue, because their vendor hasn't sent them anything yet with the issue resolved.

(Of course, all this is pure speculation and guessing.)

Edit: 45 minutes after posting the above, I got a tracking number from supermicro... which kind off negates this entire post.

[dopey]

@garyd9:

For those with supermicro boards who were approved for, and submitted the cross-ship agreement, have any of you had any news from supermicro since returning the cross-ship agreement?

…

That leads me to wonder if the "platform level change" that Intel claims works around the problem requires manufacturing a new revision of motherboards, or if it's something they can do as a "repair."  If the former, I'm wondering if Supermicro is just going to wait for a new stepping from Intel instead of bringing up new tooling/etc to produce new boards.

...

I received my replacement on Friday and it's already in place.  The board had a Tested 2/21/17 sticker on it from QA.
I honestly have no idea if it has the platform fix or not, but supposedly the platform fix can be retro-fitted.  This is what servethehome has listed for supermicro:

Supermicro: RMA for platform-level workaround available for concerned customers. We also did confirm that Supermicro has implemented the platform level workaround in products shipped from January 2017 onwards.

I suppose at this point I'll have to wait and see :)

FYI - my backup plan?  ordering a 4 port intel nic to stick into my virtual host and setup a backup pfsense instance there.  I'm also thinking of picking up one of those $50 edgerouter x devices to play with :)

[garyd9]

@dopey:

I received my replacement on Friday and it's already in place.  The board had a Tested 2/21/17 sticker on it from QA.
I honestly have no idea if it has the platform fix or not, but supposedly the platform fix can be retro-fitted.  This is what servethehome has listed for supermicro:

Did you notice if it's a REV 1 board, or if they bumped the REV to 02?  Is there any sign of jumper wires on the board?

[dopey]

@garyd9:

Did you notice if it's a REV 1 board, or if they bumped the REV to 02?  Is there any sign of jumper wires on the board?

I found nothing different other than the QA sticker.

I sent an email asking about any confirmation of the fix

Thanks for processing this.  I received the RMA on friday.  There doesn't appear to be any distinguishing marking (rev bump stuff like that) to note whether or not the board has the platform level workaround implemented for the atom cpu flaw.  Is there anyway to get some kind of confirmation that it actually has that workaround implemented?

And this was the response

Hello
The replacement has the issue fixed.

Guess I just have to trust them :)

[baggar11]

Does the new replacement board have a different stepping if you check via command line?

@dopey:

@garyd9:

Did you notice if it's a REV 1 board, or if they bumped the REV to 02?  Is there any sign of jumper wires on the board?

I found nothing different other than the QA sticker.

[garyd9]

@baggar11:

Does the new replacement board have a different stepping if you check via command line?

I got my replacement last night, and I see NOTHING different whatsoever on the board (other than its an obvious refurb that hasn't been as gently handled as my original.)  The CPU stepping is also identical: Origin="GenuineIntel" Id=0x406d8 Family=0x6 Model=0x4d Stepping=8

On my replacement (which was a cross-ship), I've had a few problems already.  I've had to clear CMOS a couple times to get it booting, and then it crashed (kernel crash) in the middle of booting pfsense, which then resulted in a corrupt filesystem (and we all know how poorly pfsense 2.3.x deals with that.)

All of these issues COULD be related to the CMOS being whacked out.

Since then, I pulled the CMOS battery, erased CMOS again (several times), reconfigured BIOS, completely reinstalled pfsense and restored a backup configuration.  So far, it doesn't seem to be doing anything bad… but it hasn't even been 24 hours since I got it working properly.

[dopey]

@baggar11:

Does the new replacement board have a different stepping if you check via command line?

The platform level workaround doesn't change the cpu stepping.  I'm not sure if Intel is shipping any new silicon yet.

@garyd9:

On my replacement (which was a cross-ship), I've had a few problems already.  I've had to clear CMOS a couple times to get it booting, and then it crashed (kernel crash) in the middle of booting pfsense, which then resulted in a corrupt filesystem (and we all know how poorly pfsense 2.3.x deals with that.)

All of these issues COULD be related to the CMOS being whacked out.

Since then, I pulled the CMOS battery, erased CMOS again (several times), reconfigured BIOS, completely reinstalled pfsense and restored a backup configuration.  So far, it doesn't seem to be doing anything bad… but it hasn't even been 24 hours since I got it working properly.

I didn't have any of those problems and mine's been in place since Friday afternoon without problems so far.

[pfcode]

My replacement is getting even worse, keeping shutdown itself for no reason within minutes after rebooting.  Installed back to my original board, working again. Obviously the replacement isn't fix at all.

[RobertLoblaw20381]

I also got a replacement from Supermicro. So far it's been good, no issues. Compared both boards side-by-side, I see no physical differences on the board itself. I also don't see any QC sticker or anything. I did notice that they removed a sticker on the top of the LAN port (that previously had a serial number on it). I only know because I can still see some adhesive from where the old sticker used to be. They replaced it with a similarly sized sticker that has the barcode, serial, and the date (2/17). When I emailed them asking how I can tell the difference, I literally got the same "The replacement has the issue fixed." response. Very frustrating.

One thing I did notice that is very different, is now on my pfSense dashboard, under System it says:

System Super Micro C2758
Serial: zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz

With my old board, it actually showed me the serial number. Now, it just shows what looks like a randomly generated UUID. I have no idea what caused that… Does anyone else have the same thing?

[garyd9]

@pfcode:

My replacement is getting even worse, keeping shutdown itself for no reason within minutes after rebooting.  Installed back to my original board, working again. Obviously the replacement isn't fix at all.

Within 24 hours…  The replacement board started to have NIC dropouts on all 3 of the i354 controllers in use.  When this happens, the switch reports that the cable is unplugged (and then later plugged in.)

I called supermicro and complained.  A lot.  (They wanted me to send them email.  I explained that email wasn't acceptable and wasn't going to make me go away.)  I ended up having to fill out another RMA.  I have to RMA this replacement board for another replacement board while the original RMA is left open (and the hold on my CC still in place.)  Once I get a working board with the intel issue supposedly resolved, I'll send back my original board and they'll release the CC hold.

Damn annoying, but still better than having to wait until the original board completely fails before they'll replace it.

@pfcode, I'd suggest calling supermicro's RMA dept and complaining a bit...

Take care
Gary

[VAMike]

yeah. so this is why I'm not rushing to replace my supermicro avoton gear…

[Derelict]

Damn annoying, but still better than having to wait until the original board completely fails before they'll replace it.

All boards, Clock Component or not, might fail. Yours might never fail.

[garyd9]

@Derelict:

All boards, Clock Component or not, might fail. Yours might never fail.

Absolutely true.  However, think of it another way:  You have a car with an expected life of 7 years.  The car manufacturer tells you that with regular maintenance and minor repairs, you can depend on the car lasting that long.

Then the car manufacturer discovers that one of their vendors gave them a part (which is 100% critical to the car, not easily replaced, etc, etc) that they've determined has a much "higher expected rate of failure after only 18 months."  No one tells you what "higher expected rate of failure" really means.  It might mean that after 18 months, the expected life drops to only 2 years… or perhaps 5 years.  Or, that only 1 in 50 million will have a shortened life expectancy.  You simply don't know, because the information is being purposely and deliberately hidden.

It's reasonable that if there was only a very tiny concern, that the vendor and/or car manufacturer would come out and say something like: "Don't worry about this!  The higher rate of failure only impacts one in every 5 million cars!"  That wasn't said... in fact, Intel is being VERY secretive about the whole thing (and forcing their direct customers to also be very secretive.)

So, being an IT paranoid individual, you assume the worst.  (Afterall, you use pfSense... and that defines you as paranoid.  A non-paranoid would just use some off the shelf gateway and not worry about firewalls, IPS/IDS, etc.)

Do you continue to drive the car, taking road trips, etc?  Or, do you first ask the car manufacturer to replace the defective part with a more reliable one?

Think of it in pfsense terms:  Someone gives you a reliable link to a list of strongly suspected hacking IP addresses (a DNSBL.)  You don't KNOW that malicious activity will come from all or any of those particular IP's.  It's reasonable to believe that you'd get hacked from an IP not on that list before one that IS on the list.  Do you install the DNSBL in snort or pfblockerNG?

The DNSBL is the news we have about the C2xxx chips.  snort/pfblockerng is sending the board in for repair or replacement.  Any questions? ;)

[garyd9]

@garyd9:

Damn annoying, but still better than having to wait until the original board completely fails before they'll replace it.

Another thing…  The "replacement" board they sent me ran about 6 degrees celsius HOTTER than my old/original with exactly the same fans, fan speeds, location, etc.  The old/original board has the CPU cores showing around 34 degrees and the replacement board was showing 40 degrees.  Room/ambient temperature is controlled at 72 degrees F.

Well, the replacement board is removed, and packed up ready to get sent back.

[dopey]

@RobertLoblaw20381:

One thing I did notice that is very different, is now on my pfSense dashboard, under System it says:

System Super Micro C2758
Serial: zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz

My original board and this one all show a random uuid.  For all I know, it's the same uuid.

I'm feeling pretty lucky compared to everyone else here then.  I've had no problems with my board, everything runs great, CPU temps are right in line with what they've always been (22-25c)

And everything just works.

[gcu_greyarea]

Gary,

If you can afford it go ahead and buy/build a new system based on another CPU platform. This entire C2000 saga is asking for trouble. You don't want a refurbed system that's been handled by - only god knows ….
You also dont want to sit on a time bomb.

Again - if you can afford it get another production system. Then take the time required to have your C2000 repaired/refunded. They will still make a good system for testing/playing or cold standby.

Your purchase dcision is your voting power.