Intel Atom C2xxx LPC failures

[gcu_greyarea]

Here’s another story:

INTEL - a company with 50 years of experience in producing CPU’s and selling millions of units says they have discovered an above projected failure rate in one of their products (C2000). They deem it important enough to inform their shareholders in their earnings forecast and release AVR54.

CISCO - a company with 30 years of experience with using CPU’s in their networking equipment and selling millions of units says there is an issue in several of their products using (C2000). They deem it important enough to stop-ship any affected products, put a workaround in place - and notify their customers.

Netgate/pfSense - a company with 2 years of experience in the hardware market, selling 100’s to 1000s of units says - nope - Intel the manufacturer of the product (C2000)and Cisco simply got it wrong.
We - pfSense/Netgate know better. There is no issue and it is unlikely to ever occur and we have made the conscious decision to not inform potential customers so that we can continue selling those faulty products in our online store.

Please do yourself a favour - and stop-ship - until you have reworked products.
You are gambling the value of a few units in your warehouse — against the future of your company.
Your customers will remember - and you risk dragging down the reputation of the pfSense software, too.

[VAMike]

@gcu_greyarea:

CISCO - a company with 30 years of experience with using CPU’s in their networking equipment and selling millions of units says there is an issue in several of their products using (C2000). They deem it important enough to stop-ship any affected products, put a workaround in place - and notify their customers.

Let's see what, exactly, cisco said:

Q: Is this a product recall?
No, this is not a product recall. Although the Cisco products with this component are currently performing normally, we expect product failures to increase over the years, beginning after the unit has been in operation for approximately 18 months. Although the issue may begin to occur around 18 months in operation, we don’t expect a noticable increase in failures until year three of runtime. For customers that determine proactive replacement is required, Cisco is offering to provide replacement products for those products under warranty or covered by any valid services contract dated as of November 16, 2016, which have this component.

So if you didn't have a service contract, you only get a new computer if it's still under warranty. How long is cisco's warranty?

All Cisco hardware and software products are covered by warranty for a minimum of 90 days

Yeah. It's all about the customers, not at all about limiting cisco's ongoing costs.

So why is a company that made $10bn in profits on $50bn revenue last year not replacing everything, the way you want netgate to? They certainly have the money to do it. Maybe because your analysis and conclusions are just wrong?

A really dedicated conspiracy theorist could even wonder if this is really all a tremendously clever way to put cisco's services competitors out of business. (Cisco has very publicly announced that everyone gets free hardware, but they're only paying for the installation work if you're on a cisco hands-on service contract. If you contract with someone else to provide the services, cisco just dumped a huge expense on your contractor after setting the expectation that all the work of replacing the hardware should be "free". And by prioritizing shipments, they can pretty much guarantee that those providers won't just hit a customer once and replace everything: they'll be tied up for months if not years going back again and again to replace a few units at a time.)

[gcu_greyarea]

VAMike, I appreciate your opinion, however - with all due respect - I think you are missing the point of my initial question. We are talking about two different things:

VAMike:
You are talking about Cisco's approach (proactive) to pull faulty units out of the field.
Netgate use a fix-on-fail approach (reactive), with generous warrany extension. I don't neccesarily like fix-on-fail and would prefer a proactive replacement - or as you call it "free stuff".
Basically this discussion is about dealing with faulty Units in the field - AFTER - the inits have been sold.

Me:
I am asking why Netgate/pfSense knowingly continue selling producs which are affected by AVR54?
This question is about how vendors chose to deal with the issue - BEFORE - customers purchase a product.

[VAMike]

@gcu_greyarea:

I am asking why Netgate/pfSense knowingly continue selling producs which are affected by AVR54?

@VAMike:

@gcu_greyarea:

Nobody has even come close to answering that question.

No, you just didn't like the answer: "because it isn't that big a deal".

[gcu_greyarea]

@VAMike:

@gcu_greyarea:

I am asking why Netgate/pfSense knowingly continue selling producs which are affected by AVR54?

@VAMike:

@gcu_greyarea:

Nobody has even come close to answering that question.

No, you just didn't like the answer: "because it isn't that big a deal".

I think we'll have to agree to disagree here …

We've had a very insightful discussion and I believe customers and potential future customers will be able to form their own opinions and make informed purchase decisions.

[garyd9]

@gcu_greyarea:

I think we'll have to agree to disagree here …

I came to that conclusion trying to talk to him a while ago (which is why I stopped wasting my time trying to get him to see more than one side of the argument.)

If it matters, I understand your point of view on this.  From a customer point of view, it sucks being told that you have to experience a failure and have your network go down for a few days (or weeks) before any action is taken on an issue that was already known about.

I also can understand why a smaller company, such as netgate, might not be willing or able to take on the massive expense of replacing hardware that hasn't yet failed.  Even if Intel gives them advance replacements for CPU's, and their m/b vendor does the same, they'd still lose a significant chunk of money.

Of course, more directly to your point, the complete silence from netgate in regards to their existing "new" stock is concerning.  I'd imagine that their legal folks are weighing in on silence as well (even if I don't understand why.)  I suppose there's a legal can of worms involved…

[a-a-ron]

I have a A1SRi-2558F and contacted SM directly yesterday, they told me to open a support ticket and then request a RMA. I did both and they are replacing the board no questions asked.

[apollo17]

I did the same, for an a1sri-2758f. I rma'd the board yesterday, they told me to make sure i include i included "Intel Atom C2000 problem, need ECO update" on the RMA form. Not sure how long it'll take, I'm in the UK and had to send it to the Netherlands.

[JeGr]

Nice for both of you, if you get new boards from Supermicro, I'd recheck again if they will already include a fix. Phoned our distribution of SM in Germany yesterday "no HW with fixes/workarounds available yet". So I'd be curious if I get a replacement within a few days if that really is a fixed board.

[apollo17]

I was very specific in my email. I asked directly if there was a fix or not, so if it is not a fix then they lied to me, for which i obviously wont be happy with considering the shipping costs. After all, it was a simple yes/no answer and thus wouldn't of given any information away. But, they deserve the benefit of the doubt. Not that i'd know one way or the other.

[Pippin]

Found this:
https://forums.freenas.org/index.php?threads/freenas-mini-motherboard-clock-signal-issue.50582/#post-349195
https://support.ixsystems.com/index.php?/Knowledgebase/Article/View/289

Not sure what to think of this:

any FreeNAS Mini or FreeNAS Mini XL manufactured on or after February 2017 will not have this issue

They have workaround?

[gcu_greyarea]

@Pippin:

Found this:
https://forums.freenas.org/index.php?threads/freenas-mini-motherboard-clock-signal-issue.50582/#post-349195
https://support.ixsystems.com/index.php?/Knowledgebase/Article/View/289

Not sure what to think of this:

any FreeNAS Mini or FreeNAS Mini XL manufactured on or after February 2017 will not have this issue

They have workaround?

It sounds to me like a company whose actions I respect, whose products I would buy and recommend to others.

[garyd9]

@Pippin:

They have workaround?

Yes, there's a platform level work-around for the issue.  That's in the original documentation from intel concerning the issue.

[dipswitch]

Crap. My Kerio NG300 appliance (on pfsense) also runs on avoton (C2558). Asked them today about their policy, no response yet  :(

[JeGr]

avoton C2558
Avoton would be C2550, C2558 (8 as last digit) are Rangeleys

[severach]

No cold spares for me. Unlike most products pfSense runs on any computer in the junk box and 4 port Intel cards are dime-a-dozen. I tested the config copy tonight to make sure it works.

Like they say in Aladdin: We all got cold spares!

[dopey]

In case anyone else has a supermicro board and wondered how this works I just asked them today:

"I'm inquiring about the currently known Intel C2000 cpu problem where the processor may fail unexpectedly over time. What are my options if I have an impacted motherboard?"

And this was the response i got less than an hour later:

"Thanks for contacting us regarding the Atom C2000.

You may submit a RMA at the following link to have the concerned issues addressed.  You may request the board to be cross-shipped during the RMA submission process.

https://www.supermicro.com/support/rma/"

[garyd9]

@dopey:

You may request the board to be cross-shipped during the RMA submission process.

Yeah, you can request it.  Heh.  Unless you purchased the board directly from supermicro and it's within 90 days, they'll likely deny the request.

If they are doing it differently for this specific issue, PLEASE let me know.  When I called them, I was told that I had no options for cross shipping as it's been 6 months since I purchased (and because I purchased from Amazon.com.)

I just moved pfsense to a VM and submitted my RMA request this evening.

[dopey]

the RMA person rejected the cross-shipping.  I ordered my motherboard august 2015.

I responded with this:

This was the response i received from technical support regarding my inquiry:
Thanks for contacting us regarding the Atom C2000.

You may submit a RMA at the following link to have the concerned issues addressed.  You may request the board to be cross-shipped during the RMA submission process.

https://www.supermicro.com/support/rma/

They told me I may request cross-ship.

I was under the impression that the intel C2000 flaw couldn't be repaired and needed to be replaced.  My motherboard is still functioning but it's a critical piece of equipment for me (my router) and the downtime would be very inconvenient.

And an hour later I got my cross-ship RMA form :)

[pfcode]

Hmmm, I submitted the RMA form,  but hours later, I got response: "This request only qualifies for repair service", ????