Intel Atom C2xxx LPC failures
-
This is not pfSense/Netagte fault. It is intel fault.
Absolutely. And we can hope that Intel will work with its customers, and not just the big ones. I mentioned in another thread that a friend had a SuperMicro board fail in a manner that is entirely consistent with this reported issue, and it took them 3 months (!) to get it back to him. The board went from California to Taiwan and back in that time. That, IMO, is unacceptable. And I'd consider SuperMicro to be, if not a "big" customer, at least one of the larger ones offering Intel's embedded hardware in what is advertised as enterprise class hardware. And I suspect, but don't know, that at least some of the Netgate/pfSense hardware is SuperMicro stuff.
This will not be easily glossed over. Intel needs to step up first, and give its customers a clear and easy path to remediation. And if that path has to trickle down through the OEMs like SuperMicro and whoever else Netgate might contract with, then those companies need to step up too.
-
https://www.crn.com.au/news/cisco-partners-pay-for-massive-product-replacement-450313
-
This is not pfSense/Netagte fault. It is intel fault.
I mentioned in another thread that a friend had a SuperMicro board fail in a manner that is entirely consistent with this reported issue, and it took them 3 months (!) to get it back to him. The board went from California to Taiwan and back in that time. That, IMO, is unacceptable.
Agreed that three months is entirely too long. Supermicro is supplying us with advanced stock so we can turn an RMA for this issue around in a day, rather than the timeline experienced by your friend. That said, your friend (probably) didn't buy from us, and there isn't much I can do if someone isn't a customer.
As I said in the blog post, we're standing behind our products, and will continue to do the right thing. If, via negotiation, we can extend the warranty for this issue to 5 or even 7 years, we will. (As a reminder, I wrote "at least 3 years". This is why.)
To get a replacement from Cisco, you'll have either purchased in the last 90 days, or entered into a very expensive "extended warranty" at the time of purchase. They are NOT replacing your Cisco device outside of these two eventualities. Their announcement is wordsmithed to lead the public to the conclusion that they are.
-
@MiB:
The reality is that no matter what Netgate, Cisco, SuperMicro post about whether they'll proactively replace all devices that contain the affected CPUs, or just failed devices and try to limit their exposure to some lame 3 year limit. They'll quickly learn that a class action is going to change their position very quickly, and empty their pockets much quicker than if they just replaced all affected units from the get go. intel is obligated to support all the costs. They've already incorporated a charge for this in their latest earnings. There are provisions within in the law that don't allow companies to hide behind time limits on manufacturing defects (latent or otherwise). Just ask Apple.
My advice, if you own an affected device, notify the supplier/manufacturer respectfully in writing that you expect a fixed replacement free of the defect within 90 days. If they don't comply, and/or don't reply, the law will set them straight and then some. Document your communications. A class action will be announced at some point.
True but the action may only affect america.
Like e.g. the nvidia gtx 970 scandal only gave americans a rebate.
I also think intel's NDA is quite possibly illegal in some countries, especially the EU, a NDA to hide design/manufacturing defects breaches various sales laws. This is not some new tech they want to keep underwraps but a released commercial product people have purchased. e.g. in the UK its illegal to sell something to someone with a known defect and not disclose it. The countries law that applies is in the place of sale, not where the company is HQ'd.
-
NDAs are simply standard practice. If you want access to proprietary information like futures, projected failure rates, direct purchasing requiements, etc., you sign an NDA. Like most companies, Intel uses bidirectional NDAs that cover all disclosures in the relationship. You should expect that any hardware manufacturer who uses Intel chips in their designs will have an an NDA with Intel. Many software companies do as well.
In other words, it's not a conspiracy. the NDAs the companies are citing are not new, nor specific to this issue. The "educated guess" in the serve the home article is simply wrong. No one who has been under NDA with Intel would suggest that.
-
Law here and law there is not the real thing we are talking about, it is more the thing that Intel or Supermicro are
able to serve us with a small program that is perhaps let us say installed on an USB Stick and with that we are all
able to deactivate this registers and then we reboot an were able to stich in a second USB pen drive with an inside
installed i2C chip that is then overtaking this part of work, then we would be all fine! If this is not able to work
around we should be waiting for another trail that will be shown by Intel or another vendor (producer) we all
are able to march. And if nothing helps out, we all know that problem now and we are able to get adequate
replacement by our own money, because our networks should be safe and secured and after this we are
sitting not in a really hard deep black hole and don´t came out. For sure this might be not ideal the most
peoples will think now, but if this units are not booting anymore the pain and stress factor is perhaps
much higher then the knowledge that something must be done before this units are failing!Its a really time bomb for sure, but really able to talk about that would we all only after a failure that is
able to show up! And not month or years before this failure will be perhaps occurring. -
@BlueKobold:
deactivate this registers and then we reboot
From what I've read, that could be as far as your proposal gets you. It might have been the last reboot ever initiated on that system… :)
-
I think gonzopancho mentioned at reddit that the ADI systems boot from i2c flash.
https://www.reddit.com/r/PFSENSE/comments/5s8pwi/intel_c_series_processor_recalls_are_pf_official/
So even id the LPC signal is not required during boot, the signal might still be required by other components ???
Then again, depending how the LPC processor component fails it may affect other parts of the processor, too.
Reading between the lines it appears that "usage" -or - heat may accelarate the deteriotion of said LPC component.
I'm sure Cisco would drive CPU's very hard as that is how you get "value" from your CPU. They wouldn't overspec the CPU to run it at 20% load. -
I agree with above poster who said class action is inbound pretty quick. This is going to get real ugly before it gets better.
-
@jwt:
Agreed that three months is entirely too long. Supermicro is supplying us with advanced stock so we can turn an RMA for this issue around in a day, rather than the timeline experienced by your friend. That said, your friend (probably) didn't buy from us, and there isn't much I can do if someone isn't a customer.
No, he didn't. Didn't mean to imply that he did at all. Glad to hear they're (they being SuperMicro) stepping up.
-
NDAs are simply standard practice. If you want access to proprietary information like futures, projected failure rates, direct purchasing requiements, etc., you sign an NDA. Like most companies, Intel uses bidirectional NDAs that cover all disclosures in the relationship. You should expect that any hardware manufacturer who uses Intel chips in their designs will have an an NDA with Intel. Many software companies do as well.
In other words, it's not a conspiracy. the NDAs the companies are citing are not new, nor specific to this issue. The "educated guess" in the serve the home article is simply wrong. No one who has been under NDA with Intel would suggest that.
depends on what is in the NDA, but a NDA that doesnt allow companies to disclose flaws is illegal in the UK. e.g. if I sold you a product with a predicted 18 month self life and didnt tell you that, then I have breached the sales act. It is a clear breach of "fit for purpose" tests. Projected fail rates doesnt quite fall into this category.
As the old motto goes, contracts cannot override law.
-
https://www.theregister.co.uk/2017/02/06/cisco_intel_decline_to_link_product_warning_to_faulty_chip/
Updated to add at 18:23 UTC, February 8
Once again, Synology has been in touch, seemingly now able to use the I word, to say: "Intel has recently notified Synology regarding the issue of the processor’s increased degradation chance of a specific component after heavy, prolonged usage."So what exactly does "heavy, prolonged usage" mean ? Can someone at pfSense Netgate confirm what Synology is saying?
Did Synology just let it slip that CPU usage (=heat) contributes to the issue?
Will my pfSense/Netgate ADI appliance die sooner if I have heavy CPU use ?
Will my appliance last longer if I add cooling ?
Will my appliance last longer if I disable CPU intensive services ?If not heat - how else could a solid state component degrade over time ?
-
thumbs up to synology for finally naming intel and the component.
What can intel do as punishment? not much as synology could just say we use a competitor cpu in future for new products.
-
If not heat - how else could a solid state component degrade over time ?
It sounds like a simple counter. If that is the case then the counter is incremented when the system is up and when reaching a certain threshold the device stops working.
Are we experiencing Planned obsolescence? ::)
-
The pfSense store still shows SG series products with the C2000 CPU.
Can someone at pfSense confirm whether these products contain the "platform level fix" - or is pfSense knowingly selling products affected by intel AVR54?
That would be very disappointing! -
That would be very disappointing!
What on earth is disappointing with that? :O As far as I was informed by as much as Netgate, Supermicro and other may already say about that problem, there is ATM no one, that actually HAS a working fix for the problem. Intel just said the know about some silicon workaround for boards with B0 and will work on a new stepping for further boards.
Are you suggesting immediatly stopping sales on all products that incorporate a C2000 SOC without actually knowing the full scale of the problem and its dependencies? Also all things related to "heat" as a problem are speculative, as nowhere was heat or even the CPU named a problem. So as far as I read - and that isn't meant as an insult - you are just panicking and jumping to conclusions.Are we experiencing Planned obsolescence? ::)
As far as I know of other manufacturers of devices, that is not the case. Many other vendors of products are already running for more than the mentioned 18 months with very few to zero error cases and those few not related to that failure. So I am waiting for more specifics to come to light for the circumstances of that failure occuring in the first place.
Greets
-
Hi JeGr,
you are correct in that I'm panicking and may be jumping to conclusions. However the reason I'm panicking is not that fact that there's an inherent flaw to the C2000 CPU's. I'm very well aware that hardware components will always fail.
I've worked nearly 20 years with Enterprise IT vendors and the reason I'm panicking is because I know how such issues have been dealt with.Cisco have chosen a proactive approach while others might take a 'fix-on-fail' approach. As far as I understand Netagte/pfSense are taking a fix-on-fail approach. You need to understand that fix-on-fail does not resonate well with the technical types, especially the security conscious technical types. What people want is peace of mind and irrespective of 'failure likelihood', fix-on-fail does not imply peace of mind.
If you ever worked in medium sized business (and upwards) you will easily find yourself in boardroom situation where you have to explain why pfSense/Netgate is taking a fix-on-fail approach while Cisco is being proactive.The fact that NDA's are in place and information is sparse will not help the IT Admin defend Netgate/pfSense's approach in the boardroom.
In this particular Case I am 100% certain that the issue is 100% understood by intel. The reluctant release of information is very considered.
Every word in intel's statement (AVR54) is very very carefully chosen to protect intel. This has a trickle down effect where component makers that use the C2000 will also very carefully chose every word in their respective statements.In order to stop customers from panicking, vendors will deliberateley steer away from too much technical information, because the real issue that needs addressing is customer confidence. Will a fix-on-fail approach retain enough customer confidence without detrimentally affecting future sales?
Company owners and shareholders that sell Systems based on C2000 want to sleep at night, too. A proactive approach would most certainly imply profit errosion in the short term (balanced against long term customer loyalty).
So whenever intel and vendors chose to share more 'technical specifics' - it has very little to do with root cause analysis, because we're way past that. At this stage customers are being Risk Managed. And when reworked boards and CPU's finally come available, you may be Case Managed.
-
I view the explanation as being relatively straightforward. On one hand you have a expensive unit with an expensive service agreement, and on the other hand you have an inexpensive unit with a simple one year warranty (which the vendor has committed to extend to 3 years for this issue). Business people understand risk/reward and will appreciate the difference. If you are really concerned, create a spreadsheet with a 3 or 5 year cost analysis. It should show quite positive, even if you allow for the purchase of a number of cold standby units.
-
Hi dennypage,
we're now comparing an expensive Cisco system with expensive smartnet contract against a cost effective ADI/Netgate system.
In you words:
Expensive + Support Contract (=peace of mind) - vs - cheap with flawed CPU component & warranty (=risky)You're reasoning that I should accept the risk because after all the ADI system is cost effective and I could afford having a cold spare sitting around.
That would be fine - if - the customer would have known about the risk - before - making that purchase decision.
Of course nobody knew about the issue until recently…. and now I'm sitting on a time bomb.Now that we do know about the issue - it would only be fair if potential customers of the pfSense and Netgate store would be informed about the processor flaw. So they can make an informed decision about the risk they're willing to take.
So going back to my original post - I would be "disappointed" if Netgate/pfSense knowingly sold me a flawed unit.
Why not put sales on hold, and wait for the reworked units ?
-
You're reasoning that I should accept the risk because after all the ADI system is cost effective and I could afford having a cold spare sitting around. That would be fine - if - the customer would have known about the risk - before - making that purchase decision.
You have always known the risk. It was inherent in the original decision. Sans purchasing cold spares, the original equation was high upfront plus recurring cost giving you 4-72 hours of potential outage (service contract depending) in the event of failure, vs. low upfront with no recurring cost giving you 4-7 days of potential outage in the event of failure during the first year (with no guarantee after). This is a simple risk tradeoff that any business person should be able to get their head around.
If the unit is critical to the business, you stock spares. This is a very common risk mitigation strategy which any business should be able to understand and accept.
