Another Netgate with storage failure, 6 in total so far
-
Yesterday, I had ANOTHER 4100 die, just 2 years old. I tried to salvage the situation by remotely walking the customer through installing pfSense onto a USB drive, but unfortunately, after I logged into the fresh installation and restored the configuration, the firewall refused to boot or even power on. There is no console activity and it just sits with a flashing orange LED. This is the second unit to die completely and refuse to power on.
I have tried to defend other pfSense issues as anomalies, but these storage and device failures are now a complete disaster.
No other firewall brand stipulates that storage must be treated gently for it to last.
This limitation might not be so bad if it was clearly visible and disclosed before purchase, but I challenge anyone to find any mention of the limitations/dangers of eMMC storage on the product pages or anywhere other than the 2 troubleshooting articles.
Let's take a look at the 6100 product page:
https://shop.netgate.com/products/6100-base-pfsenseLOW TOTAL COST OF OWNERSHIP -No artificial limits or add-ons required to make your system fully functional. -This system is designed for a long deployment lifetime. GROWS WITH YOU -From firewall to Unified Threat Management, get all the security features you need to protect your home or business. -Flexible configuration and support for multi-WAN, high availability, VPN, load balancing, reporting and monitoring, etc. -Add optional packages such as Snort or Suricata for IDS/IPS and network security monitoring. EASY GUI MANAGEMENT -Manage pfSense Plus software settings through our web-based GUI. -No fumbling with a command-line interface or typing arcane commands.The only noticeable difference between the BASE and MAX versions is the addition of a 128GB SSD for $100. The product pages mention all the great things that can be done natively and with packages, but they do not mention any storage concerns.
The product pages need a section that describes the limitations of the onboard eMMC storage with links to the "troubleshooting" documents, and advises getting the MAX version for anything other than basic, out-of-the-box usage.
Further, packages and any other system features, such as RRD graphs and logs, should include text warning of the storage issues, and contain links to the documentation. The current situation leaves a high risk that a user will make a few simple changes and unknowing turn their Netgate appliance into a ticking bomb.
To make things worse, the default configuration does not automatically perform disk health monitoring nor does it place the SMART widget on the dashboard. Monitoring eMMC storage requires using the CLI to install a package and run commands manually. Even worse, the storage of the 4200 cannot be monitored at all!
I have read recommendations from Netgate staff to disable logging the default firewall ruleset to reduce storage wear.
Enabling RAM disks could help alleviate these issues, but then all logging is lost on reboot. This would be a possible solution if the general system log was kept on disk for troubleshooting and security monitoring, but some things (like ARP watch, gateway issues etc) can still flood the log and cause disk writes.
We like pfSense, and the Netgate hardware is fine, but the Achilles heel is the eMMC storage, which is simply unfit for purpose. There are many posts online and here in the forums of people with similar issues.
For a business-class device, the onboard storage device and limitations do not make sense.
My management team is concerned and we are looking at solutions for our entire fleet of 45 Netgates before more fail and cause disruption to our customers.
If there is something we are overlooking, I would be happy to hear any suggestions.
-
A andrew_cb referenced this topic
-
Including the eMMC monitoring package with pfSense was requested 3 years ago but so far still has not been done:
-
@andrew_cb I agree with you. I make it a point to only deploy MAX versions of the Netgate due to the storage. The lowest spec i would go is the 4200 and must be MAX.
On top of the issues you mentioned, you have to take care about the amount of logging that you do. In my case, every single rule created is logged. Thats the policy. My 6100s would've died a year into service but because they are running SSDs, i am 2 years in and without any issues. If you are logging heavily or just have lots of I/O to your drive for whatever reason, selecting a Netgate with eMMC is going to cause you lots of pain. The only exception I make is the 1100. Thats a very cheap device you throw into a closet somewhere in a cafe not in a datacenter so the risks I take with it are worth having.I think its a huge flaw to advertise devices with eMMC storage. The standard should be SSD drives, full stop.
-
@andrew_cb said in Another Netgate with storage failure, 6 in total so far:
Enabling RAM disks could help alleviate these issues, but then all logging is lost on reboot. This would be a possible solution if the general system log was kept on disk for troubleshooting and security monitoring, but some things (like ARP watch, gateway issues etc) can still flood the log and cause disk writes.
You should be sending logs to a remote syslog server to be fair.
@andrew_cb said in Another Netgate with storage failure, 6 in total so far:
Even worse, the storage of the 4200 cannot be monitored at all!
If this is for a business deployment then not selecting this model is a no brainer. If there are no tools to monitor the health of the device including SNMP related monitoring for it, then don't deploy it. You are putting clients at risk putting an unreliable and un-monitorable solution in their environment.
-
You should be sending logs to a remote syslog server to be fair.
I definitely agree, but again, there is no indication that just the regular device logs are a threat to longevity. It would also require more infrastructure setup than comparable devices. For example, Sonicwall, Sophos, Fortinet, Meraki, etc devices can do the same kind of logging for 10 years without an issue. Filling up the storage space is one thing, but having it outright die in 2-3 years due to logging is ridiculous.
If this is for a business deployment then not selecting this model is a no brainer. If there are no tools to monitor the health of the device including SNMP related monitoring for it, then don't deploy it. You are putting clients at risk putting an unreliable and un-monitorable solution in their environment.
The product page has no mention of the inability to monitor the 4200's eMMC storage, which is a loss of functionality compared to the 4100 and 6100 it replaces. Unfortunately, we've already purchased and deployed several 4200 (including 3 to replace failed 4100). I completely agree about not deploying devices that cannot be monitored. We have Zabbix on all pfSense firewalls and it works great.
I agree with you. I make it a point to only deploy MAX versions of the Netgate due to the storage. The lowest spec i would go is the 4200 and must be MAX.
On top of the issues you mentioned, you have to take care about the amount of logging that you do... My 6100s would've died a year into service but because they are running SSDs, i am 2 years in and without any issues. If you are logging heavily or just have lots of I/O to your drive for whatever reason, selecting a Netgate with eMMC is going to cause you lots of pain.Would it be correct to assume that you learned these issues the hard way and have experienced storage failures in the past before switching to only MAX versions?
My main gripe is the complete lack of information, warnings, or disclaimers prior to purchasing and during general usage. and there is no way for a reasonable person to know about the risks with the onboard eMMC storage until it is too late.
-
@andrew_cb said in Another Netgate with storage failure, 6 in total so far:
Would it be correct to assume that you learned these issues the hard way and have experienced storage failures in the past before switching to only MAX versions?
I have been a pfsense user for quite some time. I am on the forums here and on reddit. The countless tales of unreliable eMMC storage is a tale as old as time so i knew that once i was going the MSP route i knew based on other users' experiences of what not to do.
Should there be a warning in the marketing? I don't know...eMMC may work really well depending on the deployment. Arista 7050CX3 switches have eMMC storage. Enterprise-grade vendor putting in crappy storage. Then again, there isn't heavy writing to the storage on a switch but I am just trying to illustrate to you that putting these parts in a networking device isn't uncommon. As i mentioned, i have shoved 1100s in a corner at a cafe and no issues for years. I also tune the logging down significantly.
-
@andrew_cb it’s not a product page but I think you’re asking for https://www.netgate.com/supported-pfsense-plus-packages
FWIW I don’t recall that we’ve ever had storage failure at any of our clients. Obviously, situations/setups can differ.
Also maybe useful for readers:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html -
@andrew_cb said in Another Netgate with storage failure, 6 in total so far:
My main gripe is the complete lack of information, warnings, or disclaimers prior to purchasing and during general usage. and there is no way for a reasonable person to know about the risks with the onboard eMMC storage until it is too late.
Very true.
To reuse (not yours) words, withing 10 years, it will be known that "emmc" isn't the best choice for very write active OSes. The emmc will probably join realtek on the "don't use these - period" list.Btw, what is general usage ?
In the past, for a firewall, it was this (example) :Using username "root". Authenticating with public key "rsa-key-20230516-pfsense" Passphrase for key "rsa-key-20230516-pfsense": Netgate 4100 - Serial: 2014221462 - Netgate Device ID: e57dfbeef5a2527a *** Welcome to Netgate pfSense Plus 24.11-RELEASE (amd64) on pfSense *** Current Boot Environment: 24.11-Release Next Boot Environment: 24.11-Release WAN (wan) -> ix3 -> v4/DHCP4: 192.168.10.4/24 v6/DHCP6: 2a01:beef:907:a600:92ec:77ff:fe29:392a/64 LAN (lan) -> igc0 -> v4: 192.168.1.1/24 v6/t6: 2a01:beef:907:a6eb:92ec:77ff:fe29:392c/64 IDRAC (opt1) -> igc2 -> v4: 192.168.100.1/24 PORTAL (opt2) -> igc1 -> v4: 192.168.2.1/24 VPNS (opt3) -> ovpns1 -> v4: 192.168.3.1/24 0) Logout / Disconnect SSH 9) pfTop 1) Assign Interfaces 10) Filter Logs 2) Set interface(s) IP address 11) Restart GUI 3) Reset admin account and password 12) PHP shell + Netgate pfSense Plus tools 4) Reset to factory defaults 13) Update from console 5) Reboot system 14) Disable Secure Shell (sshd) 6) Halt system 15) Restore recent configuration 7) Ping host 16) Restart PHP-FPM 8) Shell Enter an option:and from then on the system was idle - doing close to nothing (edit : wrong : it makes stats in the background)
These days, the new normal (example) :
and some of use wonder who picked the colors ... me, I wonder where and how all this info is stored.
Before, with our extreme dumb ISP router with 16 Kbytes of (bios ?) vram, I didn't bother. That router didn't contain any or very few settings and what the heck, the ISP replaces it after one phone call.
But I didn't have these sophisticated stats neither.It all boils down to : what where when is all this backed up ? Where is it stored ?
Look for it, and you'll see the time stamps of all those files, their sizes ... and then you start to dig it : "that is the price to pay" these days : useful, less or pure gadget, it all needed megas to store it's stuff.
That stuff gets rewritten. All the time.And yeah, being here on this forum for a while, and you know this :
You want a Netgate device with hot swap-able dual (because raid 1 !) old iron seagate red label plate based drives .... Like my NAS. No SSD newtech drives which forces me to count write cycles.
Ok, it will consume 0,10 Kwh for sure.
So, ok, plan B : where is it stored ? And can I replace it without doing SMD like soldering ?
Maybe the real question : is it reparable ?
(edit : I also want a double power unit - as all my servers - re edit : wait : Dual HA WAN/LAN ?!).Anyway, @andrew_cb, keep us posted, as I said else where : you can probably add/replace that broken eemc. You wind up having the MAX, so you can throw your 4100 back in business and come back over ... 10 years ?
-
G Gertjan referenced this topic
-
Besides the other points raised about eMMC wear caused by logging and/or the data archiving of the fancy Dashboard graph widgets, another thing to consider is that if you have a ZFS install you are automatically going to experience much more background disk writes from ZFS as compared to the old UFS.
ZFS makes regular writes to the disk as part of its normal operation. And it makes quite a bit more of those than UFS does. That's where the resiliency of ZFS comes from. But that resiliency has a price on eMMC or cheaper SSD devices. Just ZFS by itself probably adds a small incremental boost to eMMC wear, but combine that with extensive application and graph widget logging and things can escalate fast.
-
@bmeeks said in Another Netgate with storage failure, 6 in total so far:
Besides the other points raised about eMMC wear caused by logging and/or the data archiving of the fancy Dashboard graph widgets, another thing to consider is that if you have a ZFS install you are automatically going to experience much more background disk writes from ZFS as compared to the old UFS.
ZFS makes regular writes to the disk as part of its normal operation. And it makes quite a bit more of those than UFS does. That's where the resiliency of ZFS comes from. But that resiliency has a price on eMMC or cheaper SSD devices. Just ZFS by itself probably adds a small incremental boost to eMMC wear, but combine that with extensive application and graph widget logging and things can escalate fast.
So just enabling some basic logging/dashboard features, combined with ZFS (which has been the default filesystem for a while now) is enough to shorten the lifespan of a Netgate with eMMC storage?
How is someone supposed to know about these issues? (@bmeeks I know you're not a Netgate employee)
Reading through an example of eMMC longevity calculations from here yields some concerning numbers:
Workload description 84% Sequential write, 16%Random write Chunk Size IOs Distribution: 30%: 4KB, 27%: 16KB, 42%: Mix of 8KB, 32KB-256KB, 1%: 512KB eMMC Cache on specific eMMC device specs (from datasheet): MLC device physical capacity = 0.0074(TB) for 8GB device endurance cycle = 3000 for MLC Write Amplication Factor (WAF): WAF = 4.5 (estimated from the workload description above with simulation) TBW = physical capacity * endurance cycle / WAF TBW = 0.0074(TB) * 3000(cycles) / 4.5(WAF) ~= 5.0 TBW 5.0 Terabytes is the total amount of data written to the device during its lifetime of use, depending on the workload.Taking the 5.0TB writable and calculating for 3 years of life gives us:
5.0TB / 1095 days = 4.57GB per day.
4.57GB / 24 hours = 190MB per hour.
190MB / 60 minutes = 3.17MB per minute.
3.17MB per minute = 53Kb per second.So you must write no more than an average of 53Kb per second in order to get 3 years of life, 106Kbps to last 2 years, and 159KBps to last 1 year.
Being generous and increasing the numbers 10-fold still only leaves a maximum of 530Kb per second to last 3 years.
The purpose of buying a device instead of building your own is that the manufacturer is supposed to take care of choosing the correct components so that you do not have to. If the expectation is that a user must spend countless hours and years of testing and research in order to understand how to get the device to work properly, then it is far easier and more cost-effective to simply purchase a competitor's device and just pay the yearly fees.
Now I may sound bitter (and I am at the moment), but I genuinely want to provide feedback to Netgate that they can use to make changes so that others do not experience the same challenges that I am currently experiencing with device failures (and the grim prospect that more will likely fail).
With all the complexities of changing the behavior of pfSense and various packages, I believe the easiest way to avoid future problems is to either change to a more robust storage medium in the BASE versions or at least make the limitations of eMMC storage abundantly clear.
The product pages make no differentiation between the BASE and MAX versions other than increased capacity. eMMC and NVMe are mentioned, but I suspect that very few people are aware of the critical differences. After all, if Netgate chose eMMC and no further details are given, then it must not be important, right? A blurb that explains eMMC vs NVME, along with a table listing use cases/packages where the MAX version is recommended, would be a huge benefit to both users and Netgate and a great way to upsell the MAX version.
-
@andrew_cb said in Another Netgate with storage failure, 6 in total so far:
With all the complexities of changing the behavior of pfSense and various packages, I believe the easiest way to avoid future problems is to either change to a more robust storage medium in the BASE versions or at least make the limitations of eMMC storage abundantly clear.
I definitely agree with you here. eMMC technology was initially way cheaper than NVMe drives, and that likely drove the decision to use that more than any other factor. I think the disparity has decreased some with the proliferation of NVMe choices now, and NVMe seems a more solid and long-term reliable solution.
-
I agree with all your points to be frank. I personally held the belief that offering a BASE version of any model up to the 6100 is silly especially when viewed from the price perspective where its ~100 bucks difference. For an extra $100 you get better storage. If the price is negligible than why offer the base to begin with.
In my opinion, you should pay more for better performance in terms of cpu or memory. Look at the 4200 as an example. Base and MAX is the exact same in terms of specs except storage. I find it hard to believe the nvme is an additional $100 but even if true, i still have the same performance. I can see why there are people who would select the Base sku but Netgate is doing more harm than good when a cheap unreliable drive goes bad in their own flagship product which happens quite often if forum posts are to be believed. Just offer the Max version which in reality is the Base version. Thats it. One Sku per product. They do it with the 8200Depending on the deployment, i would go white box. Grab the pfsense+ license which is very cost effective and deploy a Dell or HP 1RU system which would be far more reliable , more robust.
-
Imagine you work for a busy company that is trying a different brand of delivery trucks for its fleet that operates 24/7. The delivery trucks work well and employees like them, so the company buys many more over the next few years. At first, there were no available options, but the brochure for recent models lists two axle options: BASE or MAX. The only difference mentioned is that the BASE axle is 6-lug and the MAX axle is 8-lug.
Recently, the factory-equipped axles have begun failing much sooner than those of other truck brands and previous delivery trucks you have owned. Now, a sixth truck is stuck at the side of the road waiting for a tow, missing another important delivery and losing your customer's confidence.
You begin researching axle failures and this truck model. You find that the BASE model tires are similar to the axles used on passenger vehicles. They are fine for driving around town with light loads, but highway driving, adding additional equipment, or carrying additional weight causes the axles to wear internally and fail prematurely. Changing to the MAX axle requires a truck to be out of service for 2 days. You find that other companies using these trucks are having the same problem, and the manufacturer even recommends removing the spare tire, jack, radio, passenger seat, bumpers, and mud flaps to reduce weight and extend the life of the BASE axle.
You might think, "Ridiculous!" One of the main reasons we bought these trucks is that they support a wide variety of aftermarket accessories that allow the trucks to be customized to significantly improve their functionality, as shown in the glossy brochures.
You wonder, "Since these are sold as delivery trucks, why would the manufacturer use inferior axles without making it clear that the BASE axle option is what most customers will need to use the trucks for anything but the lightest of tasks?"
The MAX model was a $1000 option that seemed unnecessary at the time, but each premature axle failure costs you $10,000 in towing charges, labor, equipment rentals, customer goodwill, and vehicle repairs. Replacing the axles before they fail will cost $7000 of lost revenue, parts, and labor per truck. You look out your office at the 40 trucks in your loading yard and wonder how you will get through this situation.
