Concerns and feedback about storage lifetime wearout on Netgate devices

andrew_cb

Within the past 12 months, I've experienced failures of Netgate 4100, 6100, and 7100 firewalls all with the same symptom: no storage device is detected. I'm now fairly confident that the onboard eMMC storage failed in all 5 devices.

Overall, Netgate hardware has been reliable except for storage device failure. However, having seen numerous Sonicwall and Sophos devices run for 10+ years with security and logging services enabled with no storage failures, it is concerning to experience storage failure on several Netgate devices and to then read that it seems to be a someone common issue.

I've seen various mentions of needing to "take it easy" on the onboard eMMC. This is also mentioned on the documentation for some packages, but generally there are no guidelines on how much of an impact packages have on storage lifetime.

https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html
https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-writes.html

Concerned, I looked at the storage health of a few production device:. An 8200 showed nearly 32TB written but at 100% health, and 2 6100 both showed 0x0b Type A and B values (100-110% lifetime used), and 0x01 Pre-EOL (less than 80% of reserved blocks used).

For our new 4200 units, I discovered that it's not possible to monitor the onboard storage health:
https://redmine.pfsense.org/issues/15978

With that in mind, I have some thoughts and feedback for the Netgate team that I hope will be helpful:

The SMART status widget is not added to the dashboard by default.
Monitoring eMMC requires manually checking using the CLI, which is extremely non-intuitive (unless you specifically try monitoring the storage health and investigate how to get it working, there is nothing to even make a user think about it).
The mmc package should be included with pfSense and/or Netgate devices, and the SMART status page should at least include a link to the documentation. Or, the mmc package should be added to System Packages with a GUI page for running it, as this would be a simple yet hugely beneficial improvement.
It's not even possible to monitor the onboard eMMC storage of the 4200. This, coupled with no warnings on the product page, means a user will not be aware of the risks with the BASE version.
The product pages have no mention or warning regarding the limitations of eMMC storage compared to SSD or NVME. The impression is that the MAX versions simply get you a larger storage drive.
The 4200 and 7100 documentation covers adding an M2 drive, but the 4100, 6100, and 8200 documentation do not. Can M2 storage be installed in these models by the user?
No write lifetime specification is given for the BASE or MAX versions, so a user can't make an informed decision on eMMC vs SSD before purchasing, and afterwards cannot easily estimate the storage lifetime
Unless a user goes digging through the documentation and stumbles upon the few pages, they will be unaware of storage liftime issues.
The relevent pages are called "Troubleshooting Disk Lifetime" and "Troubleshooting Disk Writes" - under normal circumstances a user would never come across these pages, especially not before purchasing a Netgate device.
The page confusingly states, "Contrary to popular rumors, the vast majority of modern flash storage found in SSD and eMMC drives is very resilient." How does this apply to the onboard eMMC installed in Netgate devices?

Gertjan

@andrew_cb

Humm.
Nice write up.
Most of what you have written exsietd already in my head, as I used these kind of questions when I had decide what type of device to buy, or to be more exact : what storage options ?

pfSense, when used without any addition - read : packages - runs just fine on any device, from the 1100 to he 8x00, and the only difference will be ! how many NICs, and the overall throughput.

Anyway, for the reasons you've shown, I bought a 4100, and I had the option : base or max, I went for the max, and you've summed up the real (for me) reasons. Which tells me that the mmc can be replaced (added ?) with a NVME drive rather easily on that (4100) device.

Still, I'm not using the packages that create huge amounts of write cycles.
I do use pfBlockerng with a minimal DNSBL list.

And you're right, when we go out buying that no-name router, we don't look for "what RAM" or what storage it has. But pfSense brings new issues : suddenly, there are a lot of things to click and activate.
And these new functionalities often start to 'do' things on the router (pfSense) and they log a lot so nice charts and images can be shown (and pfSense isn't even a pi-hole yet).

edit :
We all need to install and use pfSense for a while to discover what device (hardware) we actually need.
And while using pfSense, our needs change.