Concerns and feedback about storage lifetime wearout on Netgate devices

[andrew_cb]

Within the past 12 months, I've experienced failures of Netgate 4100, 6100, and 7100 firewalls all with the same symptom: no storage device is detected. I'm now fairly confident that the onboard eMMC storage failed in all 5 devices.

Overall, Netgate hardware has been reliable except for storage device failure. However, having seen numerous Sonicwall and Sophos devices run for 10+ years with security and logging services enabled with no storage failures, it is concerning to experience storage failure on several Netgate devices and to then read that it seems to be a someone common issue.

I've seen various mentions of needing to "take it easy" on the onboard eMMC. This is also mentioned on the documentation for some packages, but generally there are no guidelines on how much of an impact packages have on storage lifetime.

https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html
https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-writes.html

Concerned, I looked at the storage health of a few production device:. An 8200 showed nearly 32TB written but at 100% health, and 2 6100 both showed 0x0b Type A and B values (100-110% lifetime used), and 0x01 Pre-EOL (less than 80% of reserved blocks used).

For our new 4200 units, I discovered that it's not possible to monitor the onboard storage health:
https://redmine.pfsense.org/issues/15978

With that in mind, I have some thoughts and feedback for the Netgate team that I hope will be helpful:

• The SMART status widget is not added to the dashboard by default.

• Monitoring eMMC requires manually checking using the CLI, which is extremely non-intuitive (unless you specifically try monitoring the storage health and investigate how to get it working, there is nothing to even make a user think about it).

• The mmc package should be included with pfSense and/or Netgate devices, and the SMART status page should at least include a link to the documentation. Or, the mmc package should be added to System Packages with a GUI page for running it, as this would be a simple yet hugely beneficial improvement.

• It's not even possible to monitor the onboard eMMC storage of the 4200. This, coupled with no warnings on the product page, means a user will not be aware of the risks with the BASE version.

• The product pages have no mention or warning regarding the limitations of eMMC storage compared to SSD or NVME. The impression is that the MAX versions simply get you a larger storage drive.

• The 4200 and 7100 documentation covers adding an M2 drive, but the 4100, 6100, and 8200 documentation do not. Can M2 storage be installed in these models by the user?

• No write lifetime specification is given for the BASE or MAX versions, so a user can't make an informed decision on eMMC vs SSD before purchasing, and afterwards cannot easily estimate the storage lifetime

• Unless a user goes digging through the documentation and stumbles upon the few pages, they will be unaware of storage liftime issues.

• The relevent pages are called "Troubleshooting Disk Lifetime" and "Troubleshooting Disk Writes" - under normal circumstances a user would never come across these pages, especially not before purchasing a Netgate device.

• The page confusingly states, "Contrary to popular rumors, the vast majority of modern flash storage found in SSD and eMMC drives is very resilient." How does this apply to the onboard eMMC installed in Netgate devices?

[Gertjan]

@andrew_cb

Humm.
Nice write up.
Most of what you have written existed already in my head, as I used these kind of questions when I had decide what type of device to buy, or to be more exact : what storage options ?

pfSense, when used without any addition - read : packages - runs just fine on any device, from the 1100 to he 8x00, and the only difference will be ! how many NICs, and the overall throughput.

Anyway, for the reasons you've shown, I bought a 4100, and I had the option : base or max, I went for the max, and you've summed up the real (for me) reasons. Which tells me that the mmc can be replaced (added ?) with a NVME drive rather easily on that (4100) device.

Still, I'm not using the packages that create huge amounts of write cycles.
I do use pfBlockerng with a minimal DNSBL list.

And you're right, when we go out buying that no-name router, we don't look for "what RAM" or what storage it has. But pfSense brings new issues : suddenly, there are a lot of things to click and activate.
And these new functionalities often start to 'do' things on the router (pfSense) and they log a lot so nice charts and images can be shown (and pfSense isn't even a pi-hole yet).

edit :
We all need to install and use pfSense for a while to discover what device (hardware) we actually need.
And while using pfSense, our needs change.

[andrew_cb]

I had another 4100 die yesterday.
I realized this is the "pfSense Software" forum so I made a new post in the "Official Netgate Hardware" forum

https://forum.netgate.com/topic/195990/another-netgate-with-storage-failure-6-in-total-so-far

[elvisimprsntr]

Not even sure why eMMC is even used on an enterprise class mission critical device other than $

1. eMMC approx 3000 P/E cycles
2. Slow
3. Poor reliability
4. Non-replaceable

[Gertjan]

@andrew_cb

edit : getting old here, I'm rambling, I basically said the following already above ... but read on, there might be hope.

I never opened mine, and I should probably search for a video on Ytube where it's presented.
The question is : as I have 4100, can the emmc be replaced ?

What @elvisimprsntr mentioned crossed my mind when I went trough my 'decision process' when searching for Netgate appliance.
It wasn't really the traffic, back then, as I had only a XDSL 30 Mbit down 2 mbit up, so I could have taken a "1100". But later on I got a fiber 3 Gbit down 1 Gbit up connection .... oops.
But I couldn't really answer : do I use only pfSense and no other packages (gadgets) what so ever "at all" knowing that some of these packages "write a lot". I couldn't answer that question, but I knew that I would, at least, use pfBlockerng.
pfBlockerng, out of the box, with it's default settings, can write a lot, or more.
I didn't want to use the the "let's use a RAM drive" road neither.
So the good old "let's slam more hardware on it" prevails.
So : I went for the 4100 MAX and didn't regret it.

I thought : this extra or other drive must be easy to install, as I don't see Netgate adapting the native 4100 to a MAX by soldering in the bigger drive. It must be a screwdriver-only task.
So, at most, they open it up, rip out the exiting drive and replace it by a bigger one.
Or they add another drive (and disable the original in a BIOS --- is there even a 'PC like BIOS with settings for boot order and so on ?)

Anyway, something tells my you can (have to) do the same

[slu]

@andrew_cb said in Concerns and feedback about storage lifetime wearout on Netgate devices:

I've seen various mentions of needing to "take it easy" on the onboard eMMC. This is also mentioned on the documentation for some packages, but generally there are no guidelines on how much of an impact packages have on storage lifetime.

https://forum.netgate.com/topic/189820/how-do-i-find-out-what-write-continuously-on-my-pfsense-ssd/

[smolka_J]

This post is deleted!