eMMC Write endurance

andrew_cb

@keyser Dont know what i was searching for when i found this post but wow...very timely.
eMMC is proving to be a very disastrous move based on frequent topics here and on reddit.

@keyser you called it out over 2 years ago about how this will be problematic.

Yep, it seems that @keyser was onto this issue 2 years ago, and there are even other threads from 3 and 4 years ago as well.

Crazy that no eMMC endurance figures are given, no emmc-utils package included in base, no eMMC or SMART monitoring enabled by default, default logging firewall logging settings that can cause a lot of writes, no obvious mention about using ramdisks, insufficient warnings about packages, no mention of any limitations on the product pages... and somehow the only response is to blame the user...

Oh, and the eMMC in the 4200 CANNOT be monitored, and the emmc-utils package is NOT available in the CE repo!

Nearly 20k views in this sub-forum alone and another 80k from my Reddit PSAs about storage failure. Plus all the other posts here and on the forum. Many users that read my PSA were unaware that they needed to monitor their storage health, and many discovered that their eMMC was significantly or critically worn.

eMMC Write endurance: 12.3k views, 72 posts
Another Netgate with storage failure, 6 in total so far: 6.2k views, 81 posts
6100 Failed eMMC replaced with NVme but now no longer reboots: 1.1k views, 21 posts

This issue has been swept under the rug for far too long. Too many users have been blamed for 'misusing' their Netgate device due to some arbitrary criteria. The problem is undeniable. How much more is needed before Netgate commits to specific actions to address storage failures?

Cabledude

@andrew_cb the topics you are listing are quite problematic. Surely a quality company, Netgate must be eager to assist the customers involved I suppose?

michmoor

@andrew_cb seems maybe they’re still looking into it or hoping the problem is goes away?

stephenw10

We are looking into it.

JasonTracy

@stephenw10 I've got a 1100 that I talked my parents into buying 2 years ago. They're retired, don't use a ton of Internet & I've not installed any packages with high writes.

Just checked my eMMC on it, and it is showing 50-80% wear:

eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x05
eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x08
eMMC Pre EOL information [EXT_CSD_PRE_EOL_INFO]: 0x01

They're 1,700 miles away, and I'll be visiting them in April. I'd really like to know the best way to remediate this by that point.

I understand I'm not a big business user, but I went the path of buying Netgate hardware precisely because I expected it to be more reliable than a TP-Link router or whatever SFF PC I could build off Amazon. Being able to remotely support my parent's Internet from several states away is important to me.

If there is documentation on the best way to replace the eMMC, or add other storage to the 1110 in a supported way, I'd love to see it. Thank you!

stephenw10

Re-installing as UFS with RAM disks reduces drive writes to almost zero. Enabling RAM disks with ZFS reduces it significantly. If you're not running any hungry packages you should be able to do that.

Otherwise it's possible to install and run from USB on the 1100.

JasonTracy

@stephenw10 wilco, thank you.

jared.silva

@JasonTracy https://forum.netgate.com/topic/196372/migrating-netgate-1100-from-emmc-to-usb-flash-storage-to-keep-it-fit

brodng

I’ve got a base 4200 that’s about 6 months old. Was running pfBlocker. I have disabled it for now after reading about the eMMC storage issues.

Is it possible to check the remaining eMMC usable life on a base 4200?
How do I disable all logging on the 4200? Just until I can upgrade the 4200 to an NVMe.
Are there recommended NVMe drives that are know to work well with the 4200?

Thank you

michmoor

@brodng Just throwing in my 2cents here regarding pfblockerNG.
I am running it on a SG1100. I do have Global IP Logging enabled but i make sure my list only applies to traffic leaving my LAN (outbound). Assuming you don't have anything suspect on your LAN you shouldn't get that many hits or any at all.

If you are using DNSBL that's a bit tricky because you do need the logging. I made it a point to disable DNSBL because of the overall usefulness of it in general on pfsense. I have per-vlan blocking lists that i need to apply for different use cases and pfblocker cant do it so i use pihole.

Lastly the default deny rules. I created a rule called "Global Deny" that's at the end of my firewall list and i don't set logging to it. Tends to take care of any denies being written to disk further conserving the life of the drive.

SteveITS

@brodng First off, I wouldn't panic.

Is it possible to check the remaining eMMC usable life on a base 4200?

No, it isn't on that model. I think that's the only one though? (for others: https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html#emmc)

@brodng said in eMMC Write endurance:

How do I disable all logging

See a list of ideas at:
https://forum.netgate.com/topic/195879/netgate-2100-life-expectancy/8

stephenw10

@SteveITS said in eMMC Write endurance:

I think that's the only one though?

The RCC-VE models also used USB attached eMMC, so: 2440, 4860, 8860
Never really saw any significant eMMC issues on those though.

Some of the early 1100s used an eMMC version that the util couldn't read.

brodng

@michmoor said in eMMC Write endurance:

@brodng Just throwing in my 2cents here regarding pfblockerNG.
I do have Global IP Logging enabled but i make sure my list only applies to traffic leaving my LAN (outbound).

When I re-enable it I’ll make sure it’s setup this way.

If you are using DNSBL that's a bit tricky because you do need the logging.

Not using it either. Thought it would be difficult to maintain and setup accurately.

Lastly the default deny rules. I created a rule called "Global Deny" that's at the end of my firewall list and i don't set logging to it. Tends to take care of any denies being written to disk further conserving the life of the drive.

Will definitely do this, good idea.

brodng

@SteveITS said in eMMC Write endurance:

@brodng First off, I wouldn't panic.

I’m not panicking yet : ) Seems inevitable that the storage will have an issue. Would like to get ahead of the problem. I work remote so I would rather schedule the down time.

Is it possible to check the remaining eMMC usable life on a base 4200?

No, it isn't on that model. I think that's the only one though? (for others: https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html#emmc)

That’s a downer : ( Is it not technically feasible with the base 4200 or could it be added via a firmware update? I’m going to put in an SSD but would be great if this could be added. May help to keep people from panicking.

@brodng said in eMMC Write endurance:

How do I disable all logging

See a list of ideas at:
[https://forum.netgate.com/topic/195879/netgate-2100-life-expectancy/8](
https://forum.netgate.com/topic/195879/netgate-2100-life-expectancy/8

Thank you will check this out.

Is there a an official guide to upgrade the base 4200 with an NVMe drive? Found a YouTube video but it wasn’t very clear on how to clear the eMMC.

Didn’t realize the base model was so limited due to the eMMC compared to the Max. Figured I could upgrade later if needed. Well, now I needed, lol.

SteveITS

@brodng https://docs.netgate.com/pfsense/en/latest/solutions/netgate-4200/m-2-nvme-installation.html

JonathanLee

I would do the flash drive make a backup also when you visit your parents.

install and get it running backup config on one flash drive.
do the same process again and label it as backup flash drive.

Just roll with it stuff goes out sometimes. The option to use a usb drive helps alot

You could also go this route and create ln (liker files) that redirect my log files for snort and squid to another drive, could be usb or anything else.