eMMC Write endurance

Cabledude

@JonathanLee said in eMMC Write endurance:

Disable logging that is not needed all the time, and or use a usb drive and set the system to log to it..

Thanks Jonathan. However this issue is about a fresh install to USB storage, bypassing the eMMC. Here, the installation fails i.e. RAM appears to be full during file copying to USB drive.

stephenw10

Is it consistently failing at the same point?

Cabledude

@stephenw10
No is failed at two different points. But installation finished successfully on the 3rd try, 2nd try with the external SSD as target drive.

Strangely, in the console startup of the new installation, the unit requires to change the default admin password from pfsense to a user chosen password. I've never seen that before.

Main thing is I got it working now.

So basically adding the SSD turns an EOL SG-1100 (EOL due to broken eMMC) back into a MINT unit, if I understand this correctly.

Cabledude

@stephenw10 said in eMMC Write endurance:

To boot from USB every time you need to change the bootcmd uboot env to run usbboot;

So at the Marvell>> prompt:
setenv bootcmd='run usbboot;'
saveenv
reset

Thanks for this by the way, yet at this point I don't understand what it's for. After the installation finished I unplugged the net installer thumb drive and swapped the SSD to the 1100's USB3 port and it is consistently booting from the SSD now.

Cabledude

@stephenw10

I am continuing this in a separate topic as I am running into more issues and I don't want to pollute this eMMC topic more than I already have.

michmoor

@keyser Dont know what i was searching for when i found this post but wow...very timely.
eMMC is proving to be a very disastrous move based on frequent topics here and on reddit.

@keyser you called it out over 2 years ago about how this will be problematic.

Cabledude

@michmoor
Very good of you to call in. Which model do you have?
Just want to say that I have good experience with an SG-1100 running on external SSD. The eMMC is largely gone but the unit is performing flawlessly.

andrew_cb

@michmoor said in eMMC Write endurance:

@keyser Dont know what i was searching for when i found this post but wow...very timely.
eMMC is proving to be a very disastrous move based on frequent topics here and on reddit.

@keyser you called it out over 2 years ago about how this will be problematic.

Yep, it seems that @keyser was onto this issue 2 years ago, and there are even other threads from 3 and 4 years ago as well.

Crazy that no eMMC endurance figures are given, no emmc-utils package included in base, no eMMC or SMART monitoring enabled by default, default logging firewall logging settings that can cause a lot of writes, no obvious mention about using ramdisks, insufficient warnings about packages, no mention of any limitations on the product pages... and somehow the only response is to blame the user...

Oh, and the eMMC in the 4200 CANNOT be monitored, and the emmc-utils package is NOT available in the CE repo!

Nearly 20k views in this sub-forum alone and another 80k from my Reddit PSAs about storage failure. Plus all the other posts here and on the forum. Many users that read my PSA were unaware that they needed to monitor their storage health, and many discovered that their eMMC was significantly or critically worn.

eMMC Write endurance: 12.3k views, 72 posts
Another Netgate with storage failure, 6 in total so far: 6.2k views, 81 posts
6100 Failed eMMC replaced with NVme but now no longer reboots: 1.1k views, 21 posts

This issue has been swept under the rug for far too long. Too many users have been blamed for 'misusing' their Netgate device due to some arbitrary criteria. The problem is undeniable. How much more is needed before Netgate commits to specific actions to address storage failures?

Cabledude

@andrew_cb the topics you are listing are quite problematic. Surely a quality company, Netgate must be eager to assist the customers involved I suppose?

michmoor

@andrew_cb seems maybe they’re still looking into it or hoping the problem is goes away?

stephenw10

We are looking into it.

JasonTracy

@stephenw10 I've got a 1100 that I talked my parents into buying 2 years ago. They're retired, don't use a ton of Internet & I've not installed any packages with high writes.

Just checked my eMMC on it, and it is showing 50-80% wear:

eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x05
eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x08
eMMC Pre EOL information [EXT_CSD_PRE_EOL_INFO]: 0x01

They're 1,700 miles away, and I'll be visiting them in April. I'd really like to know the best way to remediate this by that point.

I understand I'm not a big business user, but I went the path of buying Netgate hardware precisely because I expected it to be more reliable than a TP-Link router or whatever SFF PC I could build off Amazon. Being able to remotely support my parent's Internet from several states away is important to me.

If there is documentation on the best way to replace the eMMC, or add other storage to the 1110 in a supported way, I'd love to see it. Thank you!

stephenw10

Re-installing as UFS with RAM disks reduces drive writes to almost zero. Enabling RAM disks with ZFS reduces it significantly. If you're not running any hungry packages you should be able to do that.

Otherwise it's possible to install and run from USB on the 1100.

JasonTracy

@stephenw10 wilco, thank you.

jared.silva

@JasonTracy https://forum.netgate.com/topic/196372/migrating-netgate-1100-from-emmc-to-usb-flash-storage-to-keep-it-fit

brodng

I’ve got a base 4200 that’s about 6 months old. Was running pfBlocker. I have disabled it for now after reading about the eMMC storage issues.

Is it possible to check the remaining eMMC usable life on a base 4200?
How do I disable all logging on the 4200? Just until I can upgrade the 4200 to an NVMe.
Are there recommended NVMe drives that are know to work well with the 4200?

Thank you

michmoor

@brodng Just throwing in my 2cents here regarding pfblockerNG.
I am running it on a SG1100. I do have Global IP Logging enabled but i make sure my list only applies to traffic leaving my LAN (outbound). Assuming you don't have anything suspect on your LAN you shouldn't get that many hits or any at all.

If you are using DNSBL that's a bit tricky because you do need the logging. I made it a point to disable DNSBL because of the overall usefulness of it in general on pfsense. I have per-vlan blocking lists that i need to apply for different use cases and pfblocker cant do it so i use pihole.

Lastly the default deny rules. I created a rule called "Global Deny" that's at the end of my firewall list and i don't set logging to it. Tends to take care of any denies being written to disk further conserving the life of the drive.

SteveITS

@brodng First off, I wouldn't panic.

Is it possible to check the remaining eMMC usable life on a base 4200?

No, it isn't on that model. I think that's the only one though? (for others: https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html#emmc)

@brodng said in eMMC Write endurance:

How do I disable all logging

See a list of ideas at:
https://forum.netgate.com/topic/195879/netgate-2100-life-expectancy/8

stephenw10

@SteveITS said in eMMC Write endurance:

I think that's the only one though?

The RCC-VE models also used USB attached eMMC, so: 2440, 4860, 8860
Never really saw any significant eMMC issues on those though.

Some of the early 1100s used an eMMC version that the util couldn't read.

brodng

@michmoor said in eMMC Write endurance:

@brodng Just throwing in my 2cents here regarding pfblockerNG.
I do have Global IP Logging enabled but i make sure my list only applies to traffic leaving my LAN (outbound).

When I re-enable it I’ll make sure it’s setup this way.

If you are using DNSBL that's a bit tricky because you do need the logging.

Not using it either. Thought it would be difficult to maintain and setup accurately.

Lastly the default deny rules. I created a rule called "Global Deny" that's at the end of my firewall list and i don't set logging to it. Tends to take care of any denies being written to disk further conserving the life of the drive.

Will definitely do this, good idea.