Another Netgate with storage failure, 6 in total so far

[andrew_cb]

I was directed to this post from 2021 about pfSense with ZFS writing 14-20GB per day.
This got me curious, so I checked our fleet and found that the range is 3-28GB per day, with an average of 18GB. That's about 2-3 DWPD.

This works out to an average of 6.5TB per year, and 19.7TB after 3 years.

Assuming each cell in the eMMC storage can be written 3000 times, 8GB can handle 24TB of writes before the storage is worn out.
This is the best-case scenario. Given the background filesystem behavior plus internal drive processing, writing 20TB of data of a 24TB maximum seems "good".
A 32GB eMMC could handle 96TB TBW, which would be 52GB per day maximum, and probably about 43GB (80%) per day real-world.

Comparatively, a 250GB Samsung 860 EVO has a TBW value of 150TB, which works out to 82GB per day for 5years. Halving that to approximate a 120GB drive gives a TBW of 75TB and 41 GB per day for 5 years. This fits with the typical 0.3 DWPD rating of most SSDs.

Clearly, a larger storage drive, whether eMMC or SSD, greatly increases the lifespan of the storage.
If eMMC itself is not the problem, then the small 8GB appears to be a significant factor. Increasing the onboard eMMC storage to 16 or 32GB would greatly increase its lifespan and significantly reduce the number of premature device failures.

[andrew_cb]

Good morning @jwt . Just checking to see if Netgate is still looking into this and if you have any further comments to add.

[dstaylor]

My 2 1/2-year-old 6100 is showing over 100% usage, so I've ordered an NVMe to install. I did not have any logging-intensive applications running, so it's frustrating. The cheaper mini PCs I bought for other family members' houses are just chugging along with their SSDs, more memory, and faster CPUs.

Is there a guide on installing an NVMe and reinstalling pfSense on that drive out there somewhere? I think my drive arrives on Sunday, so that will be one of my weekend tasks.

[SteveITS]

@dstaylor There isn't a guide for the 6100. Try to find some forum threads on that since I seem to recall discussions. It's possible it's not an NVMe drive, too, I don't know, but some other models use SATA. Just in case you want to check before opening it.

[dstaylor]

@SteveITS I did see mention of NVMe for the 6100, and even a couple of pictures others had posted showing the drive installed. I just have not found any guide on doing an install and making sure it selects the new drive.

Guess I'll have my VM of 2.7.2 standing by in case the whole thing dies.

[stephenw10]

It is NVMe in the 4100/6100/8200. It's not hard to fit if you have any experience assembling PCs.

[andrew_cb]

@dstaylor I feel your pain. The storage on your 6100 should not wear out so quickly!

Netgate staff say that it is not possible/supported/recommended for an end-user to install an SSD in the 4100 and 6100, so there is no official documentation.

"It's a time of purchase upgrade option."

"Upgrading, replacing, or adding, an SSD to the 6100 is not supported and may void your warranty. We do not provide an SSD for sale that can be added nor do we provide instructions on how to make the replacement as the entire system is very tightly fit into the case and you can break connections opening and re-securing the enclosure. It happened to a number of our staff during early testing and I have personally confirmed the tight fit on production models of the 4100 (which is basically the same form factor). At present time there are no uses for the additional ports/slots on the 6100."

"For those who are considering upgrading the base model this is not supported by TAC - and may void the warranty (if there is any remaining)."

Why does Netgate misleadingly advertise the slots when they have no use and the user is not supposed to touch them? Well, that is a real puzzler!

So if your Netgate device is 31 to 365 days old, you are SOL. Fortunately, this limitation does not apply to you since your device is out of warranty.

The actual SSD installation process is easy:

• Remove the rubber feet and then remove all 8 torx screws (2 front, 2 back, 4 under the feet).
• Remove the plastic filler panel that was held in by the screws.
• Gently separate the top and bottom half of your 6100.
• Install the M.2 NVMe drive into the slot.
• Carefully put the top and bottom halves together - pay attention to the LED lights and the 3 plastic "shrouds" that bend around the circuit board.
• Reinstall the screws and attach the feet.

The instructions from the 4200 are pretty similar and cover the software re-installation process.

This video of opening a 6100 should be helpful.

[andrew_cb]

@jimp @stephenw10 @kphillips @marcosm @cmcdonald Would any of you care to comment on this thread? With 4.3k views here and over 60k views on Reddit, I hope that @jwt's comments do not represent Netgate's official and only response to the issues that have been raised.

[marcosm]

@andrew_cb Some good points have been raised along with actionable suggestions to mitigate the issue. Thanks for the constructive feedback - the issue has our attention.

[dstaylor]

@andrew_cb I installed a KingSpec NVMe 256GB drive from Amazon this morning. I followed the guidance you provided and the procedure to wipe the MMC drive in the documents.

It's back up and running now. The only thing that seems to be different is I no longer have the slowly flashing diamond that shows "boot complete/ready". I do get the flashing square during boot, and the orange circle when I put it in standby. I checked that none of the "light pipes" were bent, so that is a head-scratcher. Now when it's running all the LEDs are dark. I took away the memory disk I had implemented to try and prolong the mmc drive because it filled up during my reinstall and package reload.

Best I can tell it's all good except for the LED issue.

[stephenw10]

Hmm, odd. Try running: pfSense-led.sh ready

[dstaylor]

@stephenw10 I'm running that command from where? And what output am I expecting?

[dstaylor]

@dstaylor said in Another Netgate with storage failure, 6 in total so far:

@stephenw10 I'm running that command from where? And what output am I expecting?

I found it down in /usr/local/sbin. I ran the command but there is no change.

[stephenw10]

Hmm, so just no LEDs? You can try setting any of the other states:

pfSense-led.sh
usage:
pfSense-LED booting
pfSense-LED ready
pfSense-LED update [1|0]
pfSense-LED updating

It should update the LEDs accordingly. 'ready' is the normal state after booting.

[dstaylor]

@stephenw10 I pulled the box apart again. It was a keyboard/chair interface issue; the "flap" around the lighthood had folded over a little bit and was blocking the LED.

Now the LEDs are working as expected.

[dnavas]

Please do follow the advice of zero'ing out your emmc when installing an ssd. I did not (I'm not sure that advice was even there at the time, I don't recall where I got the pointers to install my own ssd in the first place, but other than the special requirements on the type of ssd supported, I don't remember it being a particularly difficult install physically), and it worked fine until it didn't. I appreciate the help of staff here to guide me out of that predicament.

All of this discussion does make me wonder where I put my original pfsense media. It would seem to be a really bad idea not to have a recovery image in case it's needed (gulp).

[SteveITS]

@dnavas said in Another Netgate with storage failure, 6 in total so far:

bad idea not to have a recovery image

You can get the Netgate Installer from the store (free). It will download the latest version when you run it. Actually I think they added, or were talked about adding, a version selection.

https://docs.netgate.com/pfsense/en/latest/install/index.html