Another Netgate with storage failure, 6 in total so far

andrew_cb · Jan 30, 2025, 4:48 PM

@jimp @stephenw10 @kphillips @marcosm @cmcdonald Would any of you care to comment on this thread? With 4.3k views here and over 60k views on Reddit, I hope that @jwt's comments do not represent Netgate's official and only response to the issues that have been raised.

marcosm · Feb 3, 2025, 5:29 PM

@andrew_cb Some good points have been raised along with actionable suggestions to mitigate the issue. Thanks for the constructive feedback - the issue has our attention.

dstaylor · Jan 31, 2025, 4:46 PM

@andrew_cb I installed a KingSpec NVMe 256GB drive from Amazon this morning. I followed the guidance you provided and the procedure to wipe the MMC drive in the documents.

🔒 Log in to view

It's back up and running now. The only thing that seems to be different is I no longer have the slowly flashing diamond that shows "boot complete/ready". I do get the flashing square during boot, and the orange circle when I put it in standby. I checked that none of the "light pipes" were bent, so that is a head-scratcher. Now when it's running all the LEDs are dark. I took away the memory disk I had implemented to try and prolong the mmc drive because it filled up during my reinstall and package reload.

Best I can tell it's all good except for the LED issue.

stephenw10 · Jan 31, 2025, 5:05 PM

Hmm, odd. Try running: pfSense-led.sh ready

dstaylor · Jan 31, 2025, 5:19 PM

@stephenw10 I'm running that command from where? And what output am I expecting?

dstaylor · Jan 31, 2025, 5:31 PM

@dstaylor said in Another Netgate with storage failure, 6 in total so far:

@stephenw10 I'm running that command from where? And what output am I expecting?

I found it down in /usr/local/sbin. I ran the command but there is no change.

stephenw10 · Jan 31, 2025, 6:24 PM

Hmm, so just no LEDs? You can try setting any of the other states:

pfSense-led.sh
usage:
pfSense-LED booting
pfSense-LED ready
pfSense-LED update [1|0]
pfSense-LED updating

It should update the LEDs accordingly. 'ready' is the normal state after booting.

dstaylor · Jan 31, 2025, 7:16 PM

@stephenw10 I pulled the box apart again. It was a keyboard/chair interface issue; the "flap" around the lighthood had folded over a little bit and was blocking the LED.

Now the LEDs are working as expected.

dnavas · Feb 1, 2025, 10:18 AM

Please do follow the advice of zero'ing out your emmc when installing an ssd. I did not (I'm not sure that advice was even there at the time, I don't recall where I got the pointers to install my own ssd in the first place, but other than the special requirements on the type of ssd supported, I don't remember it being a particularly difficult install physically), and it worked fine until it didn't. I appreciate the help of staff here to guide me out of that predicament.

All of this discussion does make me wonder where I put my original pfsense media. It would seem to be a really bad idea not to have a recovery image in case it's needed (gulp).

SteveITS · Feb 1, 2025, 11:10 PM

@dnavas said in Another Netgate with storage failure, 6 in total so far:

bad idea not to have a recovery image

You can get the Netgate Installer from the store (free). It will download the latest version when you run it. Actually I think they added, or were talked about adding, a version selection.

https://docs.netgate.com/pfsense/en/latest/install/index.html

andrew_cb · Feb 3, 2025, 5:29 PM

@marcosm I am glad Netgate is committed to addressing the issues raised in this thread. When do you think Netgate will be able to share a high-level overview of the planned changes so the community can give feedback? It could be a new thread to help keep the discussion focused.

Feel free to contact me publicly or privately if there is anything I can assist with.

arri · Feb 8, 2025, 5:22 AM

Add me to the list. I have a number of 4100 base models installed in a different time zone that, had I known, would have been upgraded at purchase a couple of years ago. The first of them just just died on me. Fortunately I had a replacement on my shelf that I overnighted with a backup config. However, the remainder which were similarly configured are now likely at high mortality risk which led me to this thread for options.

For reference booting the dead 4100 and attempting to install a freshly downloaded copy from the store finds no valid storage devices.

If Netgate sold an appropriately overpriced nvme with B key on the store for the 4100 I would buy them just to sleep comfortably again.

In the meantime I can only turn off the packages I naively installed and cross my fingers.

stephenw10 · Feb 8, 2025, 5:33 AM

You can also enable ram disks to reduce drive writes significantly.

arri · Feb 8, 2025, 5:12 PM

@stephenw10 Thanks for the reminder, back in the day ram disks were the norm (and I may be deluding myself recalling also the default config) on my Alix boxes. I got lazy assuming these boxes would be preconfigured for longevity.

w0w · Feb 8, 2025, 7:35 PM

@stephenw10
4100 have only 4GB of RAM? In some use cases with such a small amount of memory, it's already barely enough for the device to function. And if you add a RAM disk... You'll have to significantly cut down on resource usage and configure it so that there are very few writes in general.

My hardware is not Netgate, and my SSD is a Samsung Pro SATA 256GB and RAM size is 16GB.

Over almost five years, I have accumulated about 32TB of writes (~20GB per day). Recently, I started using RAM disks, reducing writes to around 1GB or less per day. Enabling RAM disks was not straightforward—it required a lot of trial and error.

In the end, I optimized log writing and analyzed the temp folder to understand what was writing to the disk and why. However, even now, /tmp is set to 1024, and /var to 8192 due to stability concerns.

I think the main problem is that eMMC wear can easily go unnoticed. And perhaps there should have been a preset in the Plus version that, where possible, checks the wear status and notifies the user well in advance of a critical state in every possible noticeable way, through all kinds of alerts. I’m not sure if it’s already too late for this or not...

stephenw10 · Feb 8, 2025, 8:41 PM

@w0w said in Another Netgate with storage failure, 6 in total so far:

4100 have only 4GB of RAM? In some use cases with such a small amount of memory, it's already barely enough for the device to function. And if you add a RAM disk... You'll have to significantly cut down on resource usage and configure it so that there are very few writes in general.

Well, that depends on quite a lot of things. My edge device here is a 3100. That has 2GB RAM and I run RAM disks on that at 80/160MB without issue. That's running pfBlocker and Snort.

But obviously I didn't just enable all the lists and signatures.

SteveITS · Feb 8, 2025, 9:31 PM

@w0w Recent versions of pfSense don’t allocate the RAM disk space until it’s used, so it’s more flexible.

michmoor · Feb 8, 2025, 9:34 PM

@stephenw10 curious I have an 1100.
Been running into OOM situations with pfblocker and some services like snmp crashing.
I am somewhat concerned about the writes but if I move to RAM disk do my issues go away?

SteveITS · Feb 8, 2025, 10:23 PM

@michmoor A RAM disk seems unlikely to help a low memory problem. You could limit writes as noted above though.

arri · Feb 8, 2025, 10:30 PM

Fortunately all of my 4100's are only using a small portion of their RAM (especially since I can't really use any additional packages causing emmc wear now that I know better) whereas they exhausted 100% of their emmc writes in just a couple of years so the RAM disk is a helpful stopgap until I am able to physically access the devices again.

Unfortunately, it looks like the reboot necessary to engage the ram disk probably took out another one today!