Another Netgate with storage failure, 6 in total so far
-
@andrew_cb said in Another Netgate with storage failure, 6 in total so far:
25% of the blocks are not available for wear leveling
In the same vein, this is a bit of an edge case, but I've strung a few bugs together.
-
there is a bug in Plus 24.03 and 24.11 where /conf/backup is not limited to 30 files (not auto pruned). Fixed in 25.03. https://redmine.pfsense.org/issues/15994, or the release notes. Workaround is to open the /diag_confbak.php config history page in the web GUI, and wait until it either loads or times out.
-
there's a longstanding bug in pfBlocker where if DNSBL is not enabled it will still update the config file at every cron interval, e.g. hourly.
https://forum.netgate.com/topic/174231/pfblockerng-fills-pfsense-config-history
https://redmine.pfsense.org/issues/14409 -
there's another longstanding bug in pfBlocker for HA setups where changes are not synced to the secondary router unless one manually runs a Force Reload (not a force update). Thus if you have, say, disabled a list, at every cron/update it will remove it from the backup and then when the same cron also runs on the secondary pfBlocker will add it again, generating two historical config files on the secondary router.
https://redmine.pfsense.org/issues/15994
https://forum.netgate.com/topic/179060/pfblockerng-sync-not-working/
With these, one poster in my thread https://forum.netgate.com/topic/197685/config-history-not-pruning-on-ha-pair-has-3400-files/ has 20000 config files on disk. At our file sizes of ~300k that is in the 6 GB range, though it should be compressed on disk, if using ZFS.
-
-
@SteveITS FWIW #2 should be fixed with the version in the 2.8.0/25.03 branch. As for #3 that may be fixed with https://redmine.pfsense.org/issues/16231 though the change may only take effect once the cron job runs on the secondary.
-
Wow, I stop checking the forum for a bit and come back to find that the ZFS patch has been released!
Thank you to @marcosm @stephenw10 @cmcdonald @dennypage @arri @w0w @SteveITS @Gertjan @fireodo @chrcoluk and everyone else that has contributed to this discussion and process.
Hopefully, this change will help reduce the change of storage failure for all devices running pfSense, especially those using small-sized and/or eMMC storage.
It is encouraging to see that additional areas have been identified for further improvements to storage wear and space usage.
We have progressed a long way from "you're holding it wrong."
-
I'll add another disk write case, though it's in ISC DHCP which is deprecated. I find "Ignore denied clients rather than reject" sometimes will log the rejection even if the option is checked, which usually results in a log entry every few seconds. Restarting DHCP Server seems to fix it (no log), and at some point later (update? restart? random?) I will find it is logging again.
-
"Thinking out loud here" :
If a (DHCP) client was set to be rejected, it will do the reject (it will answer "no") and normally, the client should take "no" for an answer - the the sofware is RFC compliant (I guess).
Let's consider 'ISC DHCP' as the old generation.
But its 2025, the client is probably a new generation DHCP client and won't take no for an answer, do'nt bother with RFC, so it keeps on insisting.
Now, ISC DHCP start to log.It's in the admin's authority to take things one level up. It was the admin after all who decided who to serve, and who to reject.
Go visit the client, and tell him who is boss in the town.
Next step : MAC black list the guy and call it a day. -
fwiw, It appears that Netgate only offers the 4200 with the 128Gb SSD, probably due to the eMMC issues.
-
Just for reference here's my 6100 MAX failure story:
https://forum.netgate.com/topic/198361/6100-max-nvme-failed -
Wow...
very interesting thread.
I found this just yesterday and it takes me half the night to to read it from start to end
.
Actually I am using a SG-3100 device which I switched to SATA SSD abt. 3 years ago.
I was thinking about replacing it with a newer appliance, i.e. a SG-4200, thats why I am looking around here.
To be honest, there is no technical reason for that, it was just to keep pfSense at the latest.
But just this days a new v25.07 was released so I will keep my SG-3100 for a while.And BTW: the SMART values shows the SSD is still at 94% lifetime, so I can run the device may be until a 4300/4400/4x00 is availabe
.Regards
-
@FSC830 said in Another Netgate with storage failure, 6 in total so far:
Wow...
very interesting thread.
I found this just yesterday and it takes me half the night to to read it from start to end .same here, just finished.
I confirm, I had no idea. Don't recall seeing it mentioned in the marketing data, docs, nor by the gurus at YT.
I've chosen NG's hw specifically for the reliability and minimal maintenance. Knowing this probably would go 'max'.Adjusted the timings, no sync for tmp. Soon on my way for new ssd. Thanks.
Can you advice me how can I check emmc healt in SG2100?
-
@mmkkoo
https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html -
M mmkkoo referenced this topic
