Another Netgate with storage failure, 6 in total so far

SteveITS

@Mission-Ghost No Global should mean it doesn’t override the individual settings. I just set it when creating each list so if the global settings aren’t working I profess ignorance. :)

Mission-Ghost

@SteveITS said in Another Netgate with storage failure, 6 in total so far:

@Mission-Ghost No Global should mean it doesn’t override the individual settings. I just set it when creating each list so if the global settings aren’t working I profess ignorance. :)

Well, I guess it should mean it, but in context to some of of us who didn't develop the software, it isn't clear, particularly when adjacent options include "no logging" which apparently could not mean 'no' logging.

Seems like getting an English major (>gasp!<) intern to help redefine the labels to be more meaningful to customers would be a low cost, easy improvement to the usability of the product.

In any case, thank you for your generous help clarifying this. My problem is solved.

SteveITS

@andrew_cb said in Another Netgate with storage failure, 6 in total so far:

25% of the blocks are not available for wear leveling

In the same vein, this is a bit of an edge case, but I've strung a few bugs together.

there is a bug in Plus 24.03 and 24.11 where /conf/backup is not limited to 30 files (not auto pruned). Fixed in 25.03. https://redmine.pfsense.org/issues/15994, or the release notes. Workaround is to open the /diag_confbak.php config history page in the web GUI, and wait until it either loads or times out.
there's a longstanding bug in pfBlocker where if DNSBL is not enabled it will still update the config file at every cron interval, e.g. hourly.
https://forum.netgate.com/topic/174231/pfblockerng-fills-pfsense-config-history
https://redmine.pfsense.org/issues/14409
there's another longstanding bug in pfBlocker for HA setups where changes are not synced to the secondary router unless one manually runs a Force Reload (not a force update). Thus if you have, say, disabled a list, at every cron/update it will remove it from the backup and then when the same cron also runs on the secondary pfBlocker will add it again, generating two historical config files on the secondary router.
https://redmine.pfsense.org/issues/15994
https://forum.netgate.com/topic/179060/pfblockerng-sync-not-working/

With these, one poster in my thread https://forum.netgate.com/topic/197685/config-history-not-pruning-on-ha-pair-has-3400-files/ has 20000 config files on disk. At our file sizes of ~300k that is in the 6 GB range, though it should be compressed on disk, if using ZFS.

marcosm

@SteveITS FWIW #2 should be fixed with the version in the 2.8.0/25.03 branch. As for #3 that may be fixed with https://redmine.pfsense.org/issues/16231 though the change may only take effect once the cron job runs on the secondary.

andrew_cb

Wow, I stop checking the forum for a bit and come back to find that the ZFS patch has been released!

Thank you to @marcosm @stephenw10 @cmcdonald @dennypage @arri @w0w @SteveITS @Gertjan @fireodo @chrcoluk and everyone else that has contributed to this discussion and process.

Hopefully, this change will help reduce the change of storage failure for all devices running pfSense, especially those using small-sized and/or eMMC storage.

It is encouraging to see that additional areas have been identified for further improvements to storage wear and space usage.

We have progressed a long way from "you're holding it wrong."

SteveITS

I'll add another disk write case, though it's in ISC DHCP which is deprecated. I find "Ignore denied clients rather than reject" sometimes will log the rejection even if the option is checked, which usually results in a log entry every few seconds. Restarting DHCP Server seems to fix it (no log), and at some point later (update? restart? random?) I will find it is logging again.

Gertjan

@SteveITS

"Thinking out loud here" :
If a (DHCP) client was set to be rejected, it will do the reject (it will answer "no") and normally, the client should take "no" for an answer - the the sofware is RFC compliant (I guess).
Let's consider 'ISC DHCP' as the old generation.
But its 2025, the client is probably a new generation DHCP client and won't take no for an answer, do'nt bother with RFC, so it keeps on insisting.
Now, ISC DHCP start to log.

It's in the admin's authority to take things one level up. It was the admin after all who decided who to serve, and who to reject.
Go visit the client, and tell him who is boss in the town.
Next step : MAC black list the guy and call it a day.

jdstlnet

@michmoor

fwiw, It appears that Netgate only offers the 4200 with the 128Gb SSD, probably due to the eMMC issues.

tariqali

Just for reference here's my 6100 MAX failure story:
https://forum.netgate.com/topic/198361/6100-max-nvme-failed

FSC830

Wow...
very interesting thread.
I found this just yesterday and it takes me half the night to to read it from start to end .
Actually I am using a SG-3100 device which I switched to SATA SSD abt. 3 years ago.
I was thinking about replacing it with a newer appliance, i.e. a SG-4200, thats why I am looking around here.
To be honest, there is no technical reason for that, it was just to keep pfSense at the latest.
But just this days a new v25.07 was released so I will keep my SG-3100 for a while.

And BTW: the SMART values shows the SSD is still at 94% lifetime, so I can run the device may be until a 4300/4400/4x00 is availabe .

Regards