Another Netgate with storage failure, 6 in total so far

SteveITS

@andrew_cb said in Another Netgate with storage failure, 6 in total so far:

FW1 Result: 6GB more space after deleting 5 boot environments (avg 1.2GB per boot environment).

FWIW that's pretty typical, it just depends on how much changed on disk in between. The "32K" or whatever shows in the GUI is very misleading there, IMO.

andrew_cb

@SteveITS said in Another Netgate with storage failure, 6 in total so far:

@andrew_cb said in Another Netgate with storage failure, 6 in total so far:

FW1 Result: 6GB more space after deleting 5 boot environments (avg 1.2GB per boot environment).

FWIW that's pretty typical, it just depends on how much changed on disk in between. The "32K" or whatever shows in the GUI is very misleading there, IMO.

Yes, I figured that the 32K was not accurate.
The size of the current boot environment decreases to 1.7-1.9GB after the other boot environments are deleted, which is more in line with the actual space consumed.

chrcoluk

@tinfoilmatt said in Another Netgate with storage failure, 6 in total so far:

But so basically you agree that increasing the vfs.zfs.txg.timeout parameter is a net good thing...

For sure.

valnar

As someone about to buy a 4200, does anyone know if Netgate has implemented any of these strategies in their latest pfSense Plus? If not, can somebody summarize the recommended tweaks?

Mission-Ghost

@valnar buy a plus, set the vfs.zfs.txg.timeout to a value of between 15 and 30 seconds, connect it to an apc ups and use apcupsd package to initiate low power safe shutdown.

A bonus would be to buy a fan so it runs cool. Not necessary by any means but would probably help with overall service life.

My feeble understanding is that Netgate is working on zfs adjustments to address the poor ssd life but has released nothing yet.

valnar

@Mission-Ghost Thank you.

SteveITS

@valnar said in Another Netgate with storage failure, 6 in total so far:

summarize the recommended tweaks

See this post:
https://forum.netgate.com/topic/195990/another-netgate-with-storage-failure-6-in-total-so-far/21

andrew_cb

@valnar For now, I recommend these two changes:

Navigate to System > Advanced > System Tunables and add a new tunable:

Tunable vfs.zfs.tgx.timeout
Value 30

Navigate to Diagnostics > Command Prompt and run
zfs set sync=always pfSense/ROOT/default/cf

azdeltawye

@andrew_cb

Thanks for the easy step-by-step instruction!

Quick question - Do you have to run the zfs set command on each subsequent reboot or just the first time after adding the tunable?

Mission-Ghost

@andrew_cb said in Another Netgate with storage failure, 6 in total so far:

Navigate to Diagnostics > Command Prompt and run
zfs set sync=always pfSense/ROOT/default/cf

Note: the command should read:

zfs set sync=always pfSense/ROOT/Default/cf

Default

starts with a capital D.

Mission-Ghost

@azdeltawye said in Another Netgate with storage failure, 6 in total so far:

Quick question - Do you have to run the zfs set command on each subsequent reboot or just the first time after adding the tunable?

I'd be interested in the answer as well. The freeBSD help and Google Gemini (which is often wrong but never in doubt) suggests it's a one-time setting.

I missed this setting in earlier discussions, so I just executed it and it's set in the zfs config now. I haven't rebooted. Yet.

fireodo

@Mission-Ghost said in Another Netgate with storage failure, 6 in total so far:

I missed this setting in earlier discussions, so I just executed it and it's set in the zfs config now. I haven't rebooted. Yet.

If you make a entry in the "system tuning" the setting will survive even a system upgrade (if still necessary)...

elvisimprsntr

Even though I have a Kingston mSATA SDD with 2.1 years power on hours, life is down to 90%


ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  9 Power_On_Hours          0x0000   100   100   000    Old_age   Offline      -       18867
169 Remaining_Lifetime_Perc 0x0000   090   090   000    Old_age   Offline      -       90

sysctl vfs.zfs.txg.timeout=30

Lower case "default" on 2.7.2 CE

zfs list

NAME                                 USED  AVAIL  REFER  MOUNTPOINT
pfSense                             24.1G   205G    96K  /pfSense
pfSense/ROOT                        1.11G   205G    96K  none
pfSense/ROOT/default                1.11G   205G  1.10G  /
pfSense/ROOT/default/cf             6.88M   205G  6.88M  /cf
pfSense/ROOT/default/var_cache_pkg   336K   205G   336K  /var/cache/pkg
pfSense/ROOT/default/var_db_pkg     5.42M   205G  5.42M  /var/db/pkg
pfSense/home                         108K   205G   108K  /home
pfSense/reservation                   96K   227G    96K  /pfSense/reservation
pfSense/tmp                         1.37M   205G  1.37M  /tmp
pfSense/var                         35.9M   205G  3.45M  /var
pfSense/var/cache                     96K   205G    96K  /var/cache
pfSense/var/db                      2.38M   205G  2.38M  /var/db
pfSense/var/empty                     96K   205G    96K  /var/empty
pfSense/var/log                     29.8M   205G  29.8M  /var/log
pfSense/var/tmp                      136K   205G   136K  /var/tmp

zfs set sync=always pfSense/ROOT/default/cf

fireodo

@valnar said in Another Netgate with storage failure, 6 in total so far:

If not, can somebody summarize the recommended tweaks?

These are my settings:

vfs.zfs.txg.timeout=120 (you can increase this value as you wish - I was gone until 2400 in experiments with no problems)
zfs set sync=disabled zroot/tmp (pfSense/tmp)
zfs set sync=disabled zroot/var (pfSense/var)

Regards,
fireodo

Mission-Ghost

@elvisimprsntr Interesting. Why the inconsistency, I wonder?

Mission-Ghost

@fireodo said in Another Netgate with storage failure, 6 in total so far:

If you make a entry in the "system tuning" the setting will survive even a system upgrade (if still necessary)...

Sweet. Thanks.

fireodo

@Mission-Ghost said in Another Netgate with storage failure, 6 in total so far:

Sweet.

Nice that it tastes good

JonathanLee

@fireodo You should set this in cron right? Because this will default every reboot...

zfs set sync=disabled pfSense/var
zfs set sync=disabled pfSense/tmp

should they no longer show if you run "zfs list" also?

fireodo

@JonathanLee said in Another Netgate with storage failure, 6 in total so far:

You use the zfs set sync=disabled pfSense/var in cron right?

No.

Because this will default every reboot...

Not here - once set it remains that way even after reboot. (btw. on 23.04.2025 the system was rebooted, after an long power outage that exhausted the UPS, caused by work on the power line by the provider)

fireodo

@JonathanLee said in Another Netgate with storage failure, 6 in total so far:

should they no longer show if you run "zfs list" also?

You should see the status if you run:

zfs get -r sync pfSense

or

zfs get -r sync zroot

depending on what pfsense system you are.