Another Netgate with storage failure, 6 in total so far
-
Someone with a dead 4200 today. Killed by ntopng in 10 months. The user was unaware of any risks from running ntopng on 16gb of eMMC, and there is no way to monitor the eMMC on the 4200. Luckily the device is still under warranty so it's being replaced under RMA.
-
Based on what I've learned from this thread, I added a 256GB Samsung SSD to my 4200 today, replacing the built-in drive, and it's working fine. Netgate instructions had me hopping around from place to place in the documentation but did they did the job.
I don't want foreseeable future problems, so thank everyone who contributed here. Hopefully this will lead to a longer life than this box might have otherwise had.
-
@Mission-Ghost I am glad you found this thread useful. A 256GB SSD should last a long time!
-
One thing that has always stood out to me about my data has been the 8 devices with with average write rates below 50KBps.
Today I checked our devices and confirmed that those 8 outliers are all running UFS and everything else is using ZFS.
Compared to the highest UFS rate, the ZFS rate is from 2.5x to 7.5x higher.I also looked at some of the devices that have high storage wear. They are in smallish offices and are just doing basic functions. The only packages installed are Zabbix Agent and Zabbix Proxy. A few had the logging enabled for the default rules so I turned those off.
I tried to find a reason why all the devices using ZFS have such high average writes compared to the devices using UFS, but could find no explanation. We use a standardized configuration and nearly all devices are low-load, and just have the Zabbix packages. On most, the log entries for each category fit within the default 500 events shown. I copied a day's worth of general system log events into a text file - it was 38KB.
I went so far as to raise the update interval from 1 minute to 5 minutes of nearly all items in the Zabbix template, but that made no difference.300KB/sec is 18MB/min, 1.1GB/hour, 25GB/day, 9.4TB/year, 18.8TB/2 years, 28.2TB/3 years. This is in the ballpark for the maximum write life of the storage. No wonder we are seeing so many failures at the 2-3 year mark!
Comparatively, a device doing 50KB/sec would be at 4.7TB after 3years and 9.4TB after 6 years.
This could explain why our older 3100 and 7100 units on UFS have lasted 6-7 years and the eMMC is still in good health, meanwhile we have many 4100 that have failed or are near death in only 2 years.
In his thread eMMC Write endurance, @keyser noted
With ZFS, pfBlockerNG in default config with only 4 feeds loaded and NTopNG running, my box averages about 1 MB/s sustained write to the SSD.
I am only 700KBps less (300KBps vs 1000KBps) yet am not running pfblockerng or ntopng.
I will need to dig in deeper with iostat, top, and systat to try and find the cause of the writes. At this point it would appear that ZFS itself is the major cause of the increased write activity compared to UFS.
-
@stephenw10 said in Another Netgate with storage failure, 6 in total so far:
Hmm, not sure why the pkg isn't in the CE repo. I guess there wasn't much call for it at the time. Seems like we could add that pretty easily. Let me see....
Did you have any luck getting mmc-utils added to the CE repo?
-
@andrew_cb said in Another Netgate with storage failure, 6 in total so far:
I will need to dig in deeper with iostat, top, and systat to try and find the cause of the writes.
Hi,
I got a reduction from ~19GBw/day to 1,8 GBw/day by using this settings:
zfs set sync=disabled zroot/tmp (pfSense/tmp) zfs set sync=disabled zroot/var (pfSense/var) (after review my settings I saw that I had set it to disabled)and fine tuning:
vfs.zfs.txg.timeout=120(ZFS Pool in my case is "zroot" actual systems use "pfSense")
Remarc: this is a private system and private use.
-
@fireodo
A wonderful idea and discovery! It seems quite reasonable not to synchronize the tmp folder and 2 minutes delay for transaction writes. Good alternative to ram disks if it can not be used for some reason. -
@w0w said in Another Netgate with storage failure, 6 in total so far:
2 minutes delay
PS. If you test you can set the delay to greater values de amount of writing rate will decrease but you have a greater risk of loosing data when a power failure comes in ... (it reduce the robustness of ZFS filesystem)
-
In the case of a firewall, I think it is acceptable.
Most critical logs should be sent to an external syslog server, and I don't see any risks that could compromise the system. I can't think of any scenarios where this would be critical for pfSense, but I might be wrong. I don't know—some major updates are also managed by BE and shouldn't be affected. -
@andrew_cb said in Another Netgate with storage failure, 6 in total so far:
it would appear that ZFS itself is the major cause of the increased write activity
That is my understanding. ZFS results in significant write amplification but as a result is more robust on power failure.
But I thought later installs of pfsense did not use ZFS for temporary files.
-
/var should be standard sync by default anyway, was yours not?
-
@stephenw10 said in Another Netgate with storage failure, 6 in total so far:
/var should be standard sync by default anyway, was yours not?
It was, but I set it to disabled! (my decision)
Here the output of my
zfs get -r sync zrootNAME PROPERTY VALUE SOURCE zroot sync standard local zroot/ROOT sync standard inherited from zroot zroot/ROOT/default sync standard inherited from zroot zroot/ROOT/default/var_cache_pkg sync standard inherited from zroot zroot/ROOT/default/var_db_pkg sync standard inherited from zroot zroot/reservation sync standard inherited from zroot zroot/tmp sync disabled local zroot/var sync disabled localEdit: The firewall is working with this settings since 08.2021.
I experimented also with huge delays (up to 1800s what make the amount of daily written data to go down to 0,28GBw/day ...)
Its up to you to consider what weights more to you! (with a eMMC that cannot be replaced easily, I guess its better to go with that risk!) -
In 2100-max
zfs get -r sync zroot
Shell Output - zfs get -r sync zroot
cannot open 'zroot': dataset does not existShell Output - zpool status -x
all pools are healthyShell Output - zfs get -r sync NAME PROPERTY VALUE SOURCE pfSense sync standard default pfSense/ROOT sync standard default pfSense/ROOT/23_05_01_clone sync standard default pfSense/ROOT/23_05_01_clone/cf sync standard default pfSense/ROOT/23_05_01_clone/var_db_pkg sync standard default pfSense/ROOT/23_05_01_ipv4 sync standard default pfSense/ROOT/23_05_01_ipv4@2024-01-12-11:46:05-0 sync - - pfSense/ROOT/23_05_01_ipv4@2024-02-29-08:52:57-0 sync - - pfSense/ROOT/23_05_01_ipv4@2024-04-01-12:32:27-0 sync - - pfSense/ROOT/23_05_01_ipv4@2024-06-27-11:52:26-0 sync - - pfSense/ROOT/23_05_01_ipv4@2024-07-03-09:09:28-0 sync - - pfSense/ROOT/23_05_01_ipv4@2025-01-20-10:11:49-0 sync - - pfSense/ROOT/23_05_01_ipv4/cf sync standard default pfSense/ROOT/23_05_01_ipv4/cf@2025-01-20-10:11:49-0 sync - - pfSense/ROOT/23_05_01_ipv4/var_cache_pkg sync standard default pfSense/ROOT/23_05_01_ipv4/var_cache_pkg@2025-01-20-10:11:49-0 sync - - pfSense/ROOT/23_05_01_ipv4/var_db_pkg sync standard default pfSense/ROOT/23_05_01_ipv4/var_db_pkg@2025-01-20-10:11:49-0 sync - - pfSense/ROOT/23_05_01_ipv4_Backup sync standard default pfSense/ROOT/23_05_01_ipv4_Backup/cf sync standard default pfSense/ROOT/23_05_01_ipv4_Backup/var_cache_pkg sync standard default pfSense/ROOT/23_05_01_ipv4_Backup/var_db_pkg sync standard default pfSense/ROOT/23_05_01_ipv6 sync standard default pfSense/ROOT/23_05_01_ipv6@2024-07-25-15:54:45-0 sync - - pfSense/ROOT/23_05_01_ipv6/cf sync standard default pfSense/ROOT/23_05_01_ipv6/var_db_pkg sync standard default pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy sync standard default pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/cf sync standard default pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/cf@2024-01-12-11:46:05-0 sync - - pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/cf@2024-02-29-08:52:57-0 sync - - pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/cf@2024-04-01-12:32:27-0 sync - - pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/cf@2024-06-27-11:52:26-0 sync - - pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/cf@2024-07-25-15:54:45-0 sync - - pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/var_db_pkg sync standard default pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/var_db_pkg@2024-01-12-11:46:05-0 sync - - pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/var_db_pkg@2024-02-29-08:52:57-0 sync - - pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/var_db_pkg@2024-04-01-12:32:27-0 sync - - pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/var_db_pkg@2024-06-27-11:52:26-0 sync - - pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/var_db_pkg@2024-07-25-15:54:45-0 sync - - pfSense/ROOT/23_09_01_ipv4_20240703094025 sync standard default pfSense/ROOT/23_09_01_ipv4_20240703094025/cf sync standard default pfSense/ROOT/23_09_01_ipv4_20240703094025/var_cache_pkg sync standard default pfSense/ROOT/23_09_01_ipv4_20240703094025/var_db_pkg sync standard default pfSense/ROOT/24_03_01_ipv4 sync standard default pfSense/ROOT/24_03_01_ipv4/cf sync standard default pfSense/ROOT/24_03_01_ipv4/var_cache_pkg sync standard default pfSense/ROOT/24_03_01_ipv4/var_db_pkg sync standard default pfSense/ROOT/24_03_01_ipv6_20250113135850 sync standard default pfSense/ROOT/24_03_01_ipv6_20250113135850@2024-07-03-09:40:36-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850@2024-07-23-10:05:22-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850@2025-01-13-13:59:02-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/cf sync standard default pfSense/ROOT/24_03_01_ipv6_20250113135850/cf@2024-07-03-09:09:28-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/cf@2024-07-03-09:40:36-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/cf@2024-07-23-10:05:22-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/cf@2025-01-13-13:59:02-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/var_cache_pkg sync standard default pfSense/ROOT/24_03_01_ipv6_20250113135850/var_cache_pkg@2024-07-03-09:09:28-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/var_cache_pkg@2024-07-03-09:40:36-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/var_cache_pkg@2024-07-23-10:05:22-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/var_cache_pkg@2025-01-13-13:59:02-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/var_db_pkg sync standard default pfSense/ROOT/24_03_01_ipv6_20250113135850/var_db_pkg@2024-07-03-09:09:28-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/var_db_pkg@2024-07-03-09:40:36-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/var_db_pkg@2024-07-23-10:05:22-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/var_db_pkg@2025-01-13-13:59:02-0 sync - - pfSense/ROOT/auto-default-20240112115753 sync standard default pfSense/ROOT/auto-default-20240112115753@2024-01-12-11:57:53-0 sync - - pfSense/ROOT/auto-default-20240112115753/cf sync standard default pfSense/ROOT/auto-default-20240112115753/cf@2024-01-12-11:57:53-0 sync - - pfSense/ROOT/auto-default-20240112115753/var_cache_pkg sync standard default pfSense/ROOT/auto-default-20240112115753/var_cache_pkg@2024-01-12-11:57:53-0 sync - - pfSense/ROOT/auto-default-20240112115753/var_db_pkg sync standard default pfSense/ROOT/auto-default-20240112115753/var_db_pkg@2024-01-12-11:57:53-0 sync - - pfSense/ROOT/quick-20240401123227 sync standard default pfSense/ROOT/quick-20240401123227/cf sync standard default pfSense/ROOT/quick-20240401123227/var_db_pkg sync standard default pfSense/home sync standard default pfSense/reservation sync standard default pfSense/tmp sync standard default pfSense/var sync standard default pfSense/var/cache sync standard default pfSense/var/db sync standard default pfSense/var/log sync standard default pfSense/var/tmpMake sure to upvote
-
@stephenw10 is that a setting on M.2 sata SSDs and NVMe drives we should leave be?
-
@JonathanLee said in Another Netgate with storage failure, 6 in total so far:
zfs get -r sync zroot
replace "zroot" with "pfSense" in your case!
-
Shell Output - zfs get -r sync pfSense NAME PROPERTY VALUE SOURCE pfSense sync standard default pfSense/ROOT sync standard default pfSense/ROOT/23_05_01_clone sync standard default pfSense/ROOT/23_05_01_clone/cf sync standard default pfSense/ROOT/23_05_01_clone/var_db_pkg sync standard default pfSense/ROOT/23_05_01_ipv4 sync standard default pfSense/ROOT/23_05_01_ipv4@2024-01-12-11:46:05-0 sync - - pfSense/ROOT/23_05_01_ipv4@2024-02-29-08:52:57-0 sync - - pfSense/ROOT/23_05_01_ipv4@2024-04-01-12:32:27-0 sync - - pfSense/ROOT/23_05_01_ipv4@2024-06-27-11:52:26-0 sync - - pfSense/ROOT/23_05_01_ipv4@2024-07-03-09:09:28-0 sync - - pfSense/ROOT/23_05_01_ipv4@2025-01-20-10:11:49-0 sync - - pfSense/ROOT/23_05_01_ipv4/cf sync standard default pfSense/ROOT/23_05_01_ipv4/cf@2025-01-20-10:11:49-0 sync - - pfSense/ROOT/23_05_01_ipv4/var_cache_pkg sync standard default pfSense/ROOT/23_05_01_ipv4/var_cache_pkg@2025-01-20-10:11:49-0 sync - - pfSense/ROOT/23_05_01_ipv4/var_db_pkg sync standard default pfSense/ROOT/23_05_01_ipv4/var_db_pkg@2025-01-20-10:11:49-0 sync - - pfSense/ROOT/23_05_01_ipv4_Backup sync standard default pfSense/ROOT/23_05_01_ipv4_Backup/cf sync standard default pfSense/ROOT/23_05_01_ipv4_Backup/var_cache_pkg sync standard default pfSense/ROOT/23_05_01_ipv4_Backup/var_db_pkg sync standard default pfSense/ROOT/23_05_01_ipv6 sync standard default pfSense/ROOT/23_05_01_ipv6@2024-07-25-15:54:45-0 sync - - pfSense/ROOT/23_05_01_ipv6/cf sync standard default pfSense/ROOT/23_05_01_ipv6/var_db_pkg sync standard default pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy sync standard default pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/cf sync standard default pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/cf@2024-01-12-11:46:05-0 sync - - pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/cf@2024-02-29-08:52:57-0 sync - - pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/cf@2024-04-01-12:32:27-0 sync - - pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/cf@2024-06-27-11:52:26-0 sync - - pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/cf@2024-07-25-15:54:45-0 sync - - pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/var_db_pkg sync standard default pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/var_db_pkg@2024-01-12-11:46:05-0 sync - - pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/var_db_pkg@2024-02-29-08:52:57-0 sync - - pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/var_db_pkg@2024-04-01-12:32:27-0 sync - - pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/var_db_pkg@2024-06-27-11:52:26-0 sync - - pfSense/ROOT/23_05_01_ipv6_non_website_test_proxy/var_db_pkg@2024-07-25-15:54:45-0 sync - - pfSense/ROOT/23_09_01_ipv4_20240703094025 sync standard default pfSense/ROOT/23_09_01_ipv4_20240703094025/cf sync standard default pfSense/ROOT/23_09_01_ipv4_20240703094025/var_cache_pkg sync standard default pfSense/ROOT/23_09_01_ipv4_20240703094025/var_db_pkg sync standard default pfSense/ROOT/24_03_01_ipv4 sync standard default pfSense/ROOT/24_03_01_ipv4/cf sync standard default pfSense/ROOT/24_03_01_ipv4/var_cache_pkg sync standard default pfSense/ROOT/24_03_01_ipv4/var_db_pkg sync standard default pfSense/ROOT/24_03_01_ipv6_20250113135850 sync standard default pfSense/ROOT/24_03_01_ipv6_20250113135850@2024-07-03-09:40:36-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850@2024-07-23-10:05:22-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850@2025-01-13-13:59:02-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/cf sync standard default pfSense/ROOT/24_03_01_ipv6_20250113135850/cf@2024-07-03-09:09:28-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/cf@2024-07-03-09:40:36-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/cf@2024-07-23-10:05:22-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/cf@2025-01-13-13:59:02-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/var_cache_pkg sync standard default pfSense/ROOT/24_03_01_ipv6_20250113135850/var_cache_pkg@2024-07-03-09:09:28-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/var_cache_pkg@2024-07-03-09:40:36-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/var_cache_pkg@2024-07-23-10:05:22-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/var_cache_pkg@2025-01-13-13:59:02-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/var_db_pkg sync standard default pfSense/ROOT/24_03_01_ipv6_20250113135850/var_db_pkg@2024-07-03-09:09:28-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/var_db_pkg@2024-07-03-09:40:36-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/var_db_pkg@2024-07-23-10:05:22-0 sync - - pfSense/ROOT/24_03_01_ipv6_20250113135850/var_db_pkg@2025-01-13-13:59:02-0 sync - - pfSense/ROOT/auto-default-20240112115753 sync standard default pfSense/ROOT/auto-default-20240112115753@2024-01-12-11:57:53-0 sync - - pfSense/ROOT/auto-default-20240112115753/cf sync standard default pfSense/ROOT/auto-default-20240112115753/cf@2024-01-12-11:57:53-0 sync - - pfSense/ROOT/auto-default-20240112115753/var_cache_pkg sync standard default pfSense/ROOT/auto-default-20240112115753/var_cache_pkg@2024-01-12-11:57:53-0 sync - - pfSense/ROOT/auto-default-20240112115753/var_db_pkg sync standard default pfSense/ROOT/auto-default-20240112115753/var_db_pkg@2024-01-12-11:57:53-0 sync - - pfSense/ROOT/quick-20240401123227 sync standard default pfSense/ROOT/quick-20240401123227/cf sync standard default pfSense/ROOT/quick-20240401123227/var_db_pkg sync standard default pfSense/home sync standard default pfSense/reservation sync standard default pfSense/tmp sync standard default pfSense/var sync standard default pfSense/var/cache sync standard default pfSense/var/db sync standard default pfSense/var/log sync standard default pfSense/var/tmp sync standard default -
Changing the sync settings by itself doesn't make much difference in my testing. Setting the txg delta value higher makes a far larger difference but that increases the possibility of filesystem issues. And it's globally set for all the zfs mount points not just var and tmp. So there's a tradeoff.
-
To offset any potential issue when adapting ZFS : as this is uncharted theory, at least, it is on this forum, just to be sure : use and maintain a working UPS.
I think this is valid : way more complex files systems are way more complex to repair. -
@stephenw10 said in Another Netgate with storage failure, 6 in total so far:
Setting the txg delta value higher makes a far larger difference but that increases the possibility of filesystem issues.
Yes, thats the bitter pill you have to swallow ...
-
Be good if you could set it for /tmp only.....
