NVME Wear Rate on 6100

ahxcjay

Haven't had my 6100 for 2 years, yet SMART is telling me my drive is 69% worn already. It's also written 186TB. I'm not running Suricata. I have vnstat and rrd-summary installed.

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        54 Celsius
Available Spare:                    100%
Available Spare Threshold:          1%
Percentage Used:                    69%
Data Units Read:                    503,711 [257 GB]
Data Units Written:                 364,300,734 [186 TB]
Host Read Commands:                 12,095,957
Host Write Commands:                3,300,796,255
Controller Busy Time:               59,272
Power Cycles:                       45
Power On Hours:                     6,553
Unsafe Shutdowns:                   13
Media and Data Integrity Errors:    0
Error Information Log Entries:      196
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               78 Celsius
Temperature Sensor 2:               54 Celsius
Temperature Sensor 3:               54 Celsius
Temperature Sensor 4:               54 Celsius
Thermal Temp. 1 Transition Count:   2
Thermal Temp. 1 Total Time:         29945

This is surely ridiculous for a drive not even 2 years old?

keyser

@ahxcjay Search the forum or read this looong thread:
https://forum.netgate.com/topic/195990/another-netgate-with-storage-failure-6-in-total-so-far/38?_=1744872979707

This is unfortunately the new “standard” of SSD/eMMC wear because of the switch to the ZFS filesystem combined with your firewall fx. Logs a lot of traffic on rules or have packages installed.
Be glad you caught it in time. You can now adjust your logging/disc writes, or enable RAM disk if needed if you want to avoid changing the SSD in another year or two.

ahxcjay

@keyser Thank you.

Yes, looks like syslog is consuming a tonne of writes my side. I've turned off default logging for deny rules and moved things to a ram disk for the time being.

This is insane though that we're buying equipment with very finite storage lifetime.

michmoor

@ahxcjay curious but whats writing so much?

I have had my 6100 Max for 3x years. I intentionally log every rule (compliance) along with pfblocker logging. There was a brief stint where i had to have ntop-ng running for a week.
No suricata/snort.

== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        59 Celsius
Available Spare:                    100%
Available Spare Threshold:          1%
Percentage Used:                    31%
Data Units Read:                    35,772 [18.3 GB]
Data Units Written:                 28,839,860 [14.7 TB]
Host Read Commands:                 597,968
Host Write Commands:                929,740,937
Controller Busy Time:               11,409
Power Cycles:                       37
Power On Hours:                     6,553
Unsafe Shutdowns:                   30
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               83 Celsius
Temperature Sensor 2:               59 Celsius
Temperature Sensor 3:               59 Celsius
Temperature Sensor 4:               60 Celsius
Thermal Temp. 1 Transition Count:   1

ahxcjay

@michmoor Your R&W stats are way below mine, yet you're at 31%. It's a really sucky situation.

I wish I knew what was writing so much. I run NextDNS, maybe it's that?

michmoor

@ahxcjay yep it is

ahxcjay

@michmoor Currently looking for a decent B+M key SSD to replace it with. I have a remote pfSense instance running on Protectli hardware. About to replace that with a Samsung Evo 870 when I'm on-site. The current WD one it has isn't in the smartdb. The wear indicator is '101' which is clearly wrong.

tinfoilmatt

@ahxcjay If it makes you feel any better, my production box's M.2 NVMe drive is currently sitting at 133%. You got time. ;)

ahxcjay

@tinfoilmatt hah! Appreciate it! I've mounted RAM disks for now. I wouldn't mind replacing the NVMe, it's just that finding a decent B&M keyed one now is so difficult.

slu

@ahxcjay said in NVME Wear Rate on 6100:

It's also written 186TB.

I had the same issue:
https://forum.netgate.com/topic/189820/how-do-i-find-out-what-write-continuously-on-my-pfsense-ssd

Patch

@keyser said in NVME Wear Rate on 6100:

https://forum.netgate.com/topic/195990/another-netgate-with-storage-failure-6-in-total-so-far/38?_=1744872979707

I agree that thread contains an excellent analysis of the problem described in this thread together with solutions, so probably a better place to continue this conversations.

Starting near the end is easiest https://forum.netgate.com/topic/195990/another-netgate-with-storage-failure-6-in-total-so-far/

andrew_cb

@ahxcjay The KingSpec NE 2242 is one of the few widely-available options:
https://www.amazon.com/KingSpec-256GB-Performance-Internal-Ultrabook/dp/B08TTDQ5WH?th=1

ahxcjay

@andrew_cb said in NVME Wear Rate on 6100:

@ahxcjay The KingSpec NE 2242 is one of the few widely-available options:
https://www.amazon.com/KingSpec-256GB-Performance-Internal-Ultrabook/dp/B08TTDQ5WH?th=1

Installed yesterday. It's tiny.

=== START OF INFORMATION SECTION ===
Model Number:                       NE-256 2242
Serial Number:                      
Firmware Version:                   SN25845
PCI Vendor/Subsystem ID:            0x1e4b
IEEE OUI Identifier:                0x0c82d5
Total NVM Capacity:                 256,060,514,304 [256 GB]
Unallocated NVM Capacity:           0
Controller ID:                      0
NVMe Version:                       1.4
Number of Namespaces:               1
Namespace 1 Size/Capacity:          256,060,514,304 [256 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            0c82d5 9581000413
Local Time is:                      Tue Apr 22 21:16:29 2025 EDT
Firmware Updates (0x16):            3 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Log Page Attributes (0x06):         Cmd_Eff_Lg Ext_Get_Lg
Maximum Data Transfer Size:         128 Pages
Warning  Comp. Temp. Threshold:     90 Celsius
Critical Comp. Temp. Threshold:     95 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     6.50W       -        -    0  0  0  0        0       0
 1 +     5.80W       -        -    1  1  1  1        0       0
 2 +     3.60W       -        -    2  2  2  2        0       0
 3 -   0.7460W       -        -    3  3  3  3     5000   10000
 4 -   0.7260W       -        -    4  4  4  4     8000   45000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        41 Celsius
Available Spare:                    100%
Available Spare Threshold:          1%
Percentage Used:                    0%
Data Units Read:                    2,707 [1.38 GB]
Data Units Written:                 9,386 [4.80 GB]
Host Read Commands:                 50,627
Host Write Commands:                189,770
Controller Busy Time:               0
Power Cycles:                       4
Power On Hours:                     28
Unsafe Shutdowns:                   2
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               41 Celsius
Temperature Sensor 2:               49 Celsius