Another Netgate with storage failure, 6 in total so far

[ablephri]

@Gertjan Unfortunately, the COM port shows, but there's no connection on Putty.

[ablephri]

@Gertjan This is the status static lights i get on the device. Do they mean anything?

[jwt]

@ablephri come on, post the video of the 1500W heat gun that was used, like you did on reddit.

note to others in this thread: I'm working with him off-thread to get him a replacement in exchange for his broken unit.

[stephenw10]

Does the external LED start orange then change to blue?

The diag LEDs look like a normal POST.

Did you try both console ports?

[jwt]

To generally address this thread:

A larger flash device is only part of the solution. Netgate moved the entire product line to a larger eMMC device running in pSLC mode much earlier this year, without a price increase.

The larger pSLC isn't because there are more flash sectors to write to as much as it is to compensate for running the eMMC in pSLC mode, and yield the same or larger effective capacity. So now you'll see (for example) the Storage capacity on the 6100 Base as

Storage: 21.3 GB eMMC

This is really a 64GB eMMC running in pSLC mode.

Running the eMMC in pSLC mode provides a write endurance far beyond just the capacity increase of the new eMMC.

I'll try to write up something more ... technical to explain why soon.

Now I see a number of people desoldering the eMMC to try to recover. I'm not saying that isn't necessary, but I will say that it should not be necessary. I'm working with the vendor here to attempt to establish 'why' and issue a remedy.

I also want to work with the customer base for existing (pre-SLC eMMC) units to establish a path to getting a working, supported NVMe drive installed in their Netgate device prior to failure of the eMMC.

[ablephri]

@jwt Hahaha yeah. I had to give it to a "professional" who used that heat gun. I am delighted with Netgate's great customer service, and I really appreciate the patience Gonzo had with me trying to fix my problems and also get the unit replaced

[tinfoilmatt]

This post is deleted!

[aivxtla]

@stephenw10 Out of curiosity any reason for the Innodidk 3TE6 being shipped with the SG6100/8200? Is it Silicom’s shipping default or Netgate’s selection?
https://www.texim-europe.com/getfile.ashx?id=131537
Seems endurance is only 93TBW for the 128 GB vs something like the Transcend MTE452T (or MTE352T DRAMless) also an industrial SSD with DRAM cache rated around 270TBW. https://cdn.transcend-info.com/products/images/modelpic/1164/Transcend-MTE452T2_202307.pdf

The Transcend model also seems cheaper like ~$86 (when I bought the 256GB) vs $90-100+ for the inndodisk 128 GB at least for consumers, along with lower power draw and faster overall (for 128/256). I take it steeper discounts in bulk on the Innodisk?

[w0w]

@aivxtla said in Another Netgate with storage failure, 6 in total so far:

Seems endurance is only 93TBW for the 128 GB

There’s no official TBW rating for the 128GB SSD version, but based on the type of flash used, I expect it to be similar to Transcend’s TBW.

[aivxtla]

@w0w it’s listed as 93TBW in the innodisk data sheet I linked, that texim link is to the official more detailed data sheet. It’s not listed in most places and hard to find normally. It’s not much different in endurance ~100TBW than the WD SN520 which is also cheaper but definitely less than the Transcend MTE352/452, I think the NAND controller also has impact in regards to things like curtailing write amplification etc. Then again some also warranty the TBW to less than max ability depending on sales tier like the WD Blue/Red SATA SSDs with same hw.

[w0w]

@aivxtla
Overall, this could simply be variability in how the manufacturer evaluates TBW. As far as I know, there is no strict standard defining how TBW should be tested or calculated. Some manufacturers focus on producing nice-looking numbers for the datasheet, while others prioritize long-term stability.
A real-world example: recently I ran into unexplained issues on a friend’s home PC. It took quite a while to track down the cause, which—unsurprisingly—turned out to be a 5-year-old SSD with about 88% wear reported. The total written data was still far below the rated TBW, and the drive was not at 100% wear, but the problems were severe enough to make the system practically unusable.
In the end, the datasheet even stated in a footnote that the specified TBW does not guarantee stable operation of the drive. Everything depends on the criteria chosen by the manufacturer. I don’t think TBW alone should be taken too seriously, although the controller may also play a role.

[Patch]

@w0w different technology and more than twice the difference in listed performance. That’s not something I would attribute to testing methodology with out real data suggesting otherwise.

Imo a much more likely explanation is Netgate were not focusing on SSD wear as a major limit to device reliability and brand reputation at that time. They are now and I would be surprised if they made the same decision in the future.

[TVTS]

I ended up on this thread after having THREEE SG1100s all die of the same issue, ALL IN THE LAST 60 days! I've used and loved pfsense for years, but I wasn't aware how big of a problem this issue was until recently. Super disappointing to find out... I've opened a ticket, but all of these devices are older than 1 year, so I'm gonna take a wild guess and say the netgate is gonna tell me to go pound sand. Insane that they are VERY aware of this issue yet have done nothing to rectify it in years... I've deployed over 20 1100's, mostly for small business with basic needs and simple networks. They have been great given how feature rich they are for the price tag. Id happily pay an extra $50 for these devices if they would take the time to use better storage. Fingers crossed that netgate will replace any of these failed devices, but seems unlikely..

[SteveITS]

@TVTS https://docs.netgate.com/pfsense/en/latest/releases/25-07.html#:~:text=Changed:%20Reduce%20writes%20to%20disk

[stephenw10]

Yup changes to address that went in in 25.07: https://redmine.pfsense.org/issues/16210

[chrcoluk]

Apologies @andrew_cb for not responding directly to your boot environments observations, I do agree with you that heavy use of ZFS snapshots hurts write amplification, as it ultimately consumes more space on the storage, it also increases fragmentation, which can affect how ZFS operates if it hits a high enough level.

If the storage is over spec'd such as a 256 gig SSD, then I expect boot environments can be comfortably handled, but perhaps more care on something small like 32 gigs.

Luckily the txg delay increase should mitigate the majority of issues.

[matthewfearnley]

@andrew_cb Hi, I have updated the widget code a bit.
/usr/local/www/widgets/widgets/emmc_health.widget.php

<?php
require_once("functions.inc");
require_once("guiconfig.inc");

define('SEND_ALERTS', false);            // Sending alerts not tested
define('AUTO_INSTALL_MMC_UTILS', false); // Don't presume to install packages on the system

// Function to retrieve eMMC health data
function get_emmc_health() {

    // Check if eMMC device exists
    if(!file_exists("/dev/mmcsd0rpmb")) {
        //return ["status" => "error", "message" => "/dev/mmcsd0rpmb not found."];
    }

    // Check for the mmc-utils binary and install if missing
    if(!file_exists("/usr/local/sbin/mmc")) {
        if (AUTO_INSTALL_MMC_UTILS) {
            exec("pkg install -y mmc-utils", $code);
            if ($code <> 0) {
                return ["status" => "error", "message" => "Failed to install mmc-utils."];
            }
        } else {
            return ["status" => "error", "message" => "mmc-utils not installed."];
        }
    }

    $cmd = "/usr/local/sbin/mmc extcsd read /dev/mmcsd0rpmb | egrep 'LIFE|EOL'";
    $output = shell_exec($cmd);

    if (!$output) {
        return ["status" => "error", "message" => "Failed to retrieve eMMC health data."];
    }

    // Explode the output into separate lines (assumes the lines appear in the right order with no false positives)
    $outputArray = explode("\n", $output);

    // Get the value of 'TYP_A' (SLC) wear
    preg_match('/.*TYP_A]:\s+(0x[0-9A-F]+)/i', $outputArray[0], $matchA);
    // Get the value of 'TYP_B' (MLC) wear
    preg_match('/.*TYP_B]:\s+(0x[0-9A-F]+)/i', $outputArray[1], $matchB);
    // Get the value of Pre-EOL information
    preg_match('/.*EOL_INFO]:\s+(0x[0-9A-F]+)/i', $outputArray[2], $matchC);

    // Convert the wear values from hex to decimal
    $lifeA = isset($matchA[1]) ? hexdec($matchA[1]) * 10 : null;
    $lifeB = isset($matchB[1]) ? hexdec($matchB[1]) * 10 : null;
    $eol   = isset($matchC[1]) ? hexdec($matchC[1]) : null;

    if (is_null($lifeA) || is_null($lifeB) || is_null($eol)) {
        return ["status" => "error", "message" => "Invalid eMMC health data."];
    }

    return ["status" => "ok", "lifeA" => $lifeA, "lifeB" => $lifeB, "eol" => $eol];
}

// Determine color class based on wear level
function get_wear_color_class($value) {
    if ($value < 70) {
        return "success"; // Green
    } elseif ($value < 90) {
        return "warning"; // Yellow
    } else {
        return "danger"; // Red
    }
}

function get_eol_color_class($value) {
    switch ($value) {
    case 0x01: return "success"; // Green
    case 0x02: return "warning"; // Yellow
    case 0x03: return "danger"; // Red
    }
    return "warning";

    if ($value < 1) {
        return "warning"; // Yellow
    } elseif ($value < 2) {
        return "success"; // Green
    } elseif ($value < 3) {
        return "warning"; // Yellow
    } else {
        return "danger"; // Red
    }
}

function get_eol_percent($value) {
    switch ($value) {
    case 0x01: return "< 80%";
    case 0x02: return "> 80%";
    case 0x03: return "> 90%";
    }
    return "Not defined";

    if ($value < 1) {
        return "Not defined";
    } elseif ($value < 2) {
        return "< 80%";
    } elseif ($value < 3) {
        return "> 80%";
    } else {
        return "> 90%";
    }
}

// Send email notification if wear level is critical
function send_emmc_alert($lifeA, $lifeB, $eol) {
    global $config;

    $eol_percent = get_eol_percent($eol);
    $subject = "[pfSense] eMMC Wear Level Warning";
    $message = "Warning: eMMC wear level is high!\n\n" .
               "Life A: {$lifeA}%\nLife B: {$lifeB}%\nEOL: {$eol_percent}\n\n" .
               "Consider replacing the storage device.";

    if ($lifeA >= 90 || $lifeB >= 90 || $eol >= 3) {
        notify_via_smtp($subject, $message);
    }
}

// Main program logic
// Get get the eMMC health data
$data = get_emmc_health();

if (SEND_ALERTS && $data["status"] === "ok") {
    // Send an email notification if eMMC is not healthy
    send_emmc_alert($data["lifeA"], $data["lifeB"], $data["eol"]);
}

// Format the data into HTML for display in the widget
?><div class="panel panel-default">
    <!--div class="panel-heading">
        <h3 class="panel-title">eMMC Disk Health (redundant header)</h3>
    </div-->
    <div class="panel-body">
        <?php if ($data["status"] === "error"): ?>
            <div class="alert alert-danger"><?php echo $data["message"]; ?></div>
        <?php else: ?>
            <table class="table">
                <tr>
                    <th>Type A Wear (Lower is better)</th>
                    <td class="bg-<?php echo get_wear_color_class($data['lifeA']); ?>"> <?php echo ($data['lifeA'] - 10) . "-" . $data['lifeA']; ?>%</td>
                </tr>
                <tr>
                    <th>Type B Wear (Lower is better)</th>
                    <td class="bg-<?php echo get_wear_color_class($data['lifeB']); ?>"> <?php echo ($data['lifeB'] - 10) . "-" . $data['lifeB']; ?>%</td>
                </tr>
                <tr>
                    <th>Pre-EOL (% of reserved blocks used)</th>
                    <td class="bg-<?php echo get_eol_color_class($data['eol']); ?>"> <?php echo get_eol_percent($data['eol']); ?></td>
                </tr>
            </table>
        <?php endif; ?>
    </div>
</div>

And a quick include to neaten up the title (which removes the need for the redundant panel heading):
/usr/local/www/widgets/include/emmc_health.inc

<?php // /usr/local/www/widgets/include/emmc_health.inc
$emmc_health_title = gettext("eMMC Disk Health");
?>

I changed the output to report percentage ranges, and added a row for the Pre-EOL info.

I've included the Pre-EOL information in the email notification, but I've not tested this functionality, so I've disabled sending emails with a define() near the top.

There was previously a function to automatically install emmc-utils, but it wasn't called. I've added the ability to install it automatically, but also left it disabled by default with a separate define().

I've also included error reporting for trivial cases like the device not existing or mmc not existing (or if applicable, failing to install).

One caveat is that it still assumes the LifeA/LifeB/Pre-EOL info will all appear in the output in that order (and no other lines slip in). If that assumption breaks, better parsing will be needed.

(In case there's any hope of including a widget like this in pfSense, all my changes are Public Domain.)