<?php
require_once("functions.inc");
require_once("guiconfig.inc");

define('SEND_ALERTS', false);            // Sending alerts not tested
define('AUTO_INSTALL_MMC_UTILS', false); // Don't presume to install packages on the system

// Function to retrieve eMMC health data
function get_emmc_health() {

    // Check if eMMC device exists
    if(!file_exists("/dev/mmcsd0rpmb")) {
        //return ["status" => "error", "message" => "/dev/mmcsd0rpmb not found."];
    }

    // Check for the mmc-utils binary and install if missing
    if(!file_exists("/usr/local/sbin/mmc")) {
        if (AUTO_INSTALL_MMC_UTILS) {
            exec("pkg install -y mmc-utils", $code);
            if ($code <> 0) {
                return ["status" => "error", "message" => "Failed to install mmc-utils."];
            }
        } else {
            return ["status" => "error", "message" => "mmc-utils not installed."];
        }
    }

    $cmd = "/usr/local/sbin/mmc extcsd read /dev/mmcsd0rpmb | egrep 'LIFE|EOL'";
    $output = shell_exec($cmd);

    if (!$output) {
        return ["status" => "error", "message" => "Failed to retrieve eMMC health data."];
    }

    // Explode the output into separate lines (assumes the lines appear in the right order with no false positives)
    $outputArray = explode("\n", $output);

    // Get the value of 'TYP_A' (SLC) wear
    preg_match('/.*TYP_A]:\s+(0x[0-9A-F]+)/i', $outputArray[0], $matchA);
    // Get the value of 'TYP_B' (MLC) wear
    preg_match('/.*TYP_B]:\s+(0x[0-9A-F]+)/i', $outputArray[1], $matchB);
    // Get the value of Pre-EOL information
    preg_match('/.*EOL_INFO]:\s+(0x[0-9A-F]+)/i', $outputArray[2], $matchC);

    // Convert the wear values from hex to decimal
    $lifeA = isset($matchA[1]) ? hexdec($matchA[1]) * 10 : null;
    $lifeB = isset($matchB[1]) ? hexdec($matchB[1]) * 10 : null;
    $eol   = isset($matchC[1]) ? hexdec($matchC[1]) : null;

    if (is_null($lifeA) || is_null($lifeB) || is_null($eol)) {
        return ["status" => "error", "message" => "Invalid eMMC health data."];
    }

    return ["status" => "ok", "lifeA" => $lifeA, "lifeB" => $lifeB, "eol" => $eol];
}

// Determine color class based on wear level
function get_wear_color_class($value) {
    if ($value < 70) {
        return "success"; // Green
    } elseif ($value < 90) {
        return "warning"; // Yellow
    } else {
        return "danger"; // Red
    }
}

function get_eol_color_class($value) {
    switch ($value) {
    case 0x01: return "success"; // Green
    case 0x02: return "warning"; // Yellow
    case 0x03: return "danger"; // Red
    }
    return "warning";

    if ($value < 1) {
        return "warning"; // Yellow
    } elseif ($value < 2) {
        return "success"; // Green
    } elseif ($value < 3) {
        return "warning"; // Yellow
    } else {
        return "danger"; // Red
    }
}

function get_eol_percent($value) {
    switch ($value) {
    case 0x01: return "< 80%";
    case 0x02: return "> 80%";
    case 0x03: return "> 90%";
    }
    return "Not defined";

    if ($value < 1) {
        return "Not defined";
    } elseif ($value < 2) {
        return "< 80%";
    } elseif ($value < 3) {
        return "> 80%";
    } else {
        return "> 90%";
    }
}

// Send email notification if wear level is critical
function send_emmc_alert($lifeA, $lifeB, $eol) {
    global $config;

    $eol_percent = get_eol_percent($eol);
    $subject = "[pfSense] eMMC Wear Level Warning";
    $message = "Warning: eMMC wear level is high!\n\n" .
               "Life A: {$lifeA}%\nLife B: {$lifeB}%\nEOL: {$eol_percent}\n\n" .
               "Consider replacing the storage device.";

    if ($lifeA >= 90 || $lifeB >= 90 || $eol >= 3) {
        notify_via_smtp($subject, $message);
    }
}

// Main program logic
// Get get the eMMC health data
$data = get_emmc_health();

if (SEND_ALERTS && $data["status"] === "ok") {
    // Send an email notification if eMMC is not healthy
    send_emmc_alert($data["lifeA"], $data["lifeB"], $data["eol"]);
}

// Format the data into HTML for display in the widget
?><div class="panel panel-default">
    <!--div class="panel-heading">
        <h3 class="panel-title">eMMC Disk Health (redundant header)</h3>
    </div-->
    <div class="panel-body">
        <?php if ($data["status"] === "error"): ?>
            <div class="alert alert-danger"><?php echo $data["message"]; ?></div>
        <?php else: ?>
            <table class="table">
                <tr>
                    <th>Type A Wear (Lower is better)</th>
                    <td class="bg-<?php echo get_wear_color_class($data['lifeA']); ?>"> <?php echo ($data['lifeA'] - 10) . "-" . $data['lifeA']; ?>%</td>
                </tr>
                <tr>
                    <th>Type B Wear (Lower is better)</th>
                    <td class="bg-<?php echo get_wear_color_class($data['lifeB']); ?>"> <?php echo ($data['lifeB'] - 10) . "-" . $data['lifeB']; ?>%</td>
                </tr>
                <tr>
                    <th>Pre-EOL (% of reserved blocks used)</th>
                    <td class="bg-<?php echo get_eol_color_class($data['eol']); ?>"> <?php echo get_eol_percent($data['eol']); ?></td>
                </tr>
            </table>
        <?php endif; ?>
    </div>
</div>

And a quick include to neaten up the title (which removes the need for the redundant panel heading):
/usr/local/www/widgets/include/emmc_health.inc

<?php // /usr/local/www/widgets/include/emmc_health.inc
$emmc_health_title = gettext("eMMC Disk Health");
?>

I changed the output to report percentage ranges, and added a row for the Pre-EOL info.

I've included the Pre-EOL information in the email notification, but I've not tested this functionality, so I've disabled sending emails with a define() near the top.

There was previously a function to automatically install emmc-utils, but it wasn't called. I've added the ability to install it automatically, but also left it disabled by default with a separate define().

I've also included error reporting for trivial cases like the device not existing or mmc not existing (or if applicable, failing to install).

One caveat is that it still assumes the LifeA/LifeB/Pre-EOL info will all appear in the output in that order (and no other lines slip in). If that assumption breaks, better parsing will be needed.

(In case there's any hope of including a widget like this in pfSense, all my changes are Public Domain.)

If the storage is over spec'd such as a 256 gig SSD, then I expect boot environments can be comfortably handled, but perhaps more care on something small like 32 gigs.

Luckily the txg delay increase should mitigate the majority of issues.

Imo a much more likely explanation is Netgate were not focusing on SSD wear as a major limit to device reliability and brand reputation at that time. They are now and I would be surprised if they made the same decision in the future.

Seems endurance is only 93TBW for the 128 GB

There’s no official TBW rating for the 128GB SSD version, but based on the type of flash used, I expect it to be similar to Transcend’s TBW.

The Transcend model also seems cheaper like ~$86 (when I bought the 256GB) vs $90-100+ for the inndodisk 128 GB at least for consumers, along with lower power draw and faster overall (for 128/256). I take it steeper discounts in bulk on the Innodisk?

A larger flash device is only part of the solution. Netgate moved the entire product line to a larger eMMC device running in pSLC mode much earlier this year, without a price increase.

The larger pSLC isn't because there are more flash sectors to write to as much as it is to compensate for running the eMMC in pSLC mode, and yield the same or larger effective capacity. So now you'll see (for example) the Storage capacity on the 6100 Base as

Storage: 21.3 GB eMMC

This is really a 64GB eMMC running in pSLC mode.

Running the eMMC in pSLC mode provides a write endurance far beyond just the capacity increase of the new eMMC.

I'll try to write up something more ... technical to explain why soon.

Now I see a number of people desoldering the eMMC to try to recover. I'm not saying that isn't necessary, but I will say that it should not be necessary. I'm working with the vendor here to attempt to establish 'why' and issue a remedy.

I also want to work with the customer base for existing (pre-SLC eMMC) units to establish a path to getting a working, supported NVMe drive installed in their Netgate device prior to failure of the eMMC.

The diag LEDs look like a normal POST.

Did you try both console ports?

note to others in this thread: I'm working with him off-thread to get him a replacement in exchange for his broken unit.

the console used to work until the device didn't respond again.

Ok, so chances are good that the cable is ok.
Removing the EMMC was a guess then ?

The '6100' is like a PC : removing a drive won't stop the system from booting. You need the console access to select from what device : an EMMC, or some SSD drive or even a USB key drive. Whatever happens with these "drives", the console will give output.

@ablephri said in Another Netgate with storage failure, 6 in total so far:

the computer can detect the COM port but still not be functional?

On the computer side, the USB plug, and the USB chip build into the cable, will power up, and communicate with the USB host (the USB circuitry build in the PC) so the 'PC' knows that an USB device is present, and with the help of the 'ID' of the USB cable chip, it will know what kind of driver it needs. In this case : a serial connection driver, also known as a 'COM' port (in the PC world) type of device. The serial connection itself, 3 wires, a common ground, a transmit and receive line, can not be auto detected like what you see if something is "plug and play". A serial connection is something from te past century.
So plugin in the cable makes the COM port avaible to the PC, but won't tell you anything about the presence of a connection = the receive line going up and down in a fixed frequency (115000 hz - you checked this bit rate of you port ? ) so a serial connection is establish.
That's also you final test : on the console port, if you can find these 'TX' and RX' traces, locate the TX line.
If you see the TX line go up and down frequently (shortly) after power up, you know the 6100 BIOS is outputting text. If it stays silent ........

You've used the console access before? Was it working before on this device? Is your console cable ok?

Please, the console used to work until the device didn't respond again. But is it possible that the computer can detect the COM port but still not be functional? I am really frustrated. The money wasted is so sad to think about.

I had no console access

By correctly (without creating shorts or cuts in the traces) removing the EMMC and the BIOS can't find it anymore, and will look for other potential "drives" so it can boot from it.

A broken EMMC, imho, won't stop the processor from running the BIOS and communicate with you over the console.

No console is scary : no real proof the processor runs.

You've used the console access before ? Was it working before on this device ? Is your console cable ok ?

If the power rail is good, but for some reason it sinks the (example) 'common' clock signal to ground, then your processor can't work anymore.
Some good equipment (scoop, or even logic analyser) and schemas and the knowledge how to (electrically) debug this is now needed. Humm, sorry, I start to feel pessimistic about this.

IMG_9123.jpeg

Wow...
very interesting thread.
I found this just yesterday and it takes me half the night to to read it from start to end .

same here, just finished.

I confirm, I had no idea. Don't recall seeing it mentioned in the marketing data, docs, nor by the gurus at YT.
I've chosen NG's hw specifically for the reliability and minimal maintenance. Knowing this probably would go 'max'.

Adjusted the timings, no sync for tmp. Soon on my way for new ssd. Thanks.

Can you advice me how can I check emmc healt in SG2100?

And BTW: the SMART values shows the SSD is still at 94% lifetime, so I can run the device may be until a 4300/4400/4x00 is availabe .

Regards