SG-1100 recovery fails

[seanmcb]

I have an SG-1100 with an apparently totally corrupt filesystem. I ran fsck -fy - at least 20 times but no luck. Fine, I'll just erase it all.

Marvell> usb reset
Marvell> run usbrecovery

The recovery procedure runs as expected:

Selected eMMC device: mmcsd0 - ZFS

Are you sure you want to continue ? (y/N) y

Erasing the eMMC contents...
Writing the firmware to eMMC...

It eventually tells me to power down, disconnect the USB disk, and power on. I do. It then boots, but ends up acting like the reinstall didn't happen. It goes into an fsck loop, fails, and ends with:

ERROR: Impossible to mount filesystem, use interactive shell to attempt to recover it

Despite me having chosen ZFS for the reinstall, zfs pool shows no pools. It's as if the recovery procedure didn't work.

A few things jump out to me in the boot output:

Card did not respond to voltage select!
    12729   armada-3720-sg1100.dtb
    12725   armada-3720-netgate-1100.dtb

Scanning disk sdhci@d0000.blk...
Disk sdhci@d0000.blk not ready

mmc0: No compatible cards found on bus

mmc1: Failed to set VCCQ for card at relative address 2

Trying to mount root from ufs:/dev/diskid/DISK-2D431253s3 [rw,noatime]...

WARNING: / was not properly dismounted

** SU+J Recovering /dev/diskid/DISK-2D431253s3
** Reading 11730944 byte journal from inode 4.
** Building recovery table.

Any suggestions?

[rcoleman-netgate]

What happens when you choose UFS instead of the ZFS default?

[seanmcb]

@rcoleman-netgate Same result. Seems to not actually install / erase anything.

[seanmcb]

I also just had the idea to try smartctl, but...

# smartctl -a /dev/mmcsd0
smartctl 7.3 2022-02-28 r5338 [FreeBSD 12.3-STABLE arm64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

/dev/mmcsd0: Unable to detect device type
Please specify device type with the -d option.

[rcoleman-netgate]

@seanmcb the eMMC is not a S.M.A.R.T.-compatible device.

How old is this firewall? What packages were you running on it previously?

[seanmcb]

@rcoleman-netgate It was purchased December 2019, so not very old I'd say. The only package I was actually using was pfBlocker-NG. Suricata was installed but unused. It had been running fine for years and then Thursday I just suddenly had no internet. After getting the console working the next day, I saw the fsck complaints and ever since have just been trying to wipe it and start clean.

[rcoleman-netgate]

@seanmcb I would suspect that your eMMC has failed. Did you open a ticket for the latest release? If so can you enable logging on your console client, re-attempt the image process, and send the log file on the ticket? I can look at it and see if it is an eMMC in read-only mode.

[seanmcb]

@rcoleman-netgate I had also tried to rm -rf /, and it did report itself as a read-only file system.

I've created the ticket and sent the requested log. Thanks.