How to disable dying/dead eMMC?
I upgraded my SG-5100 to an M.2 SATA drive. Recently, I've started seeing tons of the following errors in my logs (repeated 100s of times a day, with different offsets and lengths):
Jul 6 16:51:10 kernel g_vfs_done():mmcsd0s1[WRITE(offset=20709376, length=4096)]error = 5 Jul 6 16:51:10 kernel Error indicated: 1 Timeout
The system seems to still be trying to mount the eMMC (red hi-light is mine):
Filesystem Size Used Avail Capacity Mounted on /dev/ufsid/60451ecc9365d53b 111G 3.2G 99G 3% / devfs 1.0K 1.0K 0B 100% /dev /dev/md0 3.4M 208K 2.9M 6% /var/run /dev/mmcsd0s1 6.7G 5.2G 1.0G 84% /tmp/mnt/cf devfs 1.0K 1.0K 0B 100% /var/dhcpd/dev
[2.4.5-RELEASE][admin@xxxxxxxxxxxxxx]/: ls -la /tmp/mnt/cf/ total 0
[2.4.5-RELEASE][admin@xxxxxxxxxxxxxx]/: fsck /dev/mmcsd0s1 ** /dev/mmcsd0s1 (NO WRITE) CANNOT READ BLK: 128 CONTINUE? [yn] y THE FOLLOWING DISK SECTORS COULD NOT BE READ: 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, CANNOT READ BLK: 127 CONTINUE? [yn] y THE FOLLOWING DISK SECTORS COULD NOT BE READ: 127,
Un-mounting the eMMC fails.
[2.4.5-RELEASE][admin@xxxxxxxxxxxxxx]/: umount /tmp/mnt/cf/ umount: unmount of /tmp/mnt/cf failed: Input/output error
What is the recommended way of completely wiping and/or disabling the old eMMC drive? Is it possible to physically remove it from the SG-5100?
If you re-installed to the m.2 SSD clean the eMMC should not be mounted at all.
Do that if you didn't.
It could still be mounted if you recovered a config from that during the install and have not rebooted.
@stephenw10 I've rebooted many times since the M.2 installation. (I did the hardware upgrade months ago.)
I'm not sure where the eMMC mount is coming from. I guess pfSense automatically mounts all available storage devices?
No it does not mount any additional drives by default.
The fact it's mounted where it is looks like a config recovery was attempted.
Check the boot log for output from the external config loader. That should ignore eMMC though.
@stephenw10 I did a config recovery (after reinstalling pfSense) when I upgraded the storage, but I've rebooted as recently as five days ago.
[2.4.5-RELEASE][admin@xxxxxxxxxxxxxx]/: cat /etc/fstab /dev/ufsid/60451ecc9365d53b / ufs rw,noatime 1 1 /dev/label/swap0 none swap sw 0 0
Is it some sort of autofs weirdness? Maybe from /etc/autofs/special_media ?
These messages are in the boot log:
[2.4.5-RELEASE][admin@xxxxxxxxxxxxxx]/etc/autofs: cat /var/run/dmesg.boot | more <snip> sdhci_pci0: <Intel Denverton eMMC 5.0 Controller> mem 0xdff9a000-0xdff9afff,0xdff99000-0xdff99fff irq 16 at device 28.0 on pci0 sdhci_pci0: 1 slot(s) allocated mmc0: <MMC/SD bus> on sdhci_pci0 <snip> mmcsd0: 8GB <MMCHC M32508 5.2 SN 398280AD MFG 06/2018 by 112 0x0000> at mmc0 50.0MHz/8bit/65535-block mmcsd0boot0: 4MB partion 1 at mmcsd0 mmcsd0boot1: 4MB partion 2 at mmcsd0 ses0 at ahciem0 bus 0 scbus1 target 0 lun 0 mmcsd0rpmb: 4MB partion 3 at mmcsd0 ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device ses0: SEMB SES Device ses1 at ahciem1 bus 0 scbus3 target 0 lun 0 ses0: (none) in 'Slot 00', SATA Slot: scbus0 target 0 ses1: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device ses1: SEMB SES Device ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: <TS128GMTS430S R0427DNR> ACS-3 ATA SATA 3.x device ada0: Serial Number G119420525 ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 122104MB (250069680 512 byte sectors) Trying to mount root from ufs:/dev/ufsid/60451ecc9365d53b [rw,noatime]... WARNING: / was not properly dismounted mmcsd0: Error indicated: 1 Timeout g_vfs_done():mmcsd0s1[WRITE(offset=163840, length=32768)]error = 5 <snip>
"WARNING: / was not properly dismounted" --> Odd. I shut the system down through the web interface.
I've pinged Netgate support too and pointed them to this thread, so I hope they can comment/join in the fun
Yeah, nothing there. I was also looking your ticket but the logs and message buffer have been spammed by the read errors so nothing can be seen.
I'd like to see the boot logs to 'bootup complete' from a clean boot. That would show if the ECL is mounting anything.
@gertjan I have not tried this as the eMMC is not even supposed to be in-use. I'll have to schedule time for a reboot + check. Even though the log is being spammed with errors, the SG-5100 is working and many people need the VPN gateway to work remotely