How to disable dying/dead eMMC?

Tendril

I upgraded my SG-5100 to an M.2 SATA drive. Recently, I've started seeing tons of the following errors in my logs (repeated 100s of times a day, with different offsets and lengths):

Jul 6 16:51:10	kernel		g_vfs_done():mmcsd0s1[WRITE(offset=20709376, length=4096)]error = 5
Jul 6 16:51:10	kernel		Error indicated: 1 Timeout

The system seems to still be trying to mount the eMMC (red hi-light is mine):

Filesystem                                          Size    Used   Avail Capacity  Mounted on
/dev/ufsid/60451ecc9365d53b                         111G     3.2G     99G     3%    /
devfs                                               1.0K     1.0K      0B   100%    /dev
/dev/md0                                            3.4M     208K    2.9M     6%    /var/run
/dev/mmcsd0s1                                       6.7G     5.2G    1.0G    84%    /tmp/mnt/cf
devfs                                               1.0K     1.0K      0B   100%    /var/dhcpd/dev

ls shows nothing.

[2.4.5-RELEASE][admin@xxxxxxxxxxxxxx]/: ls -la /tmp/mnt/cf/
total 0

fsck fails:

[2.4.5-RELEASE][admin@xxxxxxxxxxxxxx]/: fsck /dev/mmcsd0s1
** /dev/mmcsd0s1 (NO WRITE)

CANNOT READ BLK: 128
CONTINUE? [yn] y

THE FOLLOWING DISK SECTORS COULD NOT BE READ: 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143,

CANNOT READ BLK: 127
CONTINUE? [yn] y

THE FOLLOWING DISK SECTORS COULD NOT BE READ: 127,

Un-mounting the eMMC fails.

[2.4.5-RELEASE][admin@xxxxxxxxxxxxxx]/: umount /tmp/mnt/cf/
umount: unmount of /tmp/mnt/cf failed: Input/output error

What is the recommended way of completely wiping and/or disabling the old eMMC drive? Is it possible to physically remove it from the SG-5100?

Thanks!

stephenw10

If you re-installed to the m.2 SSD clean the eMMC should not be mounted at all.

Do that if you didn't.

It could still be mounted if you recovered a config from that during the install and have not rebooted.

Steve

Tendril

@stephenw10 I've rebooted many times since the M.2 installation. (I did the hardware upgrade months ago.)

I'm not sure where the eMMC mount is coming from. I guess pfSense automatically mounts all available storage devices?

stephenw10

No it does not mount any additional drives by default.

The fact it's mounted where it is looks like a config recovery was attempted.
Check the boot log for output from the external config loader. That should ignore eMMC though.

Check /etc/fstab.

Steve

Tendril

@stephenw10 I did a config recovery (after reinstalling pfSense) when I upgraded the storage, but I've rebooted as recently as five days ago.

My /etc/fstab:

[2.4.5-RELEASE][admin@xxxxxxxxxxxxxx]/: cat /etc/fstab
/dev/ufsid/60451ecc9365d53b     /       ufs     rw,noatime      1       1
/dev/label/swap0                none    swap    sw              0       0

Is it some sort of autofs weirdness? Maybe from /etc/autofs/special_media ?

These messages are in the boot log:

[2.4.5-RELEASE][admin@xxxxxxxxxxxxxx]/etc/autofs: cat /var/run/dmesg.boot | more

<snip>

sdhci_pci0: <Intel Denverton eMMC 5.0 Controller> mem 0xdff9a000-0xdff9afff,0xdff99000-0xdff99fff irq 16 at device 28.0 on pci0
sdhci_pci0: 1 slot(s) allocated
mmc0: <MMC/SD bus> on sdhci_pci0

<snip>

mmcsd0: 8GB <MMCHC M32508 5.2 SN 398280AD MFG 06/2018 by 112 0x0000> at mmc0 50.0MHz/8bit/65535-block
mmcsd0boot0: 4MB partion 1 at mmcsd0
mmcsd0boot1: 4MB partion 2 at mmcsd0
ses0 at ahciem0 bus 0 scbus1 target 0 lun 0
mmcsd0rpmb: 4MB partion 3 at mmcsd0
ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ses1 at ahciem1 bus 0 scbus3 target 0 lun 0
ses0: (none) in 'Slot 00', SATA Slot: scbus0 target 0
ses1: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses1: SEMB SES Device
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <TS128GMTS430S R0427DNR> ACS-3 ATA SATA 3.x device
ada0: Serial Number G119420525
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 122104MB (250069680 512 byte sectors)
Trying to mount root from ufs:/dev/ufsid/60451ecc9365d53b [rw,noatime]...
WARNING: / was not properly dismounted
mmcsd0: Error indicated: 1 Timeout
g_vfs_done():mmcsd0s1[WRITE(offset=163840, length=32768)]error = 5

<snip>

"WARNING: / was not properly dismounted" --> Odd. I shut the system down through the web interface.

I've pinged Netgate support too and pointed them to this thread, so I hope they can comment/join in the fun

Ian

stephenw10

Yeah, nothing there. I was also looking your ticket but the logs and message buffer have been spammed by the read errors so nothing can be seen.
I'd like to see the boot logs to 'bootup complete' from a clean boot. That would show if the ECL is mounting anything.

Steve

Gertjan

@tendril said in How to disable dying/dead eMMC?:

Odd. I shut the system down through the web interface.

and you asked it to take care of things ;) ?

Tendril

@gertjan I have not tried this as the eMMC is not even supposed to be in-use. I'll have to schedule time for a reboot + check. Even though the log is being spammed with errors, the SG-5100 is working and many people need the VPN gateway to work remotely