SG-3100 Died With Bad eMMC

jsylvia007

OK... So...

The recovery works fine, I think:

Type the name of the destination device (mmcsd0,ada0) or press the Enter key to install on mmcsd0: ada0

Selected M.2 device: ada0

Are you sure you want to continue ? (y/N) Y

Erasing the M.2 contents...
Writing the firmware to M.2...
(this may take a few minutes to complete)
0+32768 records in
512+0 records out
2147483648 bytes transferred in 30.763006 secs (69807341 bytes/sec)
Fixing M.2 labels...
Fixing u-boot environment...
dd: /dev/flash/spi0: No such file or directory
Invalid input file: /root/env
stat: No such file or directory
Invalid input file: /root/newenv
stat: No such file or directory

Done!

The system will halt now, please power off and remove the firmware
recovery storage device.




Shutdown NOW!
shutdown: [pid 1926]
Aug 13 19:41:29 Stopping cron.
Waiting for PIDS: 1860.
Stopping devd.
Waiting for PIDS: 1611.
Writing entropy file: .
Writing early boot entropy file: .
Terminated
.
Aug 13 19:41:30 3100-recovery syslogd: exiting on signal 15
Waiting (max 60 seconds) for system process `vnlru' to stop... done

Waiting (max 60 seconds) for system process `syncer' to stop... Syncing disks, vnodes remaining... 0 0 done
All buffers synced.
Uptime: 1m14s

The operating system has halted.
Power cycle or reset to reboot.

Then I issued the command to change the boot from mmc to m2... That worked and it looks like it now tries to boot, BUT, all I'm left with is a flashing blue light for the green circle in the front, and the console says:

Reset SCSI
AHCI init for unit0
Target spinup took 0 ms.
SATA link 1 timeout.

Error: SCSI Controller(s) 1B4B:9215 1B4B:9235 not found
scanning bus for devices...
  Device 0: (0:0) Vendor: ATA Prod.: ORICO Rev: X071
            Type: Hard Disk
            Capacity: 244198.3 MB = 238.4 GB (500118192 x 512)
Found 1 device(s).
** Unrecognized filesystem type **
## Starting application at 0x00200000 ...

And it just hangs there... Someone with some better knowledge help me out maybe? I dunno what I'm supposed to be seeing, but perhaps there's an issue. The m.2 SATA is brand new and was apparently recoginzed, etc.

stephenw10

Was that drive new? It's possible it had some older filesystem on it that needs to be completely removed.

You should see something like:

Board configuration detected:
Net:   
|  port  | Interface | PHY address  |
|--------|-----------|--------------|
| egiga0 |   RGMII   |     0x00     |
| egiga1 |   RGMII   |     0x01     |
| egiga2 |   SGMII   |   In-Band    |
egiga0 [PRIME], egiga1, egiga2
Hit any key to stop autoboot:  0 

Reset SCSI
AHCI init for unit0
Target spinup took 0 ms.
SATA link 1 timeout.

Error: SCSI Controller(s) 1B4B:9215 1B4B:9235 not found
scanning bus for devices...
  Device 0: (0:0) Vendor: ATA Prod.: NT-32 Rev: 1.09
            Type: Hard Disk
            Capacity: 30533.8 MB = 29.8 GB (62533296 x 512)
Found 1 device(s).
reading ubldr.bin
228648 bytes read in 11 ms (19.8 MiB/s)
## Starting application at 0x00200000 ...
Consoles: U-Boot console  
Compatible U-Boot API signature found @0x7f4dc280

FreeBSD/armv6 U-Boot loader, Revision 1.2
(Tue Jul 10 10:26:23 EDT 2018 root@buildbot3)

DRAM: 2048MB
Number of U-Boot devices: 4
U-Boot env: loaderdev not set, will probe all devices.
Found U-Boot device: disk
  Probing all disk devices...
  Checking unit=0 slice=<auto> partition=<auto>...disk0: read failed, error=1

  Checking unit=1 slice=<auto> partition=<auto>... good.
Booting from disk1s2a:
Loading /boot/defaults/loader.conf
console comconsole is invalid!
no valid consoles!
Available consoles:
    uboot
/boot/kernel/kernel text=0x1b4 text=0x84e20c text=0xa04440 data=0xac168 data=0x0+0x4c000 syms=[0x4+0xae330+0x4+0x11aaee]
|
Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...               
/boot/dtb/armada-385-80300-0148-G00-X100.dtb size=0x67f4
Loaded DTB from file 'armada-385-80300-0148-G00-X100.dtb'.
Kernel entry at 0x400200...
Kernel args: (null)
SOC: Marvell 88F6820, TClock 250MHz, Frequency 1600MHz
  Instruction cache prefetch enabled, data cache prefetch disabled
---<<BOOT>>---
GDB: debug ports: uart

FSC830

@jsylvia007 Hm, its long time ago when I did that, and actually I cant remember if I changed any boot environment values.
When looking in manual I found "By default, the M.2 SATA drive will then be the first drive
recognized by pfSense software" and as far as I remember that was all, re-installing the software at ada0...
Unfortunately I do not have access from my current location to the laptop I did connect to the console and logged all output when doing this.
Where did you find an advise to change boot environment?

Regards

stephenw10

You do have to manually update the uboot env when swtching boot device. Some while back the recovery installer was able to set it automatically but no longer does.

jsylvia007

@FSC830 - Its in the instructions now to perform it.

jsylvia007

@stephenw10 - It was a brand-new-in-package drive. Any chance I can dd the drive from the installer or recovery somehow? I don't want to pull it all apart again.

stephenw10

For reference:

Marvell>> printenv bootcmd
bootcmd=run setLED; run m2boot;
Marvell>> printenv m2boot 
m2boot=scsi reset; fatload scsi 0:$m2dev 200000 ubldr.bin; go 200000;

So it's trying to load the boot loader from a fat partition on the drive but instead finding an unknown filesystem there.

You can try:

Marvell>> scsi init
AHCI init for unit0
Target spinup took 0 ms.
SATA link 1 timeout.

Error: SCSI Controller(s) 1B4B:9215 1B4B:9235 not found
scanning bus for devices...
  Device 0: (0:0) Vendor: ATA Prod.: NT-32 Rev: 1.09
            Type: Hard Disk
            Capacity: 30533.8 MB = 29.8 GB (62533296 x 512)
Found 1 device(s).
Marvell>> scsi part 0

Partition Map for SCSI device 0  --   Partition Type: DOS

Part	Start Sector	Num Sectors	UUID		Type
  1	4158      	69993     	00000000-01	0c Boot
  2	74151     	62459145  	00000000-02	a5

Then dig deeper with:

Marvell>> fatls scsi 0:1  
   870156   u-boot.bin 
   949520   u-boot.mmc 
   228648   ubldr.bin 

3 file(s), 0 dir(s)

jsylvia007

@stephenw10 -

So... that worked? I typed out those commands and it showed me exactly what was posted...

stephenw10

Huh so it has the fat partiton on the drive and it contains the file?

I would try manually running the command in m2boot then. See exactly where it fails.

stephenw10

Oh wait what do you have m2dev set to?

Marvell>> printenv m2dev
m2dev=1

And which slot did you add the drive to?

Try setting it to 1 (or 0 if it;s already 1).

jsylvia007

@stephenw10 - WINNER WINNER CHICKEN DINNER!!!!!!!!!!!!

jsylvia007

Wait... Spoke too soon... Setting it to 1 looks like it's trying to boot / read from the eMMC and it's REALLY unhappy about that... Any way to stop it from doing that?

jsylvia007

It says "External Config Loader Started... mmcsd0" and then it goes off to never, never land, which is exactly what I would expect because the MMC is TOAST.

stephenw10

Hmm, that's painful. It should eventually get past that. Once it does I would try to clear the partitions from the eMMC. If it allows that....

jsylvia007

@stephenw10 - It did... It failed into a single user mode, I was able to issue a DD to the eMMC and after a reboot, it no longer sees the MMC device. Note that the DD made it VERY angry also, but I let it run for a bit to make sure that it destroyed as much of the partition table at the beginning as possible. It appears to be happy now!

I'm going to run another factory reset, etc., just to make sure it doesn't get wonky.

Note that when it's in a state with the failed eMMC, the USB terminal is also pretty unstable.

Of note for other folks who stumble across this... Believe it or not, the config file was still there and intact. Before I started all of this, I saw the config, was able to cat it out with the device in read-only, but couldn't get access to it. I couldn't remount it, and because it was read-only, I couldn't mount a USB device on top of it, but I was still able to get it.

I created a tmpfs on top of /mnt (the exact command escapes me, but I think it was mount -t tmpfs tmpfs /mnt. I think I had to make it read/write, but I can't remember off the top of my head how I did that piece (it might have been mount -u /mnt). From there, I was able to mount a USB stick to /mnt, and then copy the file from /cf/config/config.xml to the USB stick.

It took me a LONG time to figure out the tmpfs "hack", but it worked!

stephenw10

As a last resort you can usually just cat the config file and copy/paste it from the console directly.

FSC830

The uboot commands sound somehow familiar to me, but actually I cant remember if I needed to do so.
I cant even remember which version of pfSense was the curretn at this time when adding the m.2 SATA, 21.x or 22.x?

Regards