SG-1100 unable to perform usbrecovery

C8 · Jun 3, 2024, 3:37 AM

I've had a SG-1100 sitting on the shelf for a while after it suddenly started rebooting itself and I was pressed for time and unable to spend any time with it.

So this past weekend I rolled up my sleeves and fixed the boot loop issue (repeated fchk's) and thought while I'm at I might as well do the (very likely) disruptive version update broadcast end 2023 so I downloaded the online installer memstick files and flashed to usb drive.

Extracted from netgate-installer-aarch64.img.gz (sha256 checked out) - pfSense-plus-installer-24.02-BETA2-aarch64-20240312-0600.img

I dutifully rebooted the system from console and initiated "run usbrecovery" and so far no luck, and after several attempts with different flash drives (Kingston 32GB, brand new SanDisk 64GB and Transcend 256GB), different flash utilities (rufus, etcher), different terminal apps (termite, putty), different front-USB ports I'm no closer to an update and infact the SG-1100 is unable to boot as it stands. Yes, I have tried "usb reset" on occasion but made no difference.

The general flow (latest) consists of the following...

U-Boot 2017.03-armada-17.10.2-g6a6581a-dirty (Nov 01 2018 - 16:04:57 -0300)

Model: Marvell Armada 3720 Community Board ESPRESSOBin
CPU @ 1200 [MHz]
L2 @ 800 [MHz]
TClock @ 200 [MHz]
DDR @ 750 [MHz]
DRAM: 1 GiB
U-Boot DT blob at : 000000003f716298
Comphy-0: USB3 5 Gbps
Comphy-1: PEX0 2.5 Gbps
Comphy-2: SATA0 6 Gbps
SATA link 0 timeout.
AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
flags: ncq led only pmp fbss pio slum part sxs
PCIE-0: Link down
MMC: sdhci@d0000: 0, sdhci@d8000: 1
SF: Detected mx25u3235f with page size 256 Bytes, erase size 64 KiB, total 4 MiB
Net: eth0: neta@30000 [PRIME]
Hit any key to stop autoboot: 2
0
Marvell>> run usbrecovery
run usbrecovery
sdhci_transfer_data: Error detected in status(0x408000)!
** First descriptor is NOT a primary desc on 1:1 **
sdhci_transfer_data: Error detected in status(0x408000)!
** First descriptor is NOT a primary desc on 1:1 **
switch to partitions #0, OK
mmc1(part 0) is current device

MMC erase: dev # 1, block # 0, count 4194304 ... 4194304 blocks erased: OK
starting USB...
USB0: Register 2000104 NbrPorts 2
Starting the controller
USB XHCI 1.00
USB1: USB EHCI 1.00
scanning bus 0 for devices... 1 USB Device(s) found
scanning bus 1 for devices... 2 USB Device(s) found
scanning usb for storage devices... 1 Storage Device(s) found
reading efi/boot/bootaa64.efi
2097152 bytes read in 81 ms (24.7 MiB/s)
reading armada-3720-sg1100.dtb
18022 bytes read in 22 ms (799.8 KiB/s)

Starting EFI application at 05000000 ...

Card did not respond to voltage select!
mmc_init: -95, time 41
Scanning disk sdhci@d8000.blk...
Scanning disk usb_mass_storage.lun0...
Found 5 disks

at this point it stops and output depends on terminal app - putty cursor jumps back up the screen, termite dumps a whole lot of gibberish starting with [18t[?25h[0;0fC[?25ho[?25hn[?25hs[?25ho[?25hl[?25he[?25hs[?25h:[?25h [?25hE[?25hF[?25hI[?25h [?25hc[?25ho[?25hn[?25hs[?25ho[?25hl[?25he[?25h [?25h [?25h

I am able to initiate further commands like reboot, poweroff, etc. but obviously limited to Marvell prompt commands.

I have read in some forums this could also relate to faulty eMMC storage in which case there might (in theory) be some way to boot from USB, but I was unable to find any mention/documentation on achieving this feat.

I would be grateful for an explanation, and/or suggestion of what can be done from here to recover any use of the SG-1100?

Thank you

SteveITS · Jun 3, 2024, 4:30 AM

@C8 I think you’re using the old instructions with the new installer. Boot the installer and it should walk you through it.
https://docs.netgate.com/pfsense/en/latest/install/netinstaller.html

stephenw10 · Jun 3, 2024, 3:11 PM

That's not an eMMC issue. Most likely it's because it's still running an old uboot version and requires the compat image to boot. That's fixed in the next version of the installer but until then open a TAC ticket and request the compat recovery image:
https://www.netgate.com/tac-support-request

Steve

C8 · Jun 4, 2024, 1:05 PM

@SteveITS thank you for the recommendation, but I was following the 'install' instructions (re-install) for SG-1100 that got me into this state... I logged a support request for the 'compat' firmware to get it up & running until a compatible installer can be finalised.

C8 · Jun 4, 2024, 1:11 PM

@stephenw10 Done! case 2794882681

I assume the 'compat' image will get me up & running only, or will it also enable the current online installer to update from there?

Thank you

stephenw10 · Jun 4, 2024, 1:16 PM

Once you boot the compat recovery it should update uboot (or you can force it to update) which will then allow the current Net Installer to boot.

C8 · Jun 4, 2024, 1:23 PM

@stephenw10 super fast support - thank you. Already have the compat image which I will try and report back after some sleep.

C8 · Jun 5, 2024, 1:03 AM

Good news (as expected). Using the newer? 'compat' image I was able to update U-Boot loader...

Self-service download for memstick installer (via store) - pfSense-plus-installer-24.02-BETA2-aarch64-20240312-0600.img [1,829,020,160 bytes]
TAC provided 'compat' download for recovery - pfSense-plus-compat-recovery-24.03-RELEASE-aarch64.img [3,758,096,384 bytes]

U-Boot Loader Versions

Before: U-Boot 2017.03-armada-17.10.2-g6a6581a-dirty (Nov 01 2018 - 16:04:57 -0300)
After: U-Boot 2018.03-devel-18.12.3-gc9aa92c-dirty (Oct 07 2021 - 18:20:55 -0300)

Update (snippet from "run usbrecovery")

Marvell>> run usbrecovery
run usbrecovery
sdhci_transfer_data: Error detected in status(0x408000)!
** First descriptor is NOT a primary desc on 1:1 **
sdhci_transfer_data: Error detected in status(0x408000)!
** First descriptor is NOT a primary desc on 1:1 **
switch to partitions #0, OK
mmc1(part 0) is current device

MMC erase: dev # 1, block # 0, count 4194304 ... 4194304 blocks erased: OK
starting USB...
USB0: Register 2000104 NbrPorts 2
Starting the controller
USB XHCI 1.00
USB1: USB EHCI 1.00
scanning bus 0 for devices... 2 USB Device(s) found
scanning bus 1 for devices... 1 USB Device(s) found
scanning usb for storage devices... 1 Storage Device(s) found
reading efi/boot/bootaa64.efi
2097152 bytes read in 42 ms (47.6 MiB/s)
reading armada-3720-sg1100.dtb
18022 bytes read in 21 ms (837.9 KiB/s)

Starting EFI application at 05000000 ...

Card did not respond to voltage select!
mmc_init: -95, time 41
Scanning disk sdhci@d8000.blk...
Scanning disk usb_mass_storage.lun0...
Found 5 disks
[?25h[2J

FreeBSD EFI boot block

Loader path: /boot/loader.efi

Initializing modules: ZFS UFS

Load Path: bootefi

Load Device: bootefi

Probing 5 block devices...not supported

not supported

better

done

ZFS found no pools

UFS found 1 partition

[18t[?25h[0;0fC[?25ho[?25hn[?25hs[?25ho[?25hl[?25he[?25hs[?25h:[?25h [?25hE[?25hF[?25hI[?25h [?25hc[?25ho[?25hn[?25hs[?25ho[?25hl[?25he[?25h [?25h [?25h

<< At this point I switched to putty - silly me! >>

Fri Apr 19 01:14:19 UTC 2024

Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
da0: <USB SanDisk 3.2Gen1 1.00> Removable Direct Access SPC-4 SCSI device
da0: Serial Number 0101ee44a194d913df20b49201aae0983817add1b0629cf7f5df632c8110
da0: 400.000MB/s transfers
da0: 58656MB (120127488 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
Warning: no time-of-day clock registered, system time will not be set accurately
No suitable dump device was found.
random: randomdev_wait_until_seeded unblock wait
random: unblocking device.
Setting hostuuid: a6ca5927-a08f-4ddb-a60d-8cfd61515652.
Setting hostid: 0x6b89fb18.
Starting file system checks:
/dev/ufsid/6621c527af154bca: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ufsid/6621c527af154bca: clean, 94924 free (28 frags, 11862 blocks, 0.0% fragmentation)
Mounting local filesystems:.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/l ib/compat/pkg
32-bit compatibility ldconfig path:
Setting hostname: pfSense-aarch64-recovery.
Setting up harvesting: [CALLOUT],[UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,[NET_ETHER],NET_TUN,MOUSE,KEYBO ARD,ATTACH,CACHED
Feeding entropy: .
lo0: link state changed to UP
Starting Network: lo0 mvneta0 enc0.
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
mvneta0: flags=1008a02<BROADCAST,ALLMULTI,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
ether f0:ad:4e:0b:47:b4
media: Ethernet 1000baseT <full-duplex>
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0 metric 0 mtu 1536
options=0
groups: enc
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Starting devd.
Starting Network: mvneta0.
mvneta0: flags=1008a02<BROADCAST,ALLMULTI,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
ether f0:ad:4e:0b:47:b4
media: Ethernet 1000baseT <full-duplex>
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Starting Network: enc0.
enc0: flags=0 metric 0 mtu 1536
options=0
groups: enc
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
route: message indicates error: File exists
add host 127.0.0.1: gateway lo0 fib 0: route already in table
route: message indicates error: File exists
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Creating and/or trimming log files.
Clearing /tmp (X related).
Updating motd:.
Updating /var/run/os-release done.
Starting syslogd.
Mounting late filesystems:.
Starting cron.

Fri Apr 19 01:14:19 UTC 2024

FreeBSD/arm64 (pfSense-aarch64-recovery) (ttyu0)

FreeBSD 15.0-CURRENT (pfSense) #0 plus-RELENG_24_03-n256311-e71f834dd81: Fri Apr 19 00:42:41 UTC 2024

Welcome to FreeBSD!

Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories: https://www.FreeBSD.org/security/
FreeBSD Handbook: https://www.FreeBSD.org/handbook/
FreeBSD FAQ: https://www.FreeBSD.org/faq/
Questions List: https://www.FreeBSD.org/lists/questions/
FreeBSD Forums: https://forums.FreeBSD.org/

Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with: pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.

Show the version of FreeBSD installed: freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages: man man
FreeBSD directory layout: man hier

To change this login announcement, see motd(5).
/etc/rc.recovery: /usr/sbin/gnid: not found
Netgate 1100 firmware recovery
Serial: NTG1926000840
Netgate Device ID:

This will install the standard firmware and will erase all the existing
contents of the destination device permanently.

eMMC device: mmcsd0

Select the file system to install (UFS/ZFS) or press the Enter key to install with ZFS:

Selected eMMC device: mmcsd0 - ZFS

Are you sure you want to continue ? (y/N) y

=> Updating the Netgate 1100 U-boot
==> Reading current settings
==> Updating the U-boot image (this may take a few minutes)
64+0 records in
64+0 records out
4194304 bytes transferred in 54.730636 secs (76635 bytes/sec)
==> Updating settings
==> Restoring settings
writing u-boot env(1)... done

Erasing the eMMC contents...
Writing the firmware to eMMC...
(this may take a few minutes to complete)
0+57344 records in
896+0 records out
3758096384 bytes transferred in 208.747839 secs (18003043 bytes/sec)

Done!

The system will halt now, please power off and remove the firmware
recovery storage device.

Shutdown NOW!
shutdown: [pid 1136]
Waiting for system shutdown.
*** FINAL System shutdown message from root@pfSense-aarch64-recovery ***

System going down IMMEDIATELY

                                                                Apr 19 01:19:26 pfSense-aarch64-recovery shutdown[1136]: halt by root:

Stopping cron.
Waiting for PIDS: 1025.
Stopping devd.
Waiting for PIDS: 790.
Writing entropy file: .
Writing early boot entropy file: .
.
Terminated
Apr 19 01:19:27 pfSense-aarch64-recovery syslogd: exiting on signal 15
Waiting (max 60 seconds) for system process vnlru' to stop... done Waiting (max 60 seconds) for system process syncer' to stop...
Syncing disks, vnodes remaining... 0 done
All buffers synced.
Uptime: 5m42s

The operating system has halted.
Power cycle or reset to reboot.

Progress

TIM-1.0
WTMI-devel-18.12.1-1a13f2f
WTMI: system early-init
SVC REV: 5, CPU VDD voltage: 1.225V
NOTICE: Booting Trusted Firmware
NOTICE: BL1: v1.5(release):1f8ca7e-dirty (Marvell-devel-18.12.2)
NOTICE: BL1: Built : 18:22:47, Oct 7 2021
NOTICE: BL1: Booting BL2
NOTICE: BL2: v1.5(release):1f8ca7e-dirty (Marvell-devel-18.12.2)
NOTICE: BL2: Built : 18:22:52, Oct 7 2021
NOTICE: BL1: Booting BL31
NOTICE: BL31: v1.5(release):1f8ca7e-dirty (Marvell-devel-18.12.2)
NOTICE: BL31: Built : 18

U-Boot 2018.03-devel-18.12.3-gc9aa92c-dirty (Oct 07 2021 - 18:20:55 -0300)

Model: Netgate 1100
CPU 1200 [MHz]
L2 800 [MHz]
TClock 200 [MHz]
DDR 750 [MHz]
DRAM: 1 GiB
Comphy chip #0:
Comphy-0: USB3 5 Gbps
Comphy-1: PEX0 2.5 Gbps
Comphy-2: SATA0 6 Gbps
SATA link 0 timeout.
AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
flags: ncq led only pmp fbss pio slum part sxs
PCIE-0: Link down
MMC: sdhci@d0000: 0, sdhci@d8000: 1
Loading Environment from SPI Flash... SF: Detected mx25u3235f with page size 256 Bytes, erase size 64 KiB, total 4 MiB
OK
Model: Netgate 1100
Net: eth0: neta@30000 [PRIME]
Read - switch port: 0x1, page: 0x0, reg: 0x0, val: 0xFFFF
Read - switch port: 0x2, page: 0x0, reg: 0x0, val: 0xFFFF
Switch Ports Disabled
Hit any key to stop autoboot: 2 1 0
Card did not respond to voltage select!
17302 armada-3720-gti-doorkeeper.dtb
18022 armada-3720-netgate-1100.dtb
18022 armada-3720-sg1100.dtb
12944 armada-3720-netgate-2100.dtb
12944 armada-3720-sg2100.dtb

5 file(s), 0 dir(s)

2097152 bytes read in 105 ms (19 MiB/s)
18022 bytes read in 10 ms (1.7 MiB/s)

Starting EFI application at 07000000 ...

Card did not respond to voltage select!
Scanning disk sdhci@d0000.blk...
Disk sdhci@d0000.blk not ready
Scanning disk sdhci@d8000.blk...
Found 4 disks
Consoles: EFI console

<< lots more POST bits >>

Welcome to Netgate pfSense Plus 24.03-RELEASE...

Conclusion

At this point the expected installer starts and I should be able to install the updated pfSense from here.

Thank you for the assist - much appreciated!

stephenw10 · Jun 5, 2024, 12:12 PM

Great result!