SG-3100 ubldr.bin error after 24.03 upgrade

andrew_cb

We recently upgraded three SG-3100 to 24.03, identified as NA, EV, and MR.

NA: 23.05.1 > 23.09.1 > 24.03_1 = successful
EV: 23.05.1 > 23.09.1 > 24.03_1 = successful
MR: 23.05 > 23.09 > 23.09.1 > 24.03_1 = failed

Console connection to MR showed error " **Unable to read file ubldr.bin ** ## Starting application at 0x00200000"

We had to do a fresh install of 24.03 to get the firewall working.

For NA and EV I have the log file from /cf/conf/upgrade_log.latest.txt.
For MR I only have the output that was written to the screen during the upgrade process.

I compared the three log files and noticed that it contains the following packages that are not present in the other two logs:

aws-sdk-php82: 3.273.3
aws-sdk-php83: 3.273.3 [pfSense]
pfSense-base: 23.09.1 -> 24.03 [pfSense-core]
pfSense-boot: 23.09.1 -> 24.03 [pfSense-core]
pfSense-kernel-pfSense-3100: 23.09.1 -> 24.03 [pfSense-core]
pfSense-pkg-Cron: 0.3.8_3 -> 0.3.8_4 [pfSense]
pfSense-pkg-ipsec-profile-wizard: 1.2 -> 1.2.1 [pfSense]
pkg: 1.20.8_2 -> 1.20.9_1 [pfSense]
pfSense-pkg-aws-wizard-0.10 [pfSense] (ABI changed: 'freebsd:14:armv7:32:el:eabi:softfp' -> 'freebsd:15:armv7:32:el:eabi:softfp')
pfSense-pkg-sudo-0.3_8 [pfSense] (ABI changed: 'freebsd:14:armv7:32:el:eabi:softfp' -> 'freebsd:15:armv7:32:el:eabi:softfp')
pfSense-pkg-zabbix-agent64-1.0.6 [pfSense] (ABI changed: 'freebsd:14:armv7:32:el:eabi:softfp' -> 'freebsd:15:armv7:32:el:eabi:softfp')
pfSense-pkg-zabbix-proxy64-1.0.6 [pfSense] (ABI changed: 'freebsd:14:armv7:32:el:eabi:softfp' -> 'freebsd:15:armv7:32:el:eabi:softfp')
pfSense-u-boot-3100-20210930_1 [pfSense] (ABI changed: 'freebsd:14:*' -> 'freebsd:15:*')
pfSense-u-boot-env-20230123 [pfSense] (ABI changed: 'freebsd:14:armv7:32:el:eabi:softfp' -> 'freebsd:15:armv7:32:el:eabi:softfp')

I have attached the logs from the three firewalls and the comparison spreadsheet. Hopefully this information might point to what might have caused the boot issue.

EV Upgrade Log.txt MR Upgrade Log.txt NA Upgrade Log.txt SG-3100 24.03 Upgrade Comparison.xls

stephenw10

Hmm, did you recover it by reinstalling?

andrew_cb

@stephenw10 The firewall was at a customer site, so unfortunately we didn't have time to investigate much. We just put the firmware on a USB and ran "run recovery" to reload the firmware, and then restored using auto config backup.

stephenw10

Hmm, curious that it didn't reinstall the uboot pkgs. It implies something earlier removed them somehow. But really not much we can do here after it's been reinstalled.

Bly

@stephenw10 Hi I reply this old topic because I have a SG-3100 bricked after running an upgrade from 23.3 to 24 and it stops exactly at same point.

Let me know what I can provide you for investigation.

General initialization - Version: 1.0.0
AVS selection from EFUSE disabled (Skip reading EFUSE values)
Overriding default AVS value to: 0x23
Detected Device ID 6820
High speed PHY - Version: 2.0

Init Customer board board SerDes lanes topology details:
 | Lane # | Speed|    Type     |
 ------------------------------|
 |   0    |  3   |  SATA0      |
 |   1    |  5   |  PCIe0      |
 |   2    |  3   |  SATA1      |
 |   3    |  4   |  SGMII2     |
 |   4    |  5   |  PCIe1      |
 |   5    |  5   |  USB3 HOST1 |
 -------------------------------
PCIe Ref. Clock Buffer Control is 0xf00015bf, setting to 0xf00015bf
SOC_CONTROL_REG1 is 0xf00015bf, setting to 0x0707c0f3
PCIe, Idx 0: detected no link
PCIe, Idx 1: detected no link
High speed PHY - Ended Successfully
mv_ddr: mv_ddr-armada-17.06.1-g07f8294 (Oct  8 2018 - 12:59:22)
DDR4 Training Sequence - Switching XBAR Window to FastPath Window
mv_ddr: completed successfully

 __   __                      _ _
|  \/  | __ _ _ ____   _____| | |
| |\/| |/ _` | '__\ \ / / _ \ | |
| |  | | (_| | |   \ V /  __/ | |
|_|  |_|\__,_|_|    \_/ \___|_|_|
         _   _     ____              _
        | | | |   | __ )  ___   ___ | |_
        | | | |___|  _ \ / _ \ / _ \| __|
        | |_| |___| |_) | (_) | (_) | |_
         \___/    |____/ \___/ \___/ \__|
 ** LOADER **


U-Boot 2013.01-02879-geb5cbf6642 (Oct 08 2018 - 12:59:19) Marvell version: devel-17.06.0

Board: Rogue-1
SoC:   MV88F6820 Rev B0
       running 2 CPUs
CPU:   ARM Cortex A9 MPCore (Rev 1) LE
       CPU 0
       CPU    @ 1600 [MHz]
       L2     @ 800 [MHz]
       TClock @ 250 [MHz]
       DDR4    @ 800 [MHz]
       DDR4 32 Bit Width,FastPath Memory Access, DLB Enabled, ECC Disabled
DRAM:  2 GiB
MMC:   mv_sdh: 0
DBG: Calling spi_flash_probe from env_relocate_spec()
SF: Probing bus 0 cs 0 @ 20000000Hz mode 3
SF: Detected W25Q32JV with page size 4 KiB, total 4 MiB
PCI-e 0: Detected No Link.
PCI-e 1: Detected No Link.
USB2.0 0: Host Mode
USB3.0 1: Host Mode

Map:   Code:                    0x7fedc000:0x7ff9741c
       BSS:                     0x7ffef600
       Stack:                   0x7f4cbf20
       Heap:                    0x7f4cc000:0x7fedc000
       U-Boot Environment:      0x00100000:0x00110000 (SPI)

Board configuration detected:
Net:
|  port  | Interface | PHY address  |
|--------|-----------|--------------|
| egiga0 |   RGMII   |     0x00     |
| egiga1 |   RGMII   |     0x01     |
| egiga2 |   SGMII   |   In-Band    |
egiga0 [PRIME], egiga1, egiga2
Hit any key to stop autoboot:  0
reading ubldr.bin
** Unable to read file ubldr.bin **
## Starting application at 0x00200000 ...

stephenw10

Is that booting from emmc?

Try interrupting uboot to reach the Marvell>> prompt where it says Hit any key to stop autoboot.

Then:

Marvell>> fatls mmc 0:1
   870156   u-boot.bin 
   949520   u-boot.mmc 
   228648   ubldr.bin 

3 file(s), 0 dir(s)

The ubldr file should be there.

Bly

@stephenw10
Hi, here the result

egiga0 [PRIME], egiga1, egiga2
Hit any key to stop autoboot:  0
Marvell>> fatls mmc 0:1

0 file(s), 0 dir(s)

Marvell>>

Marvell>> fatinfo mmc 0:1
Interface:  MMC
  Device 0: Vendor: Man 000013 Snr 0bf4c02c Rev: 1.0 Prod: Q2J55L
            Type: Removable Hard Disk
            Capacity: 7296.0 MB = 7.1 GB (14942208 x 512)
Filesystem: FAT32 "FATBOOT0   "
Marvell>>

stephenw10

Mmm, well that's not good. Is it actually booting from eMMC? The 3100 can boot from an SSD.

If you run printenv there you can check the bootcmd value to see what it's trying to boot from.

If it really has no boot loader though I would just reinstall it.

Bly

@stephenw10 It is on the emmc:

bootcmd=run setLED; run emmcboot;

Feels like reinstall is the only option now. May I go on or we can look at something else?

stephenw10

I would reinstall anyway from that situation. If the bootloader is missing you can't really be sure other files are not also missing.