SG-3100 - Reboot loop - ufs_dirbad - 21.05.2

pfgateuser

Rebooted a working system using "reroot" option
Did not come back up after 10+ minutes - no networks came up and could not ping
Noticed the lights on the front cycling.
Connected console cable.
Noticed reboot loop: panics with ufs_dirbad over and over.
Depressing and holding hardware reset button will suspend the device, and releasing the button will cause a reboot, but it panics again the same way as without the hardware reset button.
Am able to get to a Marvell prompt but don't know how to use that

Attached is the last page of text, showing the panic at the end

20220317-pfsense-reboot-loop.txt

It seems to be a file system corruption problem, and if there was a way to run a file system utility it might be correctable. However since I can only get into the Marvell prompt and not the "single user mode" I can't run any Linux utilities.

Is my device bricked? If not, how do I fix it or reset it?

SteveITS

Assuming the disk isn't bad, you could get the firmware from Netgate via free ticket at go.netgate.com and reinstall. It doesn't take too long. They'll give the firmware and links to instructions in the docs.

It doesn't sound like you can get to the point to run https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html#manual-filesystem-check...

pfgateuser

@steveits

Is there a "live cd" Linux ISO that can be written to a usb and booted to recognize the disk and repair it? What distribution would you recommend for that? (I assume it has to be 32-bit and arm based?)

SteveITS

@pfgateuser No idea sorry. I would just reinstall and then it's a clean slate.

stephenw10

I would use FreeBSD not Linux. Though there may well be some Linux flavour that can repair UFS.

However you shouldn't need to do that, just boot into single user mode and run fsck from there.

We can see it's booting far past the boot loader menu to reach that error. So you should definitely be able to boot single user.

Steve

pfgateuser

@stephenw10

At what point in the startup process should I have been able to get to single user mode, and how? I was only able to get to a Marvell prompt. I watched a dozen or so reboot cycles thru the console and tried intervening several times when the log/prompt suggested I could, but Marvell is all I could get.

pfgateuser

@steveits

I opened a TAC Lite ticket and they gave me these instructions:

Sorry about the trouble.


Please use the following link to access the 22.01 firmware:
https://nc.netgate.com/nextcloud/s/ws7HnooGLWfnnzi

Here is the SHA256 checksum to verify if needed:
https://nc.netgate.com/nextcloud/s/JyEDMEqPxctb6fb
https://docs.netgate.com/pfsense/en/latest/install/download-installer-image.html#verifying-the-integrity-of-the-download

Verify you're using this image:
pfSense-plus-Netgate-3100-recovery-22.01-RELEASE-armv7.img.gz

We recommend using https://etcher.io to write the image.

Reinstall instructions:
https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/reinstall-pfsense.html

I downloaded the .img.gz and the checksum, checked it with sha256sum, gunzipp'ed it, and wrote it to a usb stick with:

dd status=progress bs=10M if=pfSense-plus-Netgate-3100-recovery-22.01-RELEASE-armv7.img of=/dev/the-usb-path

Where dev/the-usb-path was the usb stick shown by lsblk

And was able to reflash the software following these instructions:

https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/reinstall-pfsense.html

The prompt was slightly different (there was an additional "are you sure" prompt) but it was approximately the same, and resulted in a wiped but working device.

stephenw10

@pfgateuser said in SG-3100 - Reboot loop - ufs_dirbad - 21.05.2:

At what point in the startup process should I have been able to get to single user mode, and how?

Here:

 __   __                      _ _
|  \/  | __ _ _ ____   _____| | |
| |\/| |/ _` | '__\ \ / / _ \ | |
| |  | | (_| | |   \ V /  __/ | |
|_|  |_|\__,_|_|    \_/ \___|_|_|
         _   _     ____              _
        | | | |   | __ )  ___   ___ | |_ 
        | | | |___|  _ \ / _ \ / _ \| __| 
        | |_| |___| |_) | (_) | (_) | |_ 
         \___/    |____/ \___/ \___/ \__| 
 ** LOADER **


U-Boot 2013.01-02861-g15adf50-dirty (Jul 18 2017 - 23:19:03) Marvell version: devel-17.1.1

Board: Rogue-1
SoC:   MV88F6828 Rev A0
       running 2 CPUs
CPU:   ARM Cortex A9 MPCore (Rev 1) LE
       CPU 0
       CPU    @ 1600 [MHz]
       L2     @ 800 [MHz]
       TClock @ 250 [MHz]
       DDR4    @ 800 [MHz]
       DDR4 32 Bit Width,FastPath Memory Access, DLB Enabled, ECC Disabled
DRAM:  2 GiB
MMC:   mv_sdh: 0
DBG: Calling spi_flash_probe from env_relocate_spec()
SF: Probing bus 0 cs 0 @ 20000000Hz mode 3
SF: Detected N25Q128 with page size 64 KiB, total 16 MiB
PCI-e 0 (IF 0 - bus 1) Root Complex Interface, Detected Link X1, GEN 1.1
PCI-e 1 (IF 1 - bus 2) Root Complex Interface, Detected Link X1, GEN 1.1
USB2.0 0: Host Mode
USB3.0 1: Host Mode

Map:   Code:			0x7fedc000:0x7ff97568
       BSS:			0x7ffef600
       Stack:			0x7f4cbf20
       Heap:			0x7f4cc000:0x7fedc000
       U-Boot Environment:	0x00100000:0x00110000 (SPI)

Board configuration detected:
Net:   
|  port  | Interface | PHY address  |
|--------|-----------|--------------|
| egiga0 |   RGMII   |     0x00     |
| egiga1 |   RGMII   |     0x01     |
| egiga2 |   SGMII   |   In-Band    |
egiga0 [PRIME], egiga1, egiga2
Hit any key to stop autoboot:  0 

Reset SCSI
AHCI init for unit0
Target spinup took 0 ms.
SATA link 1 timeout.

Error: SCSI Controller(s) 1B4B:9215 1B4B:9235 not found
scanning bus for devices...
  Device 0: (0:0) Vendor: ATA Prod.: NT-32 Rev: 1.09
            Type: Hard Disk
            Capacity: 30533.8 MB = 29.8 GB (62533296 x 512)
Found 1 device(s).
reading ubldr.bin
228648 bytes read in 10 ms (21.8 MiB/s)
## Starting application at 0x00200000 ...
Consoles: U-Boot console  
Compatible U-Boot API signature found @0x7f4dc280

FreeBSD/armv6 U-Boot loader, Revision 1.2
(Tue Jul 10 10:26:23 EDT 2018 root@buildbot3)

DRAM: 2048MB
Number of U-Boot devices: 4
U-Boot env: loaderdev not set, will probe all devices.
Found U-Boot device: disk
  Probing all disk devices...
  Checking unit=0 slice=<auto> partition=<auto>...disk0: read failed, error=1

  Checking unit=1 slice=<auto> partition=<auto>... good.
Booting from disk1s2a:
Loading /boot/defaults/loader.conf
console comconsole is invalid!
no valid consoles!
Available consoles:
    uboot
/boot/kernel/kernel text=0xa9a600 data=0x11d4be8 data=0x0+0x54000 syms=[0x4+0xa9240+0x4+0xf4222]
\
Hit [Enter] to boot immediately, or any other key for command prompt.


Type '?' for a list of commands, 'help' for more detailed help.
loader> boot -s