SG-1100 - Failed pfsense upgrade

extremesurf

@stephenw10 This is the output when I switch the flash drives around. I should mention that previously I couldn't get the pfSense recovery flash drive to boot off USB 3.0 so I had to move it to USB 2.0.

Hit any key to stop autoboot:  0
Marvell>> usb reset
resetting USB...
USB0:   Register 2000104 NbrPorts 2
Starting the controller
USB XHCI 1.00
USB1:   USB EHCI 1.00
scanning bus 0 for devices... usb_new_device: Cannot read configuration, skipping device 13fe:5100
1 USB Device(s) found
scanning bus 1 for devices... 2 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found
Marvell>> run usbrecovery
switch to partitions #0, OK
mmc1(part 0) is current device

MMC erase: dev # 1, block # 0, count 4194304 ... 4194304 blocks erased: OK
resetting USB...
USB0:   Register 2000104 NbrPorts 2
Starting the controller
USB XHCI 1.00
USB1:   USB EHCI 1.00
scanning bus 0 for devices... usb_new_device: Cannot read configuration, skipping device 13fe:5100
1 USB Device(s) found
scanning bus 1 for devices... 2 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found
** Invalid partition 2 **

Can you elaborate on what the alternative options are? I wasn't sure how to set the boot env to boot the alternate or manually set it at the uboot prompt.

Thanks in advance, I really appreciate the help!

stephenw10

Hmm, I would try a different USB stick.

Also be sure to completely power cycle the 1100 between attempts to reset the USB bus devices.

extremesurf

@stephenw10 I used an NVMe drive in a USB enclosure and switched around the USB ports and managed to get it installed. It failed the first attempt after downloading the packages when it went to check integrity with:

Child process pid=3197 terminated abnormally: killed

After retrying the install it worked successfully the second time, but when I remove the installer drive and attempt to boot off the flash drive, it gets past the pfSense logo and then it stops at:

sdhci_xenon1-slot0: Present:  0x01f20000 | Host ctl: 0x00000025
sdhci_xenon1-slot0: Power:    0x0000000f | Blk gap:  0x00000000
sdhci_xenon1-slot0: Wake-up:  0x00000000 | Clock:    0x00000407
sdhci_xenon1-slot0: Timeout:  0x0000000c | Int stat: 0x00000000
sdhci_xenon1-slot0: Int enab: 0x01ff003b | Sig enab: 0x01ff003b
sdhci_xenon1-slot0: AC12 err: 0x00000000 | Host ctl2:0x00000000
sdhci_xenon1-slot0: Caps:     0x25ec0099 | Caps2:    0x0000af77
sdhci_xenon1-slot0: Max curr: 0x00000000 | ADMA err: 0x00000000
sdhci_xenon1-slot0: ADMA addr:0x00000000 | Slot int: 0x00000000
sdhci_xenon1-slot0: ===========================================
mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
Mounting from zfs:pfSense/ROOT/default failed with error 6; retrying for 3 more seconds
random: unblocking device.
mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
mmcsd0: Error indicated: 1 Timeout
Mounting from zfs:pfSense/ROOT/default failed with error 6.

Loader variables:
  vfs.root.mountfrom=zfs:pfSense/ROOT/default

Manual root filesystem specification:
  <fstype>:<device> [options]
      Mount <device> using filesystem <fstype>
      and with the specified (optional) option list.

    eg. ufs:/dev/da0s1a
        zfs:zroot/ROOT/default
        cd9660:/dev/cd0 ro
          (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)

  ?               List valid disk boot devices
  .               Yield 1 second (for background tasks)
  <empty line>    Abort manual input

mountroot>

stephenw10

It's still booting from emmc.

If you run at the uboot prompt (Marvel>): printenv. You'll see the uboot environment variables including bootcmd which is still showing 'run emmcboot`.

To test it at the prompt just enter: run usbboot

If that works as expected then you can change the bootcmd to run that everytime.

stephenw10

Something else you might be hitting is if the nvme and emmc (which is now probably read-only) have ZFS pools with the same name it might be trying to mount the emmc even after booting usb.

In that case you would need to reinstall the NVMe using a different zpool name than the default 'pfsense'.

extremesurf

@stephenw10 It boots ok when I move the flash drive back to USB 2.0 (although it is slow, takes 2 minutes 10 secs), but I guess it's not the worst case if it won't boot on USB 3.0.

It comes up with these errors in the Filter Reload log:

There were error(s) loading the rules: pfstl: DIOCBEGINADDRS: Operation not supported - The line in question reads [0]

Not sure if this is anything to worry about, comes up every reboot in the GUI.

Thanks again for all of your help!

stephenw10

Hmm, I have tried creating a second usb boot env in uboot that uses 'loadunit' 1, the other USB device. That might allow it to boot from the USB3 port.

That error is unrelated though. I assume you mean pfctl there? That's usually because the kernel has been upgraded and the old pfctl is still running. Is that possible?

You might see that if it's still mounting root from the emmc somehow.

extremesurf

@stephenw10 I've been playing around with this and decided to try reinstalling it again, and mid install it hung up. I tried power cycling it, restarted the install process, and while it was fetching packages it hung up again with green blinking light. Now power cycling it it just has a green blinking light now and nothing shows up in the console. I'm assuming it is bricked, but let me know if there is anything else I can try.

stephenw10

Which LED is blinking? The power LED, green circle, is hard wired to the power so it should always be on solid. If that's flashing then check the PSU.

extremesurf

@stephenw10 It was the power LED blinking. I found this thread you had been involved in, and I removed that transient protection diode and it started powering on again. However, now when I go to install though it won't finish installing, it just randomly power cycles now, even if you just leave it at a command prompt and keep watching it you will see it power cycle at some point.

I think this is the end of the road for it unfortunately, probably time to upgrade to more reliable hardware. I really appreciate all of your help, thanks again.

stephenw10

You might try a different PSU. If that has a fault it could have killed that diode and be cause issues during high load.