21.05.1 to 22.01 upgrade reboot loop

leres

I noticed a new version was available so I dusted off my backup SG-1100 and started an upgrade on it. This seemed to go well:

Installed packages to be UPGRADED:
	pfSense-u-boot-1100: 20181122 -> 20220202 [pfSense]
	pfSense-u-boot-env: 20200316 -> 20211006 [pfSense]

Number of packages to be upgraded: 2
[1/2] Upgrading pfSense-u-boot-env from 20200316 to 20211006...
[1/2] Extracting pfSense-u-boot-env-20211006: .... done
[2/2] Upgrading pfSense-u-boot-1100 from 20181122 to 20220202...
[2/2] Extracting pfSense-u-boot-1100-20220202: ....... done
>>> Removing unnecessary packages... done.
System is going to be upgraded.  Rebooting in 10 seconds.
Success

But now it's in a reboot loop:

U-Boot 2018.03-devel-18.12.3-gc9aa92c-dirty (Oct 07 2021 - 18:20:55 -0300)

Model: Netgate 1100
       CPU     1200 [MHz]
       L2      800 [MHz]
       TClock  200 [MHz]
       DDR     750 [MHz]
DRAM:  1 GiB
Comphy chip #0:
Comphy-0: USB3          5 Gbps    
Comphy-1: PEX0          2.5 Gbps  
Comphy-2: SATA0         6 Gbps    
SATA link 0 timeout.
AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
flags: ncq led only pmp fbss pio slum part sxs 
PCIE-0: Link down
MMC:   sdhci@d0000: 0, sdhci@d8000: 1
Loading Environment from SPI Flash... SF: Detected mx25u3235f with page size 256 Bytes, erase size 64 KiB, total 4 MiB
OK
Model: Netgate 1100
Net:   eth0: neta@30000 [PRIME]
Hit any key to stop autoboot:  0 
Marvell>>

I tried cycling power with no change in symptoms.

I still have an installation thumb drive I needed to get my other box going, it's labeled pfSense+ 21.02-p1, should I try installing that and taking another run at the 22.01 upgrade or is there another suggestion for completing this upgrade?

leres

I tried booting to single user and then hitting ^D... that seems to be letting the upgrade process continue.

# ^D
Configuring crash dumps...
No suitable dump device was found.
/dev/diskid/DISK-82AC8404s3: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/diskid/DISK-82AC8404s3: clean, 1400886 free (2062 frags, 174853 blocks, 0.1% fragmentation)
Filesystems are clean, continuing...
Mounting filesystems...
/etc/pfSense-rc: /etc/rc.checkclock: not found

        __
 _ __  / _|___  ___ _ __  ___  ___      _
| '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \   _| |_
| |_) |  _\__ \  __/ | | \__ \  __/  |_   _|
| .__/|_| |___/\___|_| |_|___/\___|    |_|
|_|


Welcome to Netgate pfSense Plus 21.05.1-RELEASE...

...random: unblocking device.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/lib/engines /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.32/mach/CORE
done.
>>> Removing vital flag from php74... done.
>>> Locking package pfSense-pkg-Cron... done.
>>> Locking package pfSense-pkg-acme... done.
>>> Locking package pfSense-pkg-arpwatch... done.
>>> Locking package pfSense-pkg-bind... done.
>>> Locking package pfSense-pkg-nrpe... done.
>>> Locking package pfSense-pkg-pfBlockerNG-devel... done.
>>> Locking package pfSense-pkg-sudo... done.
>>> Upgrading necessary core packages... 
Checking for upgrades (2 candidates): .. done
Processing candidates (2 candidates): .. done
Checking integrity... done (0 conflicting)
The following 2 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        pfSense-base: 21.05.1 -> 22.01 [pfSense-core]
        pfSense-default-config-serial: 21.05.1 -> 22.01 [pfSense-core]

Number of packages to be upgraded: 2

The process will require 2 MiB more space.
[1/2] Upgrading pfSense-default-config-serial from 21.05.1 to 22.01...
[1/2] Extracting pfSense-default-config-serial-22.01: .. done
[2/2] Upgrading pfSense-base from 21.05.1 to 22.01...
[2/2] Extracting pfSense-base-22.01: .. done
===> Keeping a copy of current version mtree
===> Removing schg flag from base files
===> Extracting new base tarball

rcoleman-netgate

@leres You can always make a ticket request for the 22.01 firmware image here: https://www.netgate.com/tac-support-request

leres

@rcoleman-netgate I guess I'll have to do that, after finishing the upgrade I still see a reboot loop. Thanks for the reminder.

leres

Hum... While upgrading I have a usb cable plugged in so I can get to the serial console. I find that if I exit tip while the SG-1100 is trying to boot it actually succeeds.

So I think either there's a problem with 22.01 and usb/serial console or something is broken with my specific setup. I always leave a serial cable plugged into a desktop system with a tip running in a detached screen session and I don't want to run that way and risk a reboot not succeeding so I'm may well reinstall from media to see if I can solve this.

leres

I received the 22.01 reinstall image (tac was super fast, as usual) and reinstalled. But I still find I have a reboot loop if the usb/serial console is connected to tip when the box tries to boot. In fact I've tried this with two SG-1100's

Here's a complete log of what happens in between autoboots.

>> FreeBSD EFI boot block
   Loader path: /boot/loader.efi

   Initializing modules: ZFS UFS
   Load Path: /\armada-3720-sg1100.dtb
   Load Device: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/eMMC(1)/eMMC(0)/HD(2,0x01,0,0x64001,0x1117c)
   Probing 4 block devices...not supported
not supported
not supported
better
 done
    ZFS found no pools
    UFS found 1 partition
Consoles: EFI console  
|    Reading loader env vars from /efi/freebsd/loader.env
Setting currdev to disk0p3:
| FreeBSD/arm64 EFI loader, Revision 1.1

   Command line arguments: loader.efi
   Image base: 0x3b6030c0
   EFI version: 2.70
   EFI Firmware: Das U-Boot (rev 0.00)
   Console: efi (0x1000)
   Load Device: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/eMMC(1)/eMMC(0)/HD(3,0x01,0,0x7517d,0x300010)
Trying ESP: /VenHw(e61d73b9-a384-4acc-aeab-82e828f3628b)/eMMC(1)/eMMC(0)/HD(3,0x01,0,0x7517d,0x300010)
Setting currdev to disk0p3:
Loading /boot/defaults/loader.conf
Loading /boot/defaults/loader.conf
Loading /boot/device.hints
Loading /boot/loader.conf
Load1. Boot Multi user [Enter]l
2. Boot Single user
3. Escape to loader prompt___  
4. Reboot                 ___\ 
5. Cons: Serial         /`     
 |                     /    :-|
Options:_________  ___/    /_ |
Kernel: default/kernel (1 of 1)
7. Boot Options/    /   /     |
Autoboot in 3 seconds. [Space] to pause
 | /   ______/    /   /  _    |
   __________________________ |
  /                       ___\ 
 |                      /`     
 |                     /    :-|
 |      _________  ___/    /_ |
 |    /` ____   / /__    ___/ |  ___      _                 
 |   /  /   /  /    /   /     ||/ _ \   _| |_               
 |  /  /___/  /    /   /      |\  __/  |_   _|              
 | /   ______/    /   /  _    |/\___|    |_|                
 |/   /          /   / _| |_  |                             
     /          /___/ |_   _| |                             
    /                   |_|   |
   /_________________________/  
                               
   _ __  / _|___  ___ _ __  ___  ___      _                 
  | '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \   _| |_               
  | |_) |  _\__ \  __/ | | \__ \  __/  |_   _|              
  | .__/|_| |___/\___|_| |_|___/\___|    |_|                
  |_|                                                       
                                                            

Welcome to Netgate pfSense Plus
Boot Multi user [Enter]
2. Boot Single user
3. Escape to loader prompt
4. Reboot
5. Cons: Serial

Options:
6. Kernel: default/kernel (1 of 1)
7. Boot Options
resetting ...
TIM-1.0
WTMI-devel-18.12.1-1a13f2f
WTMI: system early-init
SVC REV: 5, CPU VDD voltage: 1.237V
NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v1.5(release):1f8ca7e-dirty (Marvell-devel-18.12.2)
NOTICE:  BL1: Built : 18:22:47, Oct  7 2021
NOTICE:  BL1: Booting BL2
NOTICE:  BL2: v1.5(release):1f8ca7e-dirty (Marvell-devel-18.12.2)
NOTICE:  BL2: Built : 18:22:52, Oct  7 2021
NOTICE:  BL1: Booting BL31
NOTICE:  BL31: v1.5(release):1f8ca7e-dirty (Marvell-devel-18.12.2)
NOTICE:  BL31: Built : 18

U-Boot 2018.03-devel-18.12.3-gc9aa92c-dirty (Oct 07 2021 - 18:20:55 -0300)

Model: Netgate 1100
       CPU     1200 [MHz]
       L2      800 [MHz]
       TClock  200 [MHz]
       DDR     750 [MHz]
DRAM:  1 GiB
Comphy chip #0:
Comphy-0: USB3          5 Gbps    
Comphy-1: PEX0          2.5 Gbps  
Comphy-2: SATA0         6 Gbps    
SATA link 0 timeout.
AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
flags: ncq led only pmp fbss pio slum part sxs 
PCIE-0: Link down
MMC:   sdhci@d0000: 0, sdhci@d8000: 1
Loading Environment from SPI Flash... SF: Detected mx25u3235f with page size 256 Bytes, erase size 64 KiB, total 4 MiB
OK
Model: Netgate 1100
Net:   eth0: neta@30000 [PRIME]
Hit any key to stop autoboot:  2 ^H^H^H 1 ^H^H^H 0

stephenw10

Mmm, I would try some other console application. It looks like your current setup is sending characters continually. It should not stop at those prompts unless you enter a key deliberately.

Steve

leres

The reboot loop never stops unless I enter a key.

Also I'm not sure why tip would be a problem now after using it to connect to SG-1100 usb serial consoles for 2+ years.

Anyway I have a dialog going with tac and will try to post something here once this is resolved.

stephenw10

Ah, OK. Where does it loop then?

Nothing you've posted so far shows it rebooting.

Steve

rcoleman-netgate

@stephenw10 We were able to get him back in.

Once we booted into Single User Mode we could run

/sbin/fsck -y /

about 10 times, then 'reboot' and the 1100 came right up.

leres

I think the key difference is we were working from an osx securecrt session instead of xterm on a FreeBSD desktop. I suspect 22.01 changes some escape sequences to some that my xterm responses to...

rcoleman-netgate

@leres Or it was the use of XTerm instead of GNU Screen on FreeBSD.

leres

All I can say is this has worked for 2+ years with all versions of pfsense prior to 22.01. I either run tip from csh in an xterm window or run tip in a screen session and access the screen session from an xterm window.