After upgrade 24.03 to 24.11 reboot hangs at start @ 0xffffff....

d9001089

@jimp
I re-ran the upgrade to get the print out you wanted from the console and this time it completed successfully... I didn't do anything except for uninstall the Acme package because its been crashing do to a PHP error which I will list below, if you have any insights, and yes I will create a new ticket in another channel for this issue.

16:16:02 PHP ERROR: Type: 1, File: /usr/local/www/acme/acme_accountkeys.php, Line: 158, Message: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/www/acme/acme_accountkeys.php:158
Stack trace:
#0 {main}
thrown

stephenw10

Are you seeing that error in 24.11 if you reinstall the pkg? Or just in 24.03?

d9001089

@stephenw10
Yes, and now I don't see the acme menu option under services even when i do a reinstall. The only way to get to it is using the URL. Once I'm at acme.xml page I select account keys and that's when it crashes.

stephenw10

Ok opened a bug to track that: https://redmine.pfsense.org/issues/15892

stephenw10

Anyone hitting this console issue try booting verbose. Escape the loader menu to the prompt (OK>) and enter: boot -v

freph533

@jimp said in After upgrade 24.03 to 24.11 reboot hangs at start @ 0xffffff....:

@freph533 said in After upgrade 24.03 to 24.11 reboot hangs at start @ 0xffffff....:

Same issue observed just now on a VM that's had 0 issues with 24.03. Hard reboot on the VM booted successfully back into 24.03.

What type of VM? If it's Proxmox and running an EFI BIOS, it may be something we've hit before (adding a serial port to the VM hardware works around it) but that's different than the other problems reported in this thread.

Yes, it's in Proxmox on EFI BIOS. There's no serial port on the VM hardware so that might be part of the cause in my case, though given the feedback from others it seems like it may be something else.

@stephenw10 Given my setup with the Proxmox VM is it still useful for me to provide output on an attempted upgrade (and if so, with or without adding a serial port to the VM devices? FWIW I've never had issues through several upgrades without a serial device on this VM)?

stephenw10

Yes, if you can try booting verbose after the upgrade.

Also you could try adding a serial port to confirm if that has any effect since that might be a useful clue.

freph533

@stephenw10 said in After upgrade 24.03 to 24.11 reboot hangs at start @ 0xffffff....:

Yes, if you can try booting verbose after the upgrade.

Also you could try adding a serial port to confirm if that has any effect since that might be a useful clue.

Adding the serial device let me upgrade successfully. Upon removing the serial device and attempting multiple reboots it hangs at the same spot with the error. Adding the serial device back lets the VM boot again. So it seems like 24.11 has a hard dependency on a serial port existing whereas 24.03 does not. Screenshot 2024-12-09 190838.png

jimp

@freph533 said in After upgrade 24.03 to 24.11 reboot hangs at start @ 0xffffff....:

So it seems like 24.11 has a hard dependency on a serial port existing whereas 24.03 does not.!

It's not new in 24.11, it just seems to hit some VMs differently than others, and we haven't nailed down why yet:

https://redmine.pfsense.org/issues/15074

jackyaz

Happening to me with a 24.03 to 24.11 upgrade on a Netgate 4200 MAX

Verbose boot logs, unfortunately doesn't offer anything new:

OK boot -v
Loading kernel...
/boot/kernel/kernel text=0x1a4c98 text=0xff3048 text=0x17ed568 data=0x180+0xe80 data=0x24c808+0x3b37f8 0x8+0x1d4108+0x8+0x1e9a19
Loading configured modules...
/boot/kernel/opensolaris.ko size 0x1e2a8 at 0x37be000
/boot/kernel/zfs.ko size 0x619a40 at 0x37dd000
can't find '/boot/entropy'
can't find '/etc/hostid'
staging 0x3fa00000-0x43f40000 (not copying) tramp 0x43f40000 PT4 0x43f41000
Start @ 0xffffffff803a5000 ...
/

jackyaz

I opened a ticket with TAC support and they're telling me to "just reinstall the software". No effort to try and figure out what's wrong.

I don't doubt a reinstall will help, but what worries me is if it doesn't I'll have to then reinstall to 24.03 and that's downtime I can't afford when I use this device for my work.

I'll have to sit on 24.03 until the issue is identified.

GripePotato

@jackyaz said in After upgrade 24.03 to 24.11 reboot hangs at start @ 0xffffff....:

Netgate 4200 MAX

I also experience the problem with a Netgate 4200 MAX (vice 6100 MAX I reported earlier).

GripePotato

This thread indicates two types of devices experience problems:

Pfsense on VM
4200 MAX

The VM folks got a solution/recommendation of attaching a serial port to the VM. Is there a recommendation for the 4200 MAX other than trying again? I've tried a couple times already, both resulting in failure.

stephenw10

Still unable to replicate this locally.

Is there anything about those devices that might be unique? Perhaps you made a BIOS setting change? Have something unusual connected to it?

jackyaz

@stephenw10 No BIOS changes on my side, never even been into the BIOS!

I do have a USB cable plugged into the USB A port on the side but the other end is disconnected - I'll occasionally need to use my phone as a USB tethering modem when my fibre drops out. I can try unplugging it to see if that makes a difference? I'd be surprised if it does considering its not connected to anything!

stephenw10

Mmm, me too. It certainly shouldn't make any difference. Might be worth trying to rule it out though.

GripePotato

@stephenw10
My 4200 MAX is off the shelf with no special (e.g. BIOS) changes and nothing unusual (e.g. USB) plugged in.

jimp

@GripePotato said in After upgrade 24.03 to 24.11 reboot hangs at start @ 0xffffff....:

This thread indicates two types of devices experience problems:

Pfsense on VM

4200 MAX

The VM folks got a solution/recommendation of attaching a serial port to the VM. Is there a recommendation for the 4200 MAX other than trying again? I've tried a couple times already, both resulting in failure.

The VM issue was quite a bit different and unrelated to what's happening on the 4200, which still isn't clear.

As for the 4200, nobody has yet been able to capture the whole boot process on a system affected by the problem (See my request in https://forum.netgate.com/post/1197950 ) -- the other person who tried ended up having a successful upgrade on their next attempt.

So if you could, try the upgrade again and capture the whole next boot output from the serial console and attach it here (or send it via TAC).

It may also be worth checking to make sure you have /boot/loader.conf.lua present with appropriate content. Also make sure you have the pfSense, pfSense-plus, and pfSense-boot OS packages installed before upgrading. Check pkg info -x pfSense to see if they are in the list.

jackyaz

@jimp my output is literally all that's displayed, other than the loader menu. There's no other output whatsoever

jimp

@jackyaz said in After upgrade 24.03 to 24.11 reboot hangs at start @ 0xffffff....:

@jimp my output is literally all that's displayed, other than the loader menu. There's no other output whatsoever

I mean before that. The rest of the POST that happens from power on until it reaches the menu before it tries to boot the kernel. You're saying there is nothing else at all?

There should be output like:

Version 2.22.1285. Copyright (C) 2024 AMI                                       
BIOS Date: 02/01/2024 15:18:05 Ver: 2AZRT81                                     
Press <DEL> or <ESC> to enter setup. 

Consoles: EFI console  
    Reading loader env vars from /efi/freebsd/loader.env
Setting currdev to disk0p1:
FreeBSD/amd64 EFI loader, Revision 3.0
(Sun Dec 15 16:14:26 UTC 2024 root@freebsd)

   Command line arguments: loader.efi
   Image base: 0x54a1b000
   EFI version: 2.80
   EFI Firmware: American Megatrends (rev 5.26)
   Console: efi (0x1000)
   Load Path: \efi\freebsd\loader.efi
   Load Device: PciRoot(0x0)/Pci(0x14,0x0)/USB(0x5,0x0)/HD(1,GPT,FAE2512E-DA37-11EE-B8CE-0008A212EC83,0x28,0x82000)
   BootCurrent: 0001
   BootOrder: 0001[*] 000c 0003 0004 0005 0006 0007 0008 0009 000a
   BootInfo Path: HD(1,GPT,FAE2512E-DA37-11EE-B8CE-0008A212EC83,0x28,0x82000)/\efi\freebsd\loader.efi
Ignoring Boot0001: Only one DP found
Trying ESP: PciRoot(0x0)/Pci(0x14,0x0)/USB(0x5,0x0)/HD(1,GPT,FAE2512E-DA37-11EE-B8CE-0008A212EC83,0x28,0x82000)
Setting currdev to disk0p1:
Trying: PciRoot(0x0)/Pci(0x14,0x0)/USB(0x5,0x0)/HD(2,GPT,FAF90874-DA37-11EE-B8CE-0008A212EC83,0x82028,0x400)
Setting currdev to disk0p2:
Trying: PciRoot(0x0)/Pci(0x14,0x0)/USB(0x5,0x0)/HD(3,GPT,FB17245C-DA37-11EE-B8CE-0008A212EC83,0x82800,0x200000)
Setting currdev to disk0p3:
Trying: PciRoot(0x0)/Pci(0x14,0x0)/USB(0x5,0x0)/HD(4,GPT,FB36C0A0-DA37-11EE-B8CE-0008A212EC83,0x282800,0x1AB1000)
Loading /boot/defaults/loader.confT/default_20241216101221:
Loading /boot/defaults/loader.confault:
Loading /boot/device.hints
Loading /boot/loader.conf
Loading /boot/loader.conf.lua
Netgate 4200 detected.
Loading /boot/loader.conf.local

I haven't seen anything like that in any other post in this thread, all the captures start too late in the process (menu or later)