23.09 ->24.03 unable to update boot code on /dev/ada1

crazy-chipmunk

Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 234441607
first: 40
entries: 128
scheme: GPT
Providers:

Name: ada0p1
Mediasize: 272629760 (260M)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 20480
Mode: r1w1e2
efimedia: HD(1,GPT,91e94734-972e-11ee-a4cb-3cecef02e694,0x28,0x82000)
rawuuid: 91e94734-972e-11ee-a4cb-3cecef02e694
rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
label: efiboot0
length: 272629760
offset: 20480
type: efi
index: 1
end: 532519
start: 40
Name: ada0p2
Mediasize: 524288 (512K)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 272650240
Mode: r0w0e0
efimedia: HD(2,GPT,91ec3082-972e-11ee-a4cb-3cecef02e694,0x82028,0x400)
rawuuid: 91ec3082-972e-11ee-a4cb-3cecef02e694
rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
label: gptboot0
length: 524288
offset: 272650240
type: freebsd-boot
index: 2
end: 533543
start: 532520
Name: ada0p3
Mediasize: 1073741824 (1.0G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 273678336
Mode: r1w1e0
efimedia: HD(3,GPT,91f0a445-972e-11ee-a4cb-3cecef02e694,0x82800,0x200000)
rawuuid: 91f0a445-972e-11ee-a4cb-3cecef02e694
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: swap0
length: 1073741824
offset: 273678336
type: freebsd-swap
index: 3
end: 2631679
start: 534528
Name: ada0p4
Mediasize: 118686220288 (111G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 1347420160
Mode: r1w1e1
efimedia: HD(4,GPT,91f3ff1e-972e-11ee-a4cb-3cecef02e694,0x282800,0xdd12000)
rawuuid: 91f3ff1e-972e-11ee-a4cb-3cecef02e694
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: zfs0
length: 118686220288
offset: 1347420160
type: freebsd-zfs
index: 4
end: 234440703
start: 2631680
Consumers:
Name: ada0
Mediasize: 120034123776 (112G)
Sectorsize: 512
Mode: r3w3e6

Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 234441607
first: 40
entries: 128
scheme: GPT
Providers:

Name: ada1p1
Mediasize: 272629760 (260M)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 20480
Mode: r0w0e0
efimedia: HD(1,GPT,92020f93-972e-11ee-a4cb-3cecef02e694,0x28,0x82000)
rawuuid: 92020f93-972e-11ee-a4cb-3cecef02e694
rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
label: efiboot1
length: 272629760
offset: 20480
type: efi
index: 1
end: 532519
start: 40
Name: ada1p2
Mediasize: 524288 (512K)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 272650240
Mode: r0w0e0
efimedia: HD(2,GPT,9203962b-972e-11ee-a4cb-3cecef02e694,0x82028,0x400)
rawuuid: 9203962b-972e-11ee-a4cb-3cecef02e694
rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
label: gptboot1
length: 524288
offset: 272650240
type: freebsd-boot
index: 2
end: 533543
start: 532520
Name: ada1p3
Mediasize: 1073741824 (1.0G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 273678336
Mode: r1w1e0
efimedia: HD(3,GPT,9207392d-972e-11ee-a4cb-3cecef02e694,0x82800,0x200000)
rawuuid: 9207392d-972e-11ee-a4cb-3cecef02e694
rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
label: swap1
length: 1073741824
offset: 273678336
type: freebsd-swap
index: 3
end: 2631679
start: 534528
Name: ada1p4
Mediasize: 118686220288 (111G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 1347420160
Mode: r1w1e1
efimedia: HD(4,GPT,920a9cad-972e-11ee-a4cb-3cecef02e694,0x282800,0xdd12000)
rawuuid: 920a9cad-972e-11ee-a4cb-3cecef02e694
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: zfs1
length: 118686220288
offset: 1347420160
type: freebsd-zfs
index: 4
end: 234440703
start: 2631680
Consumers:
Name: ada1
Mediasize: 120034123776 (112G)
Sectorsize: 512
Mode: r2w2e3

stephenw10

Is that system booting uefi?

[24.03-RELEASE][admin@4200.stevew.lan]/root: sysctl machdep.bootmethod
machdep.bootmethod: UEFI

Do you know what version it was originally installed as?

crazy-chipmunk

@stephenw10
Yes booting UEFI.

Originally installed many years ago before plus etc. Converted to plus when that happened and updates went ok till 23.05.1 Then failed badly going from 23.05.1 to 23.09 with boot errors. UEFI partition was too small for the new update and the default installer would not erase the drive the re-create the partitions so had to attach the drives to another system, delete all partitions and then put the drives back into the pfsense system and then able to install CE 2.7.1. Subsequently had to upgrade back to plus 23.09 before I could use my backup config to restore.

stephenw10

Hmm, is it already mounted perhaps? What doe mount -p show?

crazy-chipmunk

@stephenw10

pfSense/ROOT/default	/			zfs	rw,noatime,nfsv4acls 	0 0
devfs			/dev			devfs	rw		0 0
/dev/gpt/efiboot0	/boot/efi		msdosfs	rw		2 2
pfSense/tmp		/tmp			zfs	rw,nosuid,noatime,nfsv4acls 	0 0
pfSense/var		/var			zfs	rw,noatime,nfsv4acls 	0 0
pfSense			/pfSense		zfs	rw,noatime,nfsv4acls 	0 0
pfSense/home		/home			zfs	rw,noatime,nfsv4acls 	0 0
pfSense/var/log		/var/log		zfs	rw,noexec,nosuid,noatime,nfsv4acls 	0 0
pfSense/var/cache	/var/cache		zfs	rw,noexec,nosuid,noatime,nfsv4acls 	0 0
pfSense/var/empty	/var/empty		zfs	rw,noatime,nfsv4acls 	0 0
pfSense/var/db		/var/db			zfs	rw,noexec,nosuid,noatime,nfsv4acls 	0 0
pfSense/var/tmp		/var/tmp		zfs	rw,nosuid,noatime,nfsv4acls 	0 0
pfSense/reservation	/pfSense/reservation	zfs	rw,noatime,nfsv4acls 	0 0
pfSense/ROOT/default/cf	/cf			zfs	rw,noexec,nosuid,noatime,nfsv4acls 	0 0
pfSense/ROOT/default/var_cache_pkg /var/cache/pkg		zfs	rw,noexec,nosuid,noatime,nfsv4acls 	0 0
pfSense/ROOT/default/var_db_pkg /var/db/pkg		zfs	rw,noexec,nosuid,noatime,nfsv4acls 	0 0
tmpfs			/var/run		tmpfs	rw		0 0

stephenw10

@crazy-chipmunk said in 23.09 ->24.03 unable to update boot code on /dev/ada1:

/dev/gpt/efiboot0

Hmm so it is still mounted from ada0 but not ada1.

crazy-chipmunk

@stephenw10
I haven't rebooted the device yet as I don't know what happens and did not want to have to resetup everything from scratch, so posted here and waiting. Thanks for taking a look.

stephenw10

I would expect it to have done nothing because it aborted the upgrade process.

I find it odd it not log updating ada0p1 but perhaps it would have done that after ada1p1.

It doesn't actually need to update the efiboot partition so it should still boot fine in the instance that the 24.03 upgrade did take place.

However we hit this in testing on some other install configurations and it just didn't upgrade. We also put in fixes to allow it to complete for everything we hit though.

I'll try to replicate it but it looks like you have a pretty standard ZFS mirror there. It could be something hardware specific.

crazy-chipmunk

@stephenw10
Yes, rebooted successfully back to 23.09.1 as you said, upgrade did not complete. No other environments visible. Would it be useful for you to have the full output from the upgrade process? I can attach here if you like. Thanks again for looking into this issue.

stephenw10

Yes any additional data you can provide would be great, thanks.

crazy-chipmunk

This post is deleted!

crazy-chipmunk

@stephenw10
I attached the full output from the failed upgrade as a txt file.

More Hardware and Setup details
Supermicro A2SDi-4C-HLN4F motherboard. Latest bios 1.9a. 8gb ECC supermicro memory
Additional pfSense packages installed: Wireguard, System_patches, OpenVPN-Client-Export, Nut

I also uninstalled the packages and tried to upgrade and that failed as well with the same error as expected

I have attached 2 text files with drive details.

pfsense update failed.txt
ada0-drive-details.txt
ada1-drive-details.txt

Thanks for all your help. Hope you can help me find a fix. Thanks again.

stephenw10

Hmm, not easy to replicate so far at least.

Since you have tried to install you should be able to try running the install-boot code directly like:

[23.09.1-RELEASE][root@plusdev-4.stevew.lan]/root: install-boot
System Configuration

Architecture: amd64
Boot Devices: /dev/ada0
              /dev/ada1
 Boot Method: uefi
  Filesystem: zfs
    Platform: QEMU Guest

Proced with updating boot code? [y/N]: y

Updating boot code...

/usr/local/sbin/../libexec/install-boot.sh -b auto -f zfs -s gpt -u ada1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada1
partcode written to ada1p2
bootcode written to ada1
ESP /dev/ada1p1 mounted on /tmp/stand-test.bSVnmC
263440KB space remaining on ESP: renaming old bootx64.efi file /efi/boot/bootx64.efi /efi/boot/bootx64-old.efi
263440KB space remaining on ESP: renaming old loader.efi file /etc/freebsd/loader.efi /etc/freebsd/loader-old.efi
Copying loader.efi to /EFI/freebsd on ESP
Creating UEFI boot entry for FreeBSD
Marking UEFI boot entry 0008 active
Copying bootx64.efi to /efi/boot on ESP
Unmounting and cleaning up temporary mount point
Finished updating ESP

/usr/local/sbin/../libexec/install-boot.sh -b auto -f zfs -s gpt -u ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada0
partcode written to ada0p2
bootcode written to ada0
ESP /dev/ada0p1 mounted on /tmp/stand-test.G3cO9h
264784KB space remaining on ESP: renaming old bootx64.efi file /efi/boot/bootx64.efi /efi/boot/bootx64-old.efi
264784KB space remaining on ESP: renaming old loader.efi file /etc/freebsd/loader.efi /etc/freebsd/loader-old.efi
Copying loader.efi to /EFI/freebsd on ESP
Creating UEFI boot entry for FreeBSD
Marking UEFI boot entry 0009 active
Copying bootx64.efi to /efi/boot on ESP
Unmounting and cleaning up temporary mount point
Finished updating ESP

Done.

Do you see any other errors there? Or any errors in the system log after running it?

stephenw10

Ok it's probably this: https://redmine.pfsense.org/issues/15083

Likely that EFI partition was never formatted when it was originally installed. Running the commands listed there to create the fat32 filesystem should allow it to complete the upgrade.

crazy-chipmunk

@stephenw10
You are correct. This is exactly what the problem was. EFI partition was never formatted after creation. I followed the steps you outlined and the upgrade completed successfully. Thank you so much for taking the time to figure out the issue and helping me. Have a great day!