Upgrade from 2.7.2 to 2.8.0 Failed and now /boot/efi/ empty
-
@Sinfonia97 Sounds like a corrupted install to me.
I would-
Save pfsense configuration
-
Do a clean install from a recent installer to ensure partitions are consistent with more recent requirements (v2.7.2 or 2.8 or 2.81)
-
Document what low level file structure is used with a good current install so you maybe able to guess what went wrong with the prior corruption (partitions sizes, hardware failure, edge case configuration, edge case update series)
-
-
@Patch Thanks, I am kind of hoping if I can figure out exactly where it needs to be mounted I can use the steps that were provided in other threads to fix the EFI. The steps used before clearly work and even help with the size of the EFI partition. My problem is figuring out where exactly /boot/efi should be mounted to create the backup of the files and the perform the rest of the steps. In what I have been finding, there possibly could be a few places to mount /boot/efi. I ran a geom -t just a bit ago and it pointed me to likely needing to mount it at /dev/ada0p1. The output of geom -t was:
Geom Class Provider ada0 DISK ada0 ada0 DEV ada0 PART ada0p1 ada0p1 DEV msdosfs.ada0p1 VFS ada0 PART ada0p2 ada0p2 DEV ada0p2 LABEL gptid/ac11fbb1-5651-11e8-b5a2-00907fd0950c gptid/ac11fbb1-5651-11e8-b5a2-00907fd0950c DEV ffs.gptid/ac11fbb1-5651-11e8-b5a2-00907fd0950c VFS ada0 PART ada0p3 ada0p3 DEV ada0p3 LABEL gptid/ac128803-5651-11e8-b5a2-00907fd0950c gptid/ac128803-5651-11e8-b5a2-00907fd0950c DEV swap SWAPI see in that output in shows msdosfs.ada0p1, and the rest lines up with what is in /dev/gptid and /etc/fstab. It also lines up with the output of gpart list showing ada0p1 as type efi:
Geom name: ada0 modified: false state: OK fwheads: 16 fwsectors: 63 last: 488397127 first: 40 entries: 128 scheme: GPT Providers: 1. Name: ada0p1 Mediasize: 209715200 (200M) Sectorsize: 512 Stripesize: 0 Stripeoffset: 20480 Mode: r1w1e2 efimedia: HD(1,GPT,ac1172b7-5651-11e8-b5a2-00907fd0950c,0x28,0x64000) rawuuid: ac1172b7-5651-11e8-b5a2-00907fd0950c rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b label: (null) length: 209715200 offset: 20480 type: efi index: 1 end: 409639 start: 40 2. Name: ada0p2 Mediasize: 245677162496 (229G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 209735680 Mode: r1w1e2 efimedia: HD(2,GPT,ac11fbb1-5651-11e8-b5a2-00907fd0950c,0x64028,0x1c99c000) rawuuid: ac11fbb1-5651-11e8-b5a2-00907fd0950c rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b label: (null) length: 245677162496 offset: 209735680 type: freebsd-ufs index: 2 end: 480247847 start: 409640 3. Name: ada0p3 Mediasize: 4172430848 (3.9G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 245886898176 Mode: r1w1e1 efimedia: HD(3,GPT,ac128803-5651-11e8-b5a2-00907fd0950c,0x1ca00028,0x7c591f) rawuuid: ac128803-5651-11e8-b5a2-00907fd0950c rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: (null) length: 4172430848 offset: 245886898176 type: freebsd-swap index: 3 end: 488397126 start: 480247848 Consumers: 1. Name: ada0 Mediasize: 250059350016 (233G) Sectorsize: 512 Mode: r3w3e8 -
What actually failed during the upgrade here? Do you have the failure output?
-
Try:
df -hi /boot/efiYou have a 200MB EFI partition but the filesystem on it may still be much smaller. That is what jimp's fix works past.
-
@stephenw10 The upgrade failed with the following error: 'failed insufficient space remaining for /boot/loader.efi', which is exactly why I was looking at jimp's fix. Only problem was, the install did a umount on /boot/efi just before I got that error. When it did that, it did not revert the umount so I have been trying to figure out exactly where mine should be mounted. Most posts refer to /boot/efi being mounted to /dev/msdosfs/EFISYS, which I don't have. Based on everything I have found, it looks like it should be mounted to /dev/ada0p1. Unfortunately when I do that I also notice that BOOTx64.efi is missing from that directory. I did find where I can replace that file with loader.efi and rename it.
Before I proceed, I just want to make sure I have mounted the correct location to /boot/efi. When this problem first happened, I did run the df -hi /boot/efi and got the following reults:
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/gptid/ac11fbb1-5651-11e8-b5a2-00907fd0950c 222G 18G 186G 9% 69k 30M 0% /I have since temporarily performed a mount_msdosfs /dev/ada0p1 /boot/efi and when I run the same command I get
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/ada0p1 766K 1.5K 765K 0% 2 510 0% /boot/efiWhich according to information I have been able to find online in other forum posts would be correct for some installation, even potentially mine.
-
My understanding is early pfsense installs created a smaller efi partition than is required for current versions of pfsense.
@Sinfonia97 You may be able to patch your system to fix that but after which you are still going to have an edge case system.
What else broke during the many updates and repair attempts?A clean install offers a more reliable solution.
-
Ah that's the issue. The partition is 200MB but the filesystem is only 766K! So that is the problem that jimps instructions should address. You should be able to copy out the EFI data, expand the filesystem to fill the partion then copy the it back.
-
@Patch I only attempted the upgrade once so nothing else broke. It was just the issue that jimp provided the fix for. Unfortunately his instructions were using /dev/msdosfs/EFISYS which my system does not have.
I probably should have mentioned early on I am running bare metal on an old watchguard M500 with an upgraded CPU that was required for the future releases of PFSense.
Here are my CPU stats in case anyone was wondering:
CPU Type Intel(R) Core(TM) i3-4130 CPU @ 3.40GHz 4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads AES-NI CPU Crypto: Yes (inactive) QAT Crypto: No -
@stephenw10 That is exactly what I've been trying to do is the jimp fix. Unfortunately I am needing help that hopefully will confirm what I think is the directory that /boot/efi should be mounted to. Once I figure that out, I should be able to finish with jimp's instructions and correct the size problem and upgrade to 2.8.
-
It looks to be mounted correctly since you have the EFI dircetory structure there. As long as you also have the actual efi file(s) there:
[25.07-RC][admin@6100.stevew.lan]/root: du -ha /boot/efi 656K /boot/efi/efi/freebsd/loader.efi 656K /boot/efi/efi/freebsd/loader-old.efi 1.3M /boot/efi/efi/freebsd 656K /boot/efi/efi/boot/bootx64.efi 656K /boot/efi/efi/boot/bootx64-old.efi 1.3M /boot/efi/efi/boot 2.6M /boot/efi/efi 2.6M /boot/efi -
@stephenw10 When I run that command I currently get the following:
512B /boot/efi/EFI/BOOT/STARTUP.NSH 1.0K /boot/efi/EFI/BOOT 1.5K /boot/efi/EFI 1.5K /boot/efiWhen I look at the STARTUP.NSH file it references BOOTx64.efi. In my research for this issue I found that I should be able to replace the missing BOOTx64.efi file with a copied and renamed loader.efi from the /boot directory. Unfortunately I have not been able to verify that is true. It appears that typically PFSense uses the loader.efi, but my STARTUP.NSH file references BOOTx64.efi.
So this is where I am at currently. Need to verify if I can use a copy of loader.efi to replace my BOOTx64.efi. Also just need to make sure based on the information I have provided that /boot/efi is mounted to the correct directory. In my case I believe it should be /dev/adaop1. Only other location I would consider mounting it to would be /dev/msdosfs/EFI as that is close to what @jimp posted in his fix for the space issue.
-
Hmm, without an efi binary there at all it will fail to boot. So don't reboot without copying back something!
I would say those files are indeed identical:
[25.07-RC][admin@6100.stevew.lan]/root: md5sum /boot/efi/efi/boot/bootx64.efi aa4529badf1fe88a5ca8941c04f144cb /boot/efi/efi/boot/bootx64.efi [25.07-RC][admin@6100.stevew.lan]/root: md5sum /boot/efi/efi/freebsd/loader.efi aa4529badf1fe88a5ca8941c04f144cb /boot/efi/efi/freebsd/loader.efi [25.07-RC][admin@6100.stevew.lan]/root: md5sum /boot/loader.efi aa4529badf1fe88a5ca8941c04f144cb /boot/loader.efiSo you should be able to copy that across
-
@stephenw10 Thank you, that's great to know. That takes care of one part. Now if I can verify that /dev/ada0p1 is the correct location to mount /boot/efi I will be able to finish following @jimp process.
-
The output of
gpart listshould confirm that. There will only be one EFI partition. -
@stephenw10 Thanks again. I ran gpart list before and this is the output I received and why I figured /dev/ada01p was the correct location to mount /boot/efi. Let me know if you see the same thing:
Geom name: ada0 modified: false state: OK fwheads: 16 fwsectors: 63 last: 488397127 first: 40 entries: 128 scheme: GPT Providers: 1. Name: ada0p1 Mediasize: 209715200 (200M) Sectorsize: 512 Stripesize: 0 Stripeoffset: 20480 Mode: r1w1e2 efimedia: HD(1,GPT,ac1172b7-5651-11e8-b5a2-00907fd0950c,0x28,0x64000) rawuuid: ac1172b7-5651-11e8-b5a2-00907fd0950c rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b label: (null) length: 209715200 offset: 20480 type: efi index: 1 end: 409639 start: 40 2. Name: ada0p2 Mediasize: 245677162496 (229G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 209735680 Mode: r1w1e2 efimedia: HD(2,GPT,ac11fbb1-5651-11e8-b5a2-00907fd0950c,0x64028,0x1c99c000) rawuuid: ac11fbb1-5651-11e8-b5a2-00907fd0950c rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b label: (null) length: 245677162496 offset: 209735680 type: freebsd-ufs index: 2 end: 480247847 start: 409640 3. Name: ada0p3 Mediasize: 4172430848 (3.9G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 245886898176 Mode: r1w1e1 efimedia: HD(3,GPT,ac128803-5651-11e8-b5a2-00907fd0950c,0x1ca00028,0x7c591f) rawuuid: ac128803-5651-11e8-b5a2-00907fd0950c rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: (null) length: 4172430848 offset: 245886898176 type: freebsd-swap index: 3 end: 488397126 start: 480247848 Consumers: 1. Name: ada0 Mediasize: 250059350016 (233G) Sectorsize: 512 Mode: r3w3e8This was also before I did a temp mount of /boot/efi to /dev/ada0p1.
-
Yup looks good.
Of course I'd still have a backup of the config etc!
-
@stephenw10 Perfect, thank you again. I already created a backup before the botched upgrade, so I should be good there. May take another one just in case.
-
Wanted to provide an update on where I am at the moment. After finishing the steps from jimp I am now back to where I should have been before attempting the upgrade.
Output from df -hi /boot/efi
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/ada0p1 197M 646K 196M 0% 0 0 - /boot/efiOutput from du -ha /boot/efi
512B /boot/efi/EFI/BOOT/STARTUP.NSH 644K /boot/efi/EFI/BOOT/BOOTx64.efi 645K /boot/efi/EFI/BOOT 645K /boot/efi/EFI 646K /boot/efiOutput from geom -t
Geom Class Provider ada0 DISK ada0 ada0 DEV ada0 PART ada0p1 ada0p1 DEV msdosfs.ada0p1 VFS ada0 PART ada0p2 ada0p2 DEV ada0p2 LABEL gptid/ac11fbb1-5651-11e8-b5a2-00907fd0950c gptid/ac11fbb1-5651-11e8-b5a2-00907fd0950c DEV ffs.gptid/ac11fbb1-5651-11e8-b5a2-00907fd0950c VFS ada0 PART ada0p3 ada0p3 DEV ada0p3 LABEL gptid/ac128803-5651-11e8-b5a2-00907fd0950c gptid/ac128803-5651-11e8-b5a2-00907fd0950c DEV swap SWAPOutput from gpart list
Geom name: ada0 modified: false state: OK fwheads: 16 fwsectors: 63 last: 488397127 first: 40 entries: 128 scheme: GPT Providers: 1. Name: ada0p1 Mediasize: 209715200 (200M) Sectorsize: 512 Stripesize: 0 Stripeoffset: 20480 Mode: r1w1e1 efimedia: HD(1,GPT,ac1172b7-5651-11e8-b5a2-00907fd0950c,0x28,0x64000) rawuuid: ac1172b7-5651-11e8-b5a2-00907fd0950c rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b label: (null) length: 209715200 offset: 20480 type: efi index: 1 end: 409639 start: 40 2. Name: ada0p2 Mediasize: 245677162496 (229G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 209735680 Mode: r1w1e2 efimedia: HD(2,GPT,ac11fbb1-5651-11e8-b5a2-00907fd0950c,0x64028,0x1c99c000) rawuuid: ac11fbb1-5651-11e8-b5a2-00907fd0950c rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b label: (null) length: 245677162496 offset: 209735680 type: freebsd-ufs index: 2 end: 480247847 start: 409640 3. Name: ada0p3 Mediasize: 4172430848 (3.9G) Sectorsize: 512 Stripesize: 0 Stripeoffset: 245886898176 Mode: r1w1e1 efimedia: HD(3,GPT,ac128803-5651-11e8-b5a2-00907fd0950c,0x1ca00028,0x7c591f) rawuuid: ac128803-5651-11e8-b5a2-00907fd0950c rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b label: (null) length: 4172430848 offset: 245886898176 type: freebsd-swap index: 3 end: 488397126 start: 480247848 Consumers: 1. Name: ada0 Mediasize: 250059350016 (233G) Sectorsize: 512 Mode: r3w3e7I know the last 2 shouldn't have changed, which they didn't. Just wanted to put them in here to document the changes.
Now I just need some time to try the upgrade again, and in case everything goes completely south I have time to rebuild.
-
Cool. Does it reboot successfully from there?
-
@stephenw10 Unfortunately I am going to have to wait till I can bring down the network to test. If I take it down now and it doesn't come back up I will be having some hell to pay from the family...lol.
