Upgrade from 2.7.2 to 2.8.0 Failed and now /boot/efi/ empty

Sinfonia97

UPDATE: Found the following in my /etc/fstab file and now I am even less sure where /boot/efi should be mounted . Per this post in Unable to upgrade from 2.7.1 to 2.7.2 I should see something like:

/dev/msdosfs/EFISYS       /boot/efi       msdosfs rw              2       2

but I see this:

/dev/gptid/ac11fbb1-5651-11e8-b5a2-00907fd0950c	/	ufs	rw	1	1
/dev/gptid/ac128803-5651-11e8-b5a2-00907fd0950c	none	swap	sw	0	0

Just trying to provide as much info as possible in case anyone can help.

Patch

@Sinfonia97 Sounds like a corrupted install to me.
I would

Save pfsense configuration
Do a clean install from a recent installer to ensure partitions are consistent with more recent requirements (v2.7.2 or 2.8 or 2.81)
Document what low level file structure is used with a good current install so you maybe able to guess what went wrong with the prior corruption (partitions sizes, hardware failure, edge case configuration, edge case update series)

Sinfonia97

@Patch Thanks, I am kind of hoping if I can figure out exactly where it needs to be mounted I can use the steps that were provided in other threads to fix the EFI. The steps used before clearly work and even help with the size of the EFI partition. My problem is figuring out where exactly /boot/efi should be mounted to create the backup of the files and the perform the rest of the steps. In what I have been finding, there possibly could be a few places to mount /boot/efi. I ran a geom -t just a bit ago and it pointed me to likely needing to mount it at /dev/ada0p1. The output of geom -t was:

Geom                                                 Class      Provider
ada0                                                 DISK       ada0
  ada0                                               DEV
  ada0                                               PART       ada0p1
    ada0p1                                           DEV
    msdosfs.ada0p1                                   VFS
  ada0                                               PART       ada0p2
    ada0p2                                           DEV
    ada0p2                                           LABEL      gptid/ac11fbb1-5651-11e8-b5a2-00907fd0950c
      gptid/ac11fbb1-5651-11e8-b5a2-00907fd0950c     DEV
      ffs.gptid/ac11fbb1-5651-11e8-b5a2-00907fd0950c VFS
  ada0                                               PART       ada0p3
    ada0p3                                           DEV
    ada0p3                                           LABEL      gptid/ac128803-5651-11e8-b5a2-00907fd0950c
      gptid/ac128803-5651-11e8-b5a2-00907fd0950c     DEV
      swap                                           SWAP

I see in that output in shows msdosfs.ada0p1, and the rest lines up with what is in /dev/gptid and /etc/fstab. It also lines up with the output of gpart list showing ada0p1 as type efi:

Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 488397127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 209715200 (200M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r1w1e2
   efimedia: HD(1,GPT,ac1172b7-5651-11e8-b5a2-00907fd0950c,0x28,0x64000)
   rawuuid: ac1172b7-5651-11e8-b5a2-00907fd0950c
   rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
   label: (null)
   length: 209715200
   offset: 20480
   type: efi
   index: 1
   end: 409639
   start: 40
2. Name: ada0p2
   Mediasize: 245677162496 (229G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 209735680
   Mode: r1w1e2
   efimedia: HD(2,GPT,ac11fbb1-5651-11e8-b5a2-00907fd0950c,0x64028,0x1c99c000)
   rawuuid: ac11fbb1-5651-11e8-b5a2-00907fd0950c
   rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 245677162496
   offset: 209735680
   type: freebsd-ufs
   index: 2
   end: 480247847
   start: 409640
3. Name: ada0p3
   Mediasize: 4172430848 (3.9G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 245886898176
   Mode: r1w1e1
   efimedia: HD(3,GPT,ac128803-5651-11e8-b5a2-00907fd0950c,0x1ca00028,0x7c591f)
   rawuuid: ac128803-5651-11e8-b5a2-00907fd0950c
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 4172430848
   offset: 245886898176
   type: freebsd-swap
   index: 3
   end: 488397126
   start: 480247848
Consumers:
1. Name: ada0
   Mediasize: 250059350016 (233G)
   Sectorsize: 512
   Mode: r3w3e8

stephenw10

What actually failed during the upgrade here? Do you have the failure output?

stephenw10

Try: df -hi /boot/efi

You have a 200MB EFI partition but the filesystem on it may still be much smaller. That is what jimp's fix works past.

Sinfonia97

@stephenw10 The upgrade failed with the following error: 'failed insufficient space remaining for /boot/loader.efi', which is exactly why I was looking at jimp's fix. Only problem was, the install did a umount on /boot/efi just before I got that error. When it did that, it did not revert the umount so I have been trying to figure out exactly where mine should be mounted. Most posts refer to /boot/efi being mounted to /dev/msdosfs/EFISYS, which I don't have. Based on everything I have found, it looks like it should be mounted to /dev/ada0p1. Unfortunately when I do that I also notice that BOOTx64.efi is missing from that directory. I did find where I can replace that file with loader.efi and rename it.

Before I proceed, I just want to make sure I have mounted the correct location to /boot/efi. When this problem first happened, I did run the df -hi /boot/efi and got the following reults:

Filesystem                                         Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/gptid/ac11fbb1-5651-11e8-b5a2-00907fd0950c    222G     18G    186G     9%     69k   30M    0%   /

I have since temporarily performed a mount_msdosfs /dev/ada0p1 /boot/efi and when I run the same command I get

Filesystem     Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/ada0p1    766K    1.5K    765K     0%       2   510    0%   /boot/efi

Which according to information I have been able to find online in other forum posts would be correct for some installation, even potentially mine.

Patch

My understanding is early pfsense installs created a smaller efi partition than is required for current versions of pfsense.

@Sinfonia97 You may be able to patch your system to fix that but after which you are still going to have an edge case system.
What else broke during the many updates and repair attempts?

A clean install offers a more reliable solution.

stephenw10

Ah that's the issue. The partition is 200MB but the filesystem is only 766K! So that is the problem that jimps instructions should address. You should be able to copy out the EFI data, expand the filesystem to fill the partion then copy the it back.

Sinfonia97

@Patch I only attempted the upgrade once so nothing else broke. It was just the issue that jimp provided the fix for. Unfortunately his instructions were using /dev/msdosfs/EFISYS which my system does not have.

I probably should have mentioned early on I am running bare metal on an old watchguard M500 with an upgraded CPU that was required for the future releases of PFSense.

Here are my CPU stats in case anyone was wondering:

CPU Type	Intel(R) Core(TM) i3-4130 CPU @ 3.40GHz
                4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
                AES-NI CPU Crypto: Yes (inactive)
                QAT Crypto: No

Sinfonia97

@stephenw10 That is exactly what I've been trying to do is the jimp fix. Unfortunately I am needing help that hopefully will confirm what I think is the directory that /boot/efi should be mounted to. Once I figure that out, I should be able to finish with jimp's instructions and correct the size problem and upgrade to 2.8.

stephenw10

It looks to be mounted correctly since you have the EFI dircetory structure there. As long as you also have the actual efi file(s) there:

[25.07-RC][admin@6100.stevew.lan]/root: du -ha /boot/efi
656K	/boot/efi/efi/freebsd/loader.efi
656K	/boot/efi/efi/freebsd/loader-old.efi
1.3M	/boot/efi/efi/freebsd
656K	/boot/efi/efi/boot/bootx64.efi
656K	/boot/efi/efi/boot/bootx64-old.efi
1.3M	/boot/efi/efi/boot
2.6M	/boot/efi/efi
2.6M	/boot/efi

Sinfonia97

@stephenw10 When I run that command I currently get the following:

512B	/boot/efi/EFI/BOOT/STARTUP.NSH
1.0K	/boot/efi/EFI/BOOT
1.5K	/boot/efi/EFI
1.5K	/boot/efi

When I look at the STARTUP.NSH file it references BOOTx64.efi. In my research for this issue I found that I should be able to replace the missing BOOTx64.efi file with a copied and renamed loader.efi from the /boot directory. Unfortunately I have not been able to verify that is true. It appears that typically PFSense uses the loader.efi, but my STARTUP.NSH file references BOOTx64.efi.

So this is where I am at currently. Need to verify if I can use a copy of loader.efi to replace my BOOTx64.efi. Also just need to make sure based on the information I have provided that /boot/efi is mounted to the correct directory. In my case I believe it should be /dev/adaop1. Only other location I would consider mounting it to would be /dev/msdosfs/EFI as that is close to what @jimp posted in his fix for the space issue.

stephenw10

Hmm, without an efi binary there at all it will fail to boot. So don't reboot without copying back something!

I would say those files are indeed identical:

[25.07-RC][admin@6100.stevew.lan]/root: md5sum /boot/efi/efi/boot/bootx64.efi 
aa4529badf1fe88a5ca8941c04f144cb  /boot/efi/efi/boot/bootx64.efi
[25.07-RC][admin@6100.stevew.lan]/root: md5sum /boot/efi/efi/freebsd/loader.efi
aa4529badf1fe88a5ca8941c04f144cb  /boot/efi/efi/freebsd/loader.efi
[25.07-RC][admin@6100.stevew.lan]/root: md5sum /boot/loader.efi
aa4529badf1fe88a5ca8941c04f144cb  /boot/loader.efi

So you should be able to copy that across

Sinfonia97

@stephenw10 Thank you, that's great to know. That takes care of one part. Now if I can verify that /dev/ada0p1 is the correct location to mount /boot/efi I will be able to finish following @jimp process.

stephenw10

The output of gpart list should confirm that. There will only be one EFI partition.

Sinfonia97

@stephenw10 Thanks again. I ran gpart list before and this is the output I received and why I figured /dev/ada01p was the correct location to mount /boot/efi. Let me know if you see the same thing:

Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 488397127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 209715200 (200M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r1w1e2
   efimedia: HD(1,GPT,ac1172b7-5651-11e8-b5a2-00907fd0950c,0x28,0x64000)
   rawuuid: ac1172b7-5651-11e8-b5a2-00907fd0950c
   rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
   label: (null)
   length: 209715200
   offset: 20480
   type: efi
   index: 1
   end: 409639
   start: 40
2. Name: ada0p2
   Mediasize: 245677162496 (229G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 209735680
   Mode: r1w1e2
   efimedia: HD(2,GPT,ac11fbb1-5651-11e8-b5a2-00907fd0950c,0x64028,0x1c99c000)
   rawuuid: ac11fbb1-5651-11e8-b5a2-00907fd0950c
   rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 245677162496
   offset: 209735680
   type: freebsd-ufs
   index: 2
   end: 480247847
   start: 409640
3. Name: ada0p3
   Mediasize: 4172430848 (3.9G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 245886898176
   Mode: r1w1e1
   efimedia: HD(3,GPT,ac128803-5651-11e8-b5a2-00907fd0950c,0x1ca00028,0x7c591f)
   rawuuid: ac128803-5651-11e8-b5a2-00907fd0950c
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 4172430848
   offset: 245886898176
   type: freebsd-swap
   index: 3
   end: 488397126
   start: 480247848
Consumers:
1. Name: ada0
   Mediasize: 250059350016 (233G)
   Sectorsize: 512
   Mode: r3w3e8

This was also before I did a temp mount of /boot/efi to /dev/ada0p1.

stephenw10

Yup looks good.

Of course I'd still have a backup of the config etc!

Sinfonia97

@stephenw10 Perfect, thank you again. I already created a backup before the botched upgrade, so I should be good there. May take another one just in case.

Sinfonia97

Wanted to provide an update on where I am at the moment. After finishing the steps from jimp I am now back to where I should have been before attempting the upgrade.

Output from df -hi /boot/efi

Filesystem     Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/ada0p1    197M    646K    196M     0%       0     0     -   /boot/efi

Output from du -ha /boot/efi

512B	/boot/efi/EFI/BOOT/STARTUP.NSH
644K	/boot/efi/EFI/BOOT/BOOTx64.efi
645K	/boot/efi/EFI/BOOT
645K	/boot/efi/EFI
646K	/boot/efi

Output from geom -t

Geom                                                 Class      Provider
ada0                                                 DISK       ada0
  ada0                                               DEV
  ada0                                               PART       ada0p1
    ada0p1                                           DEV
    msdosfs.ada0p1                                   VFS
  ada0                                               PART       ada0p2
    ada0p2                                           DEV
    ada0p2                                           LABEL      gptid/ac11fbb1-5651-11e8-b5a2-00907fd0950c
      gptid/ac11fbb1-5651-11e8-b5a2-00907fd0950c     DEV
      ffs.gptid/ac11fbb1-5651-11e8-b5a2-00907fd0950c VFS
  ada0                                               PART       ada0p3
    ada0p3                                           DEV
    ada0p3                                           LABEL      gptid/ac128803-5651-11e8-b5a2-00907fd0950c
      gptid/ac128803-5651-11e8-b5a2-00907fd0950c     DEV
      swap                                           SWAP

Output from gpart list

Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 488397127
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 209715200 (200M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r1w1e1
   efimedia: HD(1,GPT,ac1172b7-5651-11e8-b5a2-00907fd0950c,0x28,0x64000)
   rawuuid: ac1172b7-5651-11e8-b5a2-00907fd0950c
   rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
   label: (null)
   length: 209715200
   offset: 20480
   type: efi
   index: 1
   end: 409639
   start: 40
2. Name: ada0p2
   Mediasize: 245677162496 (229G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 209735680
   Mode: r1w1e2
   efimedia: HD(2,GPT,ac11fbb1-5651-11e8-b5a2-00907fd0950c,0x64028,0x1c99c000)
   rawuuid: ac11fbb1-5651-11e8-b5a2-00907fd0950c
   rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 245677162496
   offset: 209735680
   type: freebsd-ufs
   index: 2
   end: 480247847
   start: 409640
3. Name: ada0p3
   Mediasize: 4172430848 (3.9G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 245886898176
   Mode: r1w1e1
   efimedia: HD(3,GPT,ac128803-5651-11e8-b5a2-00907fd0950c,0x1ca00028,0x7c591f)
   rawuuid: ac128803-5651-11e8-b5a2-00907fd0950c
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 4172430848
   offset: 245886898176
   type: freebsd-swap
   index: 3
   end: 488397126
   start: 480247848
Consumers:
1. Name: ada0
   Mediasize: 250059350016 (233G)
   Sectorsize: 512
   Mode: r3w3e7

I know the last 2 shouldn't have changed, which they didn't. Just wanted to put them in here to document the changes.

Now I just need some time to try the upgrade again, and in case everything goes completely south I have time to rebuild.

stephenw10

Cool. Does it reboot successfully from there?