23.05 fails to boot on Hyper-V after VM power off, workaround

ttmcmurry

Hello! This is an extension of other posts and redmine issues with booting on Hyper-V.

My host: Windows Server 2022, fully up to date as of May 2023. Hyper-V role installed. AMD Ryzen 5700G, 32GB RAM, Intel 4x10Gbit SFP+ NIC

My VM: Generation 10, UEFI, 2 vCPU, 6GB RAM, 20GB VHDX, 1 vNIC (trunked, all vlans) via 1 vSwitch, All integration services offered, No automated checkpoints, all other settings are defaults, except MAC Spoofing is enabled.

pfSense: 23.05. Updated on 5/24/2023. Prior pfSense+ 23.01, updated on 4/24/2023. Prior pfSense CE 2.6, updated on 4/8/2023. Prior pfSense CE 2.5.2, in November 2022.

Reproduce:

Take Hyper-V checkpoint just in case
Upgrade pfSense 23.05 from 23.01
pfSense downloads, updates, reboots, gets back to UI all seems normal
Take Hyper-V checkpoint again
Halt system
Power on VM
Stuck at UEFI screen (see linked forum posts for the screenshot)
Roll back via last checkpoint, pfSense is working (it has not been halted from the VM point of view)

Workaround:

Assuming Hyper-V checkpoints have been made...
Halt pfSense
Boot pfSense
Hangs at UEFI boot screen
Power cycle VM, at pfSense boot screen...
Select Option 8
Select Option 2, then 2 again (cycle from boot option 1 -> 2 -> back to 1)
Select Option 1
Select Option 1 to boot
pfSense Boots normally

The workaround needs to be manually performed at each boot, otherwise it hangs at the aforementioned UEFI boot screen. It is not necessary to boot into the last image (in my case 23.01), it is only necessary to toggle the boot entry from 1->2->1.

There seems to be something wrong with the boot entry when zfs/default is automatically selected at power-on. Cycling from default to the backup (in my case 23.01) then back to default (23.05) fixes the issue, then pfSense boots normally.

Based on the notes in redmine #13895, it seems reopening the bug/creating a new bug report is in order.

Sources:
https://redmine.pfsense.org/issues/13895
https://forum.netgate.com/topic/176572/hyper-v-upgrade-from-22-05-to-23-01-b-20221217-1429-fails-to-boot
https://forum.netgate.com/topic/176778/pfsense-refuses-to-boot-on-hyper-v

Bob.Dig

@ttmcmurry said in 23.05 fails to boot on Hyper-V after VM power off, workaround:

MAC Spoofing

After having problems some months ago, I advise to disable MAC Spoofing with hyper-v.

ttmcmurry

@Bob-Dig

Does enabling mac spoofing affect the VM's ability to boot at UEFI?

Bob.Dig

@ttmcmurry said in 23.05 fails to boot on Hyper-V after VM power off, workaround:

Does enabling mac spoofing affect the VM's ability to boot at UEFI?

I don't remember anymore what the exact problem was, only that I mitigated it by disabling MAC-Spoofing, sry.

andrew_241

@ttmcmurry said in 23.05 fails to boot on Hyper-V after VM power off, workaround:

The workaround needs to be manually performed at each boot, otherwise it hangs at the aforementioned UEFI boot screen. It is not necessary to boot into the last image (in my case 23.01), it is only necessary to toggle the boot entry from 1->2->1.

I noticed that I had to do that a few months ago with 23.01. But I no longer have access to those options at the boot screen for some reason.

I spun up a Windows Server 2019 VM and installed a nested VM (configuration version 9.0) of pfSense 23.05. It was able to boot. Exporting the nested pfSense VM and moving it "up one level" to run as a VM under Windows Server 2022 resulted in the freezing issue. Upgrading the configuration version to 10.0 did not help.

I'm currently running pfSense 23.05 as a VM with UFS, and have no problems.

SteveITS

@ttmcmurry I’m not seeing it on this page, is Secure Boot off?
https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-hyper-v.html
“Warning
Secure boot must be disabled for the VM to boot pfSense software.”

ttmcmurry

@SteveITS Secure Boot is off. It's indeed impossible to boot pfSense on Hyper-V, if it is enabled. I have personally tested that theory. As you said, this is covered in their documentation.

ttmcmurry

Since no further details have come to light, I have created a new bug in redmine:

https://redmine.pfsense.org/issues/14450