Latest pfsense+ 23.01 beta freezes on Hyper-v

andrew_241

For Windows Server 2022, the maximum configuration version is 10.0. I typed this in in PowerShell:

[Server]: PS C:\> Update-VMVersion "pfSense Router and Firewall"

Confirm
Are you sure you want to perform this action?
Performing a configuration version update of "pfSense Router and Firewall" will prevent it from being migrated to or
imported on previous versions of Windows. This operation is not reversible.

[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): y
WARNING: No update was performed for the virtual machine "pfSense Router and Firewall" because its configuration
version is already at the maximum level supported by this server.  For clustered servers, the maximum supported
configuration version can be limited by the cluster functional level.

Bob.Dig

@andrew_241 10 working fine here.

Interesting.
Maybe MS is just doing MS things and it depends on the age of your Windows Installation and what not... I have problems running a service in a debian VM in Hyper-V, I always have to boot it twice.

vbjp

@bob-dig
Only consumer version Hyper-v on win 11 support version 11 Windows server even 2022 is on version 10. Enterprise is always slow with latest versions. Server 2019 is stuck on 9.

stephenw10

Anyone hitting this able to confirm it only affects ZFS installs?

jimp

I was wondering about the VM version, but I guess that confirms it. I'd hoped it was just the default being 10 on the older platforms but they could still support the higher one. But that was likely just a red herring anyway since others are working on 10.

I've tried changing my VM in various ways to be similar to the problem one above (higher CPU count, more RAM, dynamic memory on/off, etc.) but nothing so far has made it fail. It boots up perfectly no matter what I do here.

andrew_241

@bob-dig

I may try with a fresh install of Windows Server 2022 when 23.01 is released.

I'm curious; my VM configuration dump says the VM has two COM ports, but Hyper-V doesn't pass the physical one I have to the VMs - third party software is required for that. This isn't exposed in the Hyper-V Manager GUI. Does your VMCX configuration file show ComPort1 and ComPort2?

Bob.Dig

@andrew_241 said in Latest pfsense+ 23.01 beta freezes on Hyper-v:

Does your VMCX configuration file show ComPort1 and ComPort2?

I can't tell because I don't know any of the syntax of you command. I am just an advanced home user, don't know nothing about powershell etc.

jimp

My VM config that works also shows the COM ports, so that's unlikely to be related.

One of our TAC crew who could reproduce this noticed that they could get theirs to boot if they removed the DVD drive from the VM, so that's something else to try.

Mine works with the DVD drive attached (and with no media mounted). Theirs appeared to be set the same but failed until they removed it.

vbjp

Sadly but removing DVD drive did not solve the problem. Even vm that booted successfully after upgrade started displaying same behavior after some reboots. There was the confusion that it only crashing with console open when booting, that seems no longer be the case. It seems that sometimes it just boots after upgrade, but breaks later and sometimes it just broken as soon as you upgrade.

jimp

Might be a long shot, but do your configurations on those VMs have the serial console enabled even though they don't have COM ports mapped in Hyper-V?

There is a known issue in that area with FreeBSD on Azure Gen2 VMs but I wouldn't think it would manifest on local Hyper-V setups since even though the VM contains COM port configurations it doesn't actually have any uart controllers or serial ports visible to the guest.

vbjp

Most likely related thread
https://forum.netgate.com/topic/176572/hyper-v-upgrade-from-22-05-to-23-01-b-20221217-1429-fails-to-boot/7

Will test rc tomorrow.

Also 2.7 ce builds worked ok

vbjp

Tried installing 23.01 rc

2.6ce to 22.1 then 22.5 and 23.1 rc
First boot after upgrade successful then tried rebooting same pattern crashing
First boot after upgrade
rc first.PNG
Reboot
rc reboot.PNG

jimp

I have yet to see an answer to my last question:

Do these VMs have the serial console option enabled in settings? (System > Advanced, Admin Access tab)? If so, disable it before upgrading.

vbjp

@jimp they don't have serial enabled at least when it booted after upgrade I checked in ui and serial check box was unchecked if that's what you mean.

Bob.Dig

@jimp said in Latest pfsense+ 23.01 beta freezes on Hyper-v:

Do these VMs have the serial console option enabled in settings? (System > Advanced, Admin Access tab)? If so, disable it before upgrading.

Mine doesn't and I never had any problem. I don't even know what a serial console is.

jimp

If you can, look at /boot/loader.conf and /boot/loader.conf.local and post the contents from before the upgrade (or the first boot after if that works would be even better).

The similar problem that happens in Hyper-V is definitely loader related, but it has to do with serial output. We have a potential fix but since it's in the loader it's a bit too disruptive to get into this release at this stage. Even though Hyper-V doesn't have serial consoles per se in Gen2 it may still interfere. Most of us still can't even reproduce the boot failure here in Hyper-V which is a bit maddening.

vbjp

@jimp
Got contents of loader.conf couldn't find loader.conf.local (sorry for picture format was easiest to grab this way)
This is from 22.5 before upgrade to RC. Rolled back to vm snapshot.
(Tried upgrade to 23.1 again but this time it crashed even on first boot)

Contents of boot folder no local file

jimp

There isn't always a .local file unless someone created it manually, but it's good to check for it in case it is there. Nothing stands out in that loader config, definitely no sign of serial or console config which is what I was curious about.

Did you check for and remove any optical drives from that VM? We have one person here who only saw it crash/fail to boot with an optical drive on the VM.

vbjp

@jimp
No DVD drive

Bob.Dig

@bob-dig said in Latest pfsense+ 23.01 beta freezes on Hyper-v:

Maybe MS is just doing MS things and it depends on the age of your Windows Installation and what not... I have problems running a service in a debian VM in Hyper-V, I always have to boot it twice.

~~After this MS-patchday my problem with the debian VM is gone. But I had some other changes too... it is a possibility.~~