Latest pfsense+ 23.01 beta freezes on Hyper-v
-
We managed to replicate this locally and are investigating. So far it's not obvious what the difference is that triggers it.
You might try installing as UFS, in our tests that boots fine which seems like a big clue!There's a bug open for it:
https://redmine.pfsense.org/issues/13895Steve
-
I'm having the same problem. In case it may be of help, here is a dump of my VMCX configuration file, with the server name and VMCX filename removed:
[Server]: PS C:\> $tempVM = (Compare-VM -Copy -Path "{Path to .vmcx}" -GenerateNewID).VM [Server]: PS C:\> $tempVM | select * ParentCheckpointId : ParentCheckpointName : VMName : pfSense Router and Firewall VMId : dcfe7a58-eadd-4d11-b5ec-ff674db02362 CheckpointFileLocation : C:\ProgramData\Microsoft\Windows\Hyper-V ConfigurationLocation : C:\ProgramData\Microsoft\Windows\Hyper-V SmartPagingFileInUse : False SmartPagingFilePath : C:\ProgramData\Microsoft\Windows\Hyper-V SnapshotFileLocation : C:\ProgramData\Microsoft\Windows\Hyper-V AutomaticStartAction : Start AutomaticStartDelay : 0 AutomaticStopAction : ShutDown AutomaticCriticalErrorAction : Pause AutomaticCriticalErrorActionTimeout : 30 AutomaticCheckpointsEnabled : False CPUUsage : 0 MemoryAssigned : 0 MemoryDemand : 0 MemoryStatus : NumaAligned : NumaNodesCount : NumaSocketCount : Heartbeat : IntegrationServicesState : IntegrationServicesVersion : 0.0 Uptime : 00:00:00 OperationalStatus : {Ok} PrimaryOperationalStatus : Ok SecondaryOperationalStatus : StatusDescriptions : {Operating normally} PrimaryStatusDescription : Operating normally SecondaryStatusDescription : Status : Operating normally ReplicationHealth : Normal ReplicationMode : None ReplicationState : Disabled ResourceMeteringEnabled : False CheckpointType : Production EnhancedSessionTransportType : VMBus Groups : Version : 10.0 VirtualMachineType : PlannedVirtualMachine VirtualMachineSubType : Generation2 Notes : State : Off ComPort1 : VMComPort (Name = 'COM 1', VMName = 'pfSense Router and Firewall') [Id = 'Microso ft:DCFE7A58-EADD-4D11-B5EC-FF674DB02362\8E3A359F-559A-4B6A-98A9-1690A6100ED7\0', VMId = 'dcfe7a58-eadd-4d11-b5ec-ff674db02362'] ComPort2 : VMComPort (Name = 'COM 2', VMName = 'pfSense Router and Firewall') [Id = 'Microso ft:DCFE7A58-EADD-4D11-B5EC-FF674DB02362\8E3A359F-559A-4B6A-98A9-1690A6100ED7\1', VMId = 'dcfe7a58-eadd-4d11-b5ec-ff674db02362'] DVDDrives : {DVD Drive on SCSI controller number 0 at location 1} FibreChannelHostBusAdapters : {} FloppyDrive : HardDrives : {Hard Drive on SCSI controller number 0 at location 0} RemoteFxAdapter : VMIntegrationService : {Guest Service Interface, Heartbeat, Key-Value Pair Exchange, Shutdown...} DynamicMemoryEnabled : False MemoryMaximum : 1099511627776 MemoryMinimum : 536870912 MemoryStartup : 8589934592 ProcessorCount : 8 BatteryPassthroughEnabled : False Generation : 2 IsClustered : False ParentSnapshotId : ParentSnapshotName : Path : C:\ProgramData\Microsoft\Windows\Hyper-V SizeOfSystemFiles : 45090 GuestControlledCacheTypes : False LowMemoryMappedIoSpace : 134217728 HighMemoryMappedIoSpace : 536870912 HighMemoryMappedIoBaseAddress : 68182605824 LockOnDisconnect : Off CreationTime : 1/25/2023 4:17:00 PM Id : dcfe7a58-eadd-4d11-b5ec-ff674db02362 Name : pfSense Router and Firewall NetworkAdapters : {Network Adapter, Network Adapter} CimSession : CimSession: . ComputerName : SERVER IsDeleted : FalsePerhaps someone can compare this output to theirs if they have a functioning install of the 23.01 beta in Hyper-V on Windows Server 2022.
-
@andrew_241 said in Latest pfsense+ 23.01 beta freezes on Hyper-v:
Version : 10.0
The only real difference I see is mine that work are version
11.0, not version10.0.Can you stop the VM, and in the Hyper-V manager select Actions > Upgrade Configuration Version so it gets updated to
11.0?I don't know about Hyper-V but we see things like that all the time in ESX where newer FreeBSD versions will tickle some behavior that either needs a newer VM config version or newer OS selected.
That may only be on Windows 11, though. My Windows 10 box is powered off at the moment so I can't double check that until tomorrow.
-
For Windows Server 2022, the maximum configuration version is 10.0. I typed this in in PowerShell:
[Server]: PS C:\> Update-VMVersion "pfSense Router and Firewall" Confirm Are you sure you want to perform this action? Performing a configuration version update of "pfSense Router and Firewall" will prevent it from being migrated to or imported on previous versions of Windows. This operation is not reversible. [Y] Yes [A] Yes to All [N] No [L] No to All [?] Help (default is "Y"): y WARNING: No update was performed for the virtual machine "pfSense Router and Firewall" because its configuration version is already at the maximum level supported by this server. For clustered servers, the maximum supported configuration version can be limited by the cluster functional level. -
@andrew_241 10 working fine here.
Interesting.
Maybe MS is just doing MS things and it depends on the age of your Windows Installation and what not... I have problems running a service in a debian VM in Hyper-V, I always have to boot it twice. -
@bob-dig
Only consumer version Hyper-v on win 11 support version 11 Windows server even 2022 is on version 10. Enterprise is always slow with latest versions. Server 2019 is stuck on 9. -
Anyone hitting this able to confirm it only affects ZFS installs?
-
I was wondering about the VM version, but I guess that confirms it. I'd hoped it was just the default being 10 on the older platforms but they could still support the higher one. But that was likely just a red herring anyway since others are working on 10.
I've tried changing my VM in various ways to be similar to the problem one above (higher CPU count, more RAM, dynamic memory on/off, etc.) but nothing so far has made it fail. It boots up perfectly no matter what I do here.
-
I may try with a fresh install of Windows Server 2022 when 23.01 is released.
I'm curious; my VM configuration dump says the VM has two COM ports, but Hyper-V doesn't pass the physical one I have to the VMs - third party software is required for that. This isn't exposed in the Hyper-V Manager GUI. Does your VMCX configuration file show ComPort1 and ComPort2?
-
@andrew_241 said in Latest pfsense+ 23.01 beta freezes on Hyper-v:
Does your VMCX configuration file show ComPort1 and ComPort2?
I can't tell because I don't know any of the syntax of you command. I am just an advanced home user, don't know nothing about powershell etc.
-
My VM config that works also shows the COM ports, so that's unlikely to be related.
One of our TAC crew who could reproduce this noticed that they could get theirs to boot if they removed the DVD drive from the VM, so that's something else to try.
Mine works with the DVD drive attached (and with no media mounted). Theirs appeared to be set the same but failed until they removed it.
-
Sadly but removing DVD drive did not solve the problem. Even vm that booted successfully after upgrade started displaying same behavior after some reboots. There was the confusion that it only crashing with console open when booting, that seems no longer be the case. It seems that sometimes it just boots after upgrade, but breaks later and sometimes it just broken as soon as you upgrade.
-
Might be a long shot, but do your configurations on those VMs have the serial console enabled even though they don't have COM ports mapped in Hyper-V?
There is a known issue in that area with FreeBSD on Azure Gen2 VMs but I wouldn't think it would manifest on local Hyper-V setups since even though the VM contains COM port configurations it doesn't actually have any uart controllers or serial ports visible to the guest.
-
Most likely related thread
https://forum.netgate.com/topic/176572/hyper-v-upgrade-from-22-05-to-23-01-b-20221217-1429-fails-to-boot/7Will test rc tomorrow.
Also 2.7 ce builds worked ok
-
Tried installing 23.01 rc
2.6ce to 22.1 then 22.5 and 23.1 rc
First boot after upgrade successful then tried rebooting same pattern crashing
First boot after upgrade
Reboot
-
I have yet to see an answer to my last question:
Do these VMs have the serial console option enabled in settings? (System > Advanced, Admin Access tab)? If so, disable it before upgrading.
Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!
Need help fast? Netgate Global Support!
Do not Chat/PM for help!
-
@jimp they don't have serial enabled at least when it booted after upgrade I checked in ui and serial check box was unchecked if that's what you mean.
-
@jimp said in Latest pfsense+ 23.01 beta freezes on Hyper-v:
Do these VMs have the serial console option enabled in settings? (System > Advanced, Admin Access tab)? If so, disable it before upgrading.
Mine doesn't and I never had any problem. I don't even know what a serial console is.
-
If you can, look at
/boot/loader.confand/boot/loader.conf.localand post the contents from before the upgrade (or the first boot after if that works would be even better).The similar problem that happens in Hyper-V is definitely loader related, but it has to do with serial output. We have a potential fix but since it's in the loader it's a bit too disruptive to get into this release at this stage. Even though Hyper-V doesn't have serial consoles per se in Gen2 it may still interfere. Most of us still can't even reproduce the boot failure here in Hyper-V which is a bit maddening.
-
@jimp
Got contents of loader.conf couldn't find loader.conf.local (sorry for picture format was easiest to grab this way)
This is from 22.5 before upgrade to RC. Rolled back to vm snapshot.
(Tried upgrade to 23.1 again but this time it crashed even on first boot)
Contents of boot folder no local file
