Latest pfsense+ 23.01 beta freezes on Hyper-v

jimp

There isn't always a .local file unless someone created it manually, but it's good to check for it in case it is there. Nothing stands out in that loader config, definitely no sign of serial or console config which is what I was curious about.

Did you check for and remove any optical drives from that VM? We have one person here who only saw it crash/fail to boot with an optical drive on the VM.

vbjp

@jimp
No DVD drive

Bob.Dig

@bob-dig said in Latest pfsense+ 23.01 beta freezes on Hyper-v:

Maybe MS is just doing MS things and it depends on the age of your Windows Installation and what not... I have problems running a service in a debian VM in Hyper-V, I always have to boot it twice.

~~After this MS-patchday my problem with the debian VM is gone. But I had some other changes too... it is a possibility.~~

vbjp

Just quick update. The official release of 23.01 still does not boot in Hyper-v.
Tried on 2 different hosts (Both Windows server 2022 standard all updates installed) still getting stuck at boot.

fiblimitless

@vbjp said in Latest pfsense+ 23.01 beta freezes on Hyper-v:

Just quick update. The official release of 23.01 still does not boot in Hyper-v.
Tried on 2 different hosts (Both Windows server 2022 standard all updates installed) still getting stuck at boot.

same issue while updating with all settings/packages/configuration in place.

If you open the Hyper-V machine and check the logs while updating, you will see a log saying disk out of space, and the system continues, as it finished. I believe that's the reason. corrupt/partial update.

Negate minimum requirement says you needs a minimum of 8GB. (mine was 5GB fixed)

You can expand the vhdx file and shell into pfsense and resize the partitions.

Here is a good link on how adjust your disk after expanding the vhdx
https://people.freebsd.org/~rodrigc/doc/handbook/disks-growing.html

Or

backup your pfsense
create a new vhdx disk (I set mine to 15GB)
install pfSense-CE-2.6.0
update to pfsense + 22.05 then 23.01
restore the backup.

I can confirm it worked for me.
Cheers.

vbjp

@fiblimitless my pfsense vm has 30gb (other one has 60gb both not booting) drive assigned to it is that not enough? Possibly not same problem.

fiblimitless

@vbjp shell into pfsense check your partitions: gpart show

what do you have?

stephenw10

Yes, is the disk space error actually whilst updating the main root filesystem or possibly the EFI partition?

Snk

I'm having the same problem... I tried to perform a new installation increasing the total disk size from 20 to 40GB and I still couldn't complete the update.
Hangs on: masks 0x00ff0000. 0x0000ff00. 0x000000ff. 0xff000000

stephenw10

Are you seeing a drive space error whilst updating?

Snk

@stephenw10

I don't see any space errors in the GUI and not even in the console (accessed through Hyper-V)

stephenw10

Ok, so it just fails to boot at the first reboot during upgrade?
That's what we might expect given the previous failures.

Do you have the upgrade log showing what was updated before that reboot?
Specifically did it update the boot loader?

Steve

Snk

@stephenw10

I ordered the update via GUI, I realized that some packages are pending update in the "Package Updater", but as recommended on the Netgate website, I didn't do any package update.
pfsense downloads the full update and tells you to restart the VM.
When starting the VM, it hangs at masks 0x00ff0000. 0x0000ff00. 0x000000ff. 0xff000000 and I don't have any more logs being shown in the VM console.
Exactly the same message posted by the colleague appears:

Even redoing the installation from scratch and increasing the VM disk size, the same error occurs.
Are there any other procedures you would recommend so that I can gather more logs?

stephenw10

If you SSH to the firewall and then upgrade from the console menu using option 13 you can see exactly what happens before the reboot. That might give some idea as to what's happening.

However we did look into this for some time before release and didn't yet find the root cause.

https://redmine.pfsense.org/issues/13895

vbjp

@fiblimitless
Sorry for late reply was busy.
Here is what I get when I ran gpart show on my 22.05 vm before upgrade to 23.01 (can't run on upgraded as it does not boot)

Snk

Any updates on the error in question?

Snk

Any news, guys?

jimp

@snk said in Latest pfsense+ 23.01 beta freezes on Hyper-v:

Any news, guys?

Check the Redmine linked a few posts above.

stephenw10

There is one update and that's that this appears probably linked to the loader issues we are seeing in Azure: https://redmine.pfsense.org/issues/13895#note-9

Yet that doesn't apply directly because we are still unable to replicate it with multiple NICs defined.

However is everyone who is hitting this issue in Hyper-V using a VM with 3 or more NICs defined?

Snk

In my case I only have two nic's in my VM.
It would be the WAN and the LAN. However, with the LAN having a bridged vswitch between 3 ports.
Worse than even installing again and upgrading by upgrade, when jumping from version 22.05 to 23.01 the error persists.
I've tried several combinations of configurations, but without success... Keeping the ZFS standard.
I'll try the UFS pattern later.