QEMU VM: Version 2.8 Won't Boot
-
Hi All,
Tried to upgrade my pfsense vm today. Its virtualized under Incus in TrueNAS and is working as expected in 2.7.2. When trying to upgrade to 2.8, everything seems to work as planned, but when it comes up in 2.8 it hangs here
Under 2.7 these seem to be QEMU PCIe rootport devices
Is there a change in the driver for QEMU in FreeBSD 15.0 thats breaking this for me?
-
This post is deleted! -
and here
ce versionNot exactly the same, but its always that some devices are missing in freebsd15 and kvm
-
@netblues Interesting. So for you, it boots, but devices were just not showing up at all?
In my case, I'm doing PCI-E passthru of an Intel X710, not a virtio network adapter. Although there is a virtio network adapter in the system, even if its not being used.
-
@nickf1227 I get all sorts of cases.
Under redhat kvm, no virtio devices at all.At hetzner kvm, I only get ONE!! and it misses the rest.
Certainly there is something going on with virtialization and freebsd15
Proxmox has better luck too.
What worked for me is changing the emulated machine from q35 to 440fx.
But then this doesn;t support passthrough If i recall. -
@netblues Right...so the newer kernel works with an emulated chipset form 1996 but not an emulated chipset from 2010. Lol.
Sigh.
All I wanted was working DNS with Kea DHCP. :(
-
@nickf1227 The thing is, that if this is broken upstream, its hard for pfsense to fix.
Unless of course it fails on aws or azure ;-)
But I doubt it. aws is xen, not kvm.Up to now, we can't agree on a replicated issue :( to build a solid case.
-
@netblues Sure, this is clearly a FreeBSD problem, no doubt. I just wish we'd see genuine efforts to get an open source firewall on Linux, from Netgate or otherwise. With the marketshare of KVM, this kinda thing shouldnt happen.
-
Are these systems using Q35+SeaBIOS or Q35+OVMF (UEFI)?
Granted I have only tried on Proxmox but mine are all working, even if I force modern virtio NICs. I don't have anything using passthrough, though.
My VMs use either i440fx+SeaBIOS or (more commonly) q35+OVMF, though I did try one q35+SeaBIOS and it still worked here, too.
That first screenshot it's hard to tell if it froze there or just stopped displaying kernel messages on that console. Maybe try adding a serial port to the VM (if it doesn't already have one) and attach to that and see if you get any other output there.
-
@jimp I tried the uefi path, on q35 but it fails to boot completely.
Most probably it needs a reinstall.
Reinstalling with plus to a different vm isn't an option due to licensing too.(and I can't justify spending $120 AND put debug effort too.)
-
@netblues said in QEMU VM: Version 2.8 Won't Boot:
@jimp I tried the uefi path, on q35 but it fails to boot completely.
Most probably it needs a reinstall.If you were running BIOS before, then it would need reinstalled. It needs an EFI disk added to the VM and then installed as EFI so it has a proper EFI partition/loader/etc. You'd most likely also need to make sure Secure Boot is disabled in there.
-
@jimp Yes, indeed. Secure boot was disabled, but still, with no efi, its futile.
-
@nickf1227 said in QEMU VM: Version 2.8 Won't Boot:
an open source firewall on Linux
Well, there is ipfire, but with all the bells and whistles pfsense has, rebasing to linux is only for april fools day
-
@jimp Hey man, thanks for your reply
That screenshot comes from the console itself while its booting, a-la a display device/VNC. Happy to give it another go with a serial connection, but I dont think thats possible without a virtio-VM agent?
-
@nickf1227 said in QEMU VM: Version 2.8 Won't Boot:
@jimp Hey man, thanks for your reply
That screenshot comes from the console itself while its booting, a-la a display device/VNC. Happy to give it another go with a serial connection, but I dont think thats possible without a virtio-VM agent?
I'm not sure what all might be available on TrueNAS but at least on Proxmox for a serial console on a guest you run
qm terminal <id>from the Proxmox host (e.g. over ssh). IIRC that's just part ofqemu-serverso it may be there already.It may not be printing anything there but it's good to rule it out at least.
-
Since you're hitting this locally try setting the loader value:
debug.acpi.disabled="pci"It looks like this is an ACPI bug preventing the OS finding the PCI devices. Setting that falls back to probing the bus/tree.
I expect it to still fail but the changes output would be useful in diagnosing the problem.
Edit: Actually that seems to prevent any detection in Q35....
-
@stephenw10 Finally its all related to uefi.
Booting via bios to q35 and newer freebsd leads to missing virtio interfaces.Reinstalling with uefi boot makes virtio interfaces available.
This is probably an upgade nightmare, since there is no warning.
A warning can be easily added by checking if an update is on q35 without uefi boot.Just an idea
-
It's a bug in the ACPI driver that happened to be present when we branched 25.03 and 2.8: https://reviews.freebsd.org/D48285
We picked back the fix two weeks ago it should be in the current public beta (May 10th). Are you testing that?
-
@stephenw10 So you are saying that q35 with bios boot would work?
I need to boot 2.8 beta first under q35.
If the fix isn't there how can I test it? -
Yes as I understand it that issue is fixed in the May 10th 2.8 beta. I assume you were testing with an earlier beta version?
