boot failure upgrading to 25.03.b.20250306.0140 i

[netblues]

@stephenw10 said in boot failure upgrading to 25.03.b.20250306.0140 i:

Mmm, so just to review; if you change the disk type to something other than vitio block device (modern) it finds it and boots but still fails to find the virtio (modern) NICs?

Yes, this is exactly the case.

If I change virtio ethernet to e1000, it also finds network interfaces.

[netblues]

@stephenw10 fiddling around I think I found the curlpit.
At some point kvm switched default emulated machine from i440fx to q35.
Latest supported os under latest kvm (or at least redhat supported) is freebsd14

So if hypervisor defaults are

Then virtio drivers don't work under freebsd15

changing to

makes virtio work again.

The problem is that changing this with virsh edit isn't easy, since it also changes pci configuration.
The obvious way forward is creating a new freebsd 14 vm, with manual config.
Add interfaces as needed,
adjust vlans/passthrough/bridges and maybe mac addresses as need
and then move the disk subsystem to the new config with virsh edit.

This allowed me to upgrade ce 2.7.2 to ce 2.8 without issues, and most probably it will also work
for plus,
I don't know if older chipset has any performance or other penalties, but pf used to work under this, and only changed to q35 recently.
I suspect freebsd15 dropped/changed something in virtio driver support.

[netblues]

@netblues And also plus updated to 25.03 beta, with no issues. Dirty upgrade too.

[stephenw10]

Hmm, interesting. I tried usssing a Q35 machine in Proxmox and it booted fine. That in itself also didn't change the PCI device type either.

I think it must be changing the presented PCIe controller since it seems to fail to detect entirely any NICs or the drive controller. Still couldn't make Proxmox do it though even with a bunch of custom command hackery.

[netblues]

@stephenw10 actually pcie isn't supported in i440fx, you have to change this to pci
In older kvm implementations, creating a new freebsd13.1 would create a q35 based virtual machine.
Now if you create a new freebsd14 i440fx is selected.

Proxmox isn't a redhat derivative as you know , too.

In any case, no issues in new installations. Upgrades will hurt "randomly" since this isn't really pfsense related. Its upstream freebsd

[stephenw10]

@netblues said in boot failure upgrading to 25.03.b.20250306.0140 i:

actually pcie isn't supported in i440fx,

Hmm, interesting. I wonder if my change didn't really apply somehow then.

[jimp]

Were you using an odd mix like q35+BIOS instead of q35+UEFI?

FreeBSD 15.x recently changed some things in probing of older ISA style devices, which I wouldn't expect to impact things here, but it's possible some quirk in that combination triggered an issue.

I use q35+OVMF/UEFI on nearly all my Proxmox VMs with all virtio devices (Proxmox 8.3.5) and everything runs perfectly.

I still have a few that are using i440fx+BIOS and all virtio devices and they're good, too.

[stephenw10]

Hmm, yes I probably did end up with Q35+BIOS.

Do you see the PCI devices shown as (modern) in the boot logs?

[jimp]

I don't see any mention of modern in my logs, they still show as VirtIO (legacy) even on recently created VMs.

In the info qtree of all the VMs it doesn't seem to express a preference either way:

                disable-legacy = "off"
                disable-modern = false

Same on every device

[netblues]

@jimp said in boot failure upgrading to 25.03.b.20250306.0140 i:

Were you using an odd mix like q35+BIOS instead of q35+UEFI?

Everything was/is q35 + bios, but the issue was that after booting couldn't load disk voiume.
Just changing driver to scsci /or sata from virtio would allow startup normaly till network cards not being detected at later stage.

This can't be a uefi thing IMHO.

[jimp]

@netblues said in boot failure upgrading to 25.03.b.20250306.0140 i:

@jimp said in boot failure upgrading to 25.03.b.20250306.0140 i:

Were you using an odd mix like q35+BIOS instead of q35+UEFI?

Everything was/is q35 + bios, but the issue was that after booting couldn't load disk voiume.
Just changing driver to scsci /or sata from virtio would allow startup normaly till network cards not being detected at later stage.

This can't be a uefi thing IMHO.

Declaring that without trying it seems hasty. There isn't much of a compelling reason to run q35+BIOS so it's probably a much less tested combination comparatively.

It absolutely can make a difference in how the system probes devices and boots.

[netblues]

@jimp

Tried, but it will take more than changing bios settings. Most probably a reinstall

[jimp]

I setup a fresh VM as q35+BIOS and it worked fine and showed legacy virtio devices by default.

I forced modern devices with qm set 157 --args "-global virtio-net-pci.disable-legacy=on -global virtio-blk-pci.disable-legacy=on" and it still booted fine and worked on the network, dmesg output shows modern for the disk/block device and network devices.

This is on a fresh 2.8.0 VM but it should be using the same base OS version as 25.03 builds.

That's all on Proxmox though so it's possible KVM is doing something differently, maybe it's a newer version of KVM/qemu or something else changed.

[stephenw10]

Yeah it feels like whatever is upstream of those PCI devices fails to be detected and hence anything downstream isn't seen at all. I did try setting -global virtio-pci.disable-legacy=on, which should be valid, but it fails to boot entirely.

[jimp]

I tried that and hit the same failure, hence my limiting it to NICs and disk devices, which did work.

One thing I did read when looking into this is that it may make a difference if the root device is PCI vs PCIe but I didn't look into how to check/change that on Proxmox yet.

Though at least I did confirm that the modern NICs and block devices do work on their own, so the original stated issue is somewhat of a red herring and the workaround may have allowed it to function, but not for the reason that was thought.

[stephenw10]

Yup, that's exactly what I did. Good to get confirmation at least.

[Maff]

good Morning, i am currently running 25.03-BETA (amd64)
built on Tue Feb 4. and when ever i try to upgrade to the latest i get this failure but i am not using any virtual machines, is a minisfourm ms-01, onboard nics and nvme drive

any pointers i could look for?

[Maff]

@Maff

fresh install cleared it up