Broadcom bge / HP 331T error on boot - Attaching PHYs failed

Casper042

I have an HP 331T passed through to a QEMU VM, but when the VM boots I get errors as soon as it gets past port 1 (bge0) and it won't discover the other 3 ports on the card.

I found some info on the main site and added 3 System Tunables, but it doesn't seem to help.

Anyone have ideas as to how to get this working?
I know the card in general works because screenshot 3 was taken over the network to the card (VM has zero Virtual NICs)

https://imgur.com/a/vMBAaJo

Thanks

stephenw10

Hmm, so it does attach as expected to the 1st PHY and creates bge0?

Does it work if those NICs are not passed through as PCIe devices?

Pictures of the log files are generally far less useful than the actual log text. If only because it's then searchable.

You could try booting in verbose mode to see if that give you any additional error output.

Interrupt boot at the loader menu and go to the loader prompt. Enter: boot -v

Steve

Casper042

@stephenw10 Thanks Steve, I will grab the logs and then swap the SSD in the machine (don't want to rebuild the bare metal OS just yet) tomorrow and do a fresh install of pfSense on bare metal and get the same boot -v logs from there for comparison.

Where do I grab the logs from after I boot -v?
What's the log file name/path? Or is there a spot in the GUI that makes it easy to download?

stephenw10

You can see the boot log in the gui or it would be in /var/log/dmesg.boot

What I meant was do the NICs work if you just add them in the hypervisor with virtual NICs in pfSense?

Casper042

OK I have NFC what I am doing in regards to getting the Bridge interfaces added to Ubuntu and adding them to the VM.
I thought I figured it out, but after adding the 2 virtual nics to the VM and booting it, no dhcp address is coming through on LAN interface.

sudo ip link add brg0 type bridge
sudo ip link set enp1s0f0 master brg0

Then I added the Virtual NIC and set brg0 as the Bridge interface,

Anyway, I am going to go swap the boot drive and try pfSense on bare metal as well.

Here are the logs before I started trying to revert the vfio changes and add the bridge interfaces and all that above:
https://github.com/Casper042/pfSense_bge/

Casper042

So installing Bare Metal seems to work fine.
Not sure if we should keep this here or punt it to the VM category.

I have uploaded the logs to the same GitHub as above.
I installed latest pfSense ISO via Ventoy on a dedicated NVMe drive.
It detected the onboard Intel and the 4 port Broadcom just fine, zero issues or tunables required.
I even auto detected the 2 ports (bge0/bge3) during the setup wizard.

So now I am just wondering what I did wrong on the VM...

stephenw10

Yup it could be the PCIe pass through just doesn't do all that's required for that NIC. Seems like a virtualization issue as you say.

Casper042

I have a buddy who used to write Linux drivers for a big company, though not sure how much he knows about BSD.
I might hit him up in a few days if I can't figure it out.

If pfSense somehow supported Docker/Podman this would be a non issue, but since it's BSD kernel I don't see that happening.