@robsonvitorm
First off, to be hyper specific, for my tests HW is a Lenovo m720q with a AJ??929 x8 Riser Adapter and an HP/HPE 331T (Broadcom 5719 4 port)
pfSense on Bare Metal = All 4 Ports work as expected, no PHY error on boot.
pfSense on ProxMox VE 8.1 (Debian based OS) = All 4 ports work as expected, Intel_IOMMU enabled in Grub
pfSense on Ubuntu 22.04 LTS (Debian based OS) = Only bge0 works, bge1 fails to init. Intel_IOMMU and MSR enabled in Grub
Intel_IOMMU:
Edit /etc/default/grub
Append GRUB_CMDLINE_LINUX_DEFAULT with additional "intel_iommu=on" at the end.
MSRs:
Edit /etc/default/grub
Append GRUB_CMDLINE_LINUX_DEFAULT with additional "kvm.ignore_msrs=1" at the end.
After either change:
sudo grub-mkconfig -o /boot/grub/grub.cfg
On the Ubuntu Host, I also blacklisted the bge drivers so the Bare Metal Ubuntu did not load them and attach to the NIC Ports:
EDIT/Create: /etc/modprobe.d/vfio.conf
blacklist tg3
options vfio-pci ids=14e4:1657
# Prevent Broadcom bg3 driver from loading in Bare Metal OS - keep card ports free for VM passthrough
Then push changes to initram: sudo update-initramfs -u
This step did not seem to be needed with Proxmox.
These mods to the OS came from a thread I found on GPU passthrough, which I adapted for NIC passthrough:
https://askubuntu.com/questions/1406888/ubuntu-22-04-gpu-passthrough-qemu
(Though I just noticed that I did not do Step 14, so will have to swap that drive back in and see if it makes any difference)
-Casper