VM on Proxmox Stops Passing Traffic even with VT-d NICs

yakatz

My virtualized pfSense on Proxmox stops passing traffic every so often - I can still access the admin interface and I can see the logs complaining that the WAN gateway is not accessible.

I have seen a lot of posts about issues with pfSense on proxmox and the vtnet driver, but I also tried vmx and had the same issue. I enabled PCI passthrough, so my network interfaces are now igb0 (WAN) and ixl0 (vlans 300-307, LAN/OPT1-7, all tagged).

The underlying system is a Supermicro SYS-5019D-FN8TP.

The only other error I see is a kernel message complaining that another MAC address is claiming the WAN public IP address and when I check the ARP tables, that MAC address is the BMC of the chassis and is showing the correct MAC/IP combination on the internal network.

Any thoughts of what else I can check?

viragomann

@yakatz said in VM on Proxmox Stops Passing Traffic even with VT-d NICs:

The only other error I see is a kernel message complaining that another MAC address is claiming the WAN public IP address

This is a problem you should solve for sure.

and when I check the ARP tables, that MAC address is the BMC of the chassis and is showing the correct MAC/IP combination on the internal network.

Is it a separate NIC or a shared one?
It has an IP on the internal network? This would mean, it should not have an L2 connection to the WAN.

How did you configure both? static or DHCP?

NollipfSense

@yakatz said in VM on Proxmox Stops Passing Traffic even with VT-d NICs:

I enabled PCI passthrough

As you already know that Proxmox is based on Debian and I shall assumed that you're on version 8 which has a "modern" kernel. This kernel, because it's booted by the host doesn't like PCI pass-through; so, as soon as one pass-through any PCI device, the kernel writes to a denylist and block. So, one has to create vfio-pci.conf file in
/etc/modprobe.d and add: options vfio-pci disable_denylist=1
Sadly, this info is not in the manual yet

Alternatively, if pfSense have vtnet0 and vtnet1 assigned to the appropriate ethernet port, just plug in the cable and it will automatically passthrough...no need to passthrough the entire NIC.

yakatz

It looks like a firmware update put the IPMI on the motherboard in Failover NIC mode instead of Dedicated mode. I switched it back to dedicated and it has been stable for a few days.