Thank you all for your responses, here is some more information - I was a bit tired last night at 3am. BLUF - the nic passthrough works great and is much faster than going through the Proxmox bridges once you get the physical ports mapped over and manually re-assign all the VLAN's and interfaces, etc. However, it inexplicably causes all kinds of other bizarre issues with pfSense seemingly unrelated to interfaces at all. I am all but certain if I rolled a new box with all the hardware setup exactly this way and manually reconfigured every setting via the GUI that it would work perfectly.
The version is pfsense Plus 24.11 fully (23.11 was a typo).
(restoring onto 2.7.2) Keep in mind that a backed up pfSense config is meant to be restored on the same machine. It could be restored into another device, but be ready to deal with the Interface changes.
Yes, I went through and made sure that the interfaces are in the exact same order and have the exact same name as on the old system using screenshots. I also have made sure that all of the VLAN's are changed to the correct parent interface (LAN) which changed. I've done this 6 or 7 times, so I know it is being done correctly.
pfSense-upgrade -d -c gets multiple errors
I will restore the broken VM and post the full details of all errors and commands hopefully tonight.
So it was working as expected but slow before you tried to move to passing the NICs through to pfSense in Proxmox.
Let me explain better - I recently replaced a 4 port 2.5G nic with a 4 port 10G nic as part of an upgrade to the rest of the LAN to 10G. It currently works fine with the 4 port 10G nic when used via Proxmox bridge interfaces, but the bridges simply are not able to get more than ~2G through them.
The 4 port nic currently has 3 of the ports mapped to bridges, and those bridges are assigned to the pfSense VM. All of the other VM's and the Proxmox host itself are on two separate physical NIC cards. I didn't grab the output of pciconf while booted with the Nic passed through, but I can do that tonight. It is 10Gtek and the part number is XL710-10G-4S(4xSFP+) - it's a cheap card but I've tried four other brands that were no good and I'm using these in 3 other servers. The firmware is fully upgraded. I added Nic3 to move the VMLan off of that physical Nic so I can pass the entire Nic card through to the pfSense. In the broken configuration with Nic2 passed through to the VM I was able to see at least 5G without making any additional tweaks to the OS or Nic such as mtu, offload settings, etc.
Upon starting up the pfsense with the 10G nic passed through it of course asks me to assign interfaces. LAN is now ixl1 and WAN is now ixl3. At this point I have internet and can connect to the pfsense gui via the LAN - so the passthrough is working and the ethernet ports are mapped correctly. However, booting takes forever and the gui is painfully slow. There seem to be lots of errors related to the pfsense Plus registration at boot, I will capture dmesg and add it. Also in the GUI if I go to System -> register rather than saying "Your device does not require registration" it appears ready to accept an activation token - however even if I put in the token the gui just freezes for 20 minutes and nothing changes.
Could this be a license issue for pfsense plus? If so, how am I supposed to swap hardware in my pfsense devices in the middle of the night - do I have to do a bunch of license coordination stuff with Netgate the day before to swap out a Nic? This can't possibly be the case...can it?
Server Nics
x used; - not used
Nic1 2.5G | x | -> Proxmox host
Nic2 10G | - x x x | -> Pfsense VM
Nic3 10G | x | -> VMLan (All other VM's)
Label is a literal label sticker on the server case for the 4 port Nic
#Label Name BEFORE AFTER MAC
0 ----- ----- ---- ixl0 98:b7:85:XX:XX:XX CONFIRMED
1 pflan LAN vtnet0 ixl1 98:b7:85:XX:XX:XX CONFIRMED
2 pfsync pfsync vtnet2 ixl2 98:b7:85:XX:XX:XX CONFIRMED
3 WAN WAN10500 vtnet1 ixl3 98:b7:85:XX:XX:XX CONFIRMED
Using Proxmox bridges:
$ pciconf -lv
...
virtio_pci4@pci0:6:21:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x1af4 device=0x1000 subvendor=0x1af4 subdevice=0x0001
vendor = 'Red Hat, Inc.'
device = 'Virtio network device'
class = network
subclass = ethernet
Proxmox host before passthrough:
# lshw -C network
...
*-network:3
description: Ethernet interface
product: Ethernet Controller X710 for 10GbE SFP+
vendor: Intel Corporation
physical id: 0.3
bus info: pci@0000:01:00.3
logical name: enp1s0f3np3
version: 01
serial: 98:b7:85:XX:XX:XX
size: 10Gbit/s
capacity: 10Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm msi msix pciexpress bus_master cap_list rom ethernet physical fibre 10000bt-fd
configuration: autonegotiation=off broadcast=yes driver=i40e driverversion=6.8.12-10-pve duplex=full firmware=9.54 0x8000fb7a 1.2527.0 latency=0 link=yes multicast=yes speed=10Gbit/s
resources: iomemory:600-5ff iomemory:600-5ff irq:16 memory:60e0000000-60e07fffff memory:60e2800000-60e2807fff memory:80a00000-80a7ffff memory:60e2000000-60e21fffff memory:60e2820000-60e289ffff
...
VLAN's - there are 10 but only 2 of them are in use and are not critical. The pfsense LAN port obviously needs to be VLAN aware, and on Debian you would configure these settings for the Nic's in /etc/network/interfaces i.e.:
auto enp1s0f1np1
iface enp1s0f1np1 inet manual
post-up /sbin/ethtool -K enp1s0f1np1 rxvlan off
post-up /sbin/ethtool -K enp1s0f1np1 rx-vlan-offload off
auto pflan
iface pflan inet manual
bridge-ports enp1s0f1np1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 20 30 40 50 60 70 80 90 55
