pfSense 2.7.2 SFXGE Virtual Function (Solarflare SFC9100 family)
-
Hello, I have been running pfSense as a guest on a proxmox host. I have a solarflare card, which I pass through using PCI passthrough of both interfaces (sfxge0 which is my WAN and sfxge1 which is my LAN). This works fine, but I am required to use virtio (paravirtualized network) to bridge the lan connection to proxmox and other VMS.
I would like to use the virtual functions on the solarflare card using SR-IOV. I am able to set up VFs on the host, and they work fine in other VMs using PCI pass through (fedora, debian, etc).
The following on my host gives me the VFs:
echo 3 > /sys/class/net/enp1s0f1np1/device/sriov_numvfs
echo 0000:01:00.2 > /sys/bus/pci/devices/0000\:01\:00.2/driver/unbind
echo 0000:01:00.3 > /sys/bus/pci/devices/0000\:01\:00.3/driver/unbind
echo 0000:01:00.4 > /sys/bus/pci/devices/0000\:01\:00.4/driver/unbindHowever, when I attempt to passthrough the VF into pfSense, I am not able to load the driver for the interface. I have attempted "hw.pci.honor_msi_blacklist=0" in my loader conf (no effect). This seems to be a problem with how freebsd is handling the PCI interface/VF (possibly the solarflare driver) when setting up interrupts:
During boot:
sfxge1: <Solarflare SFC9100 family> mem 0xfb800000-0xfbffffff,0xfc000000-0xfc003fff at device 0.0 on pci2
sfxge1: Couldn't setup interrupts
device_attach: sfxge1 attach returned 12The VF is showing up as follows:
none0@pci0:2:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x1924 device=0x1903 subvendor=0x1924 subdevice=0x800c
vendor = 'Solarflare Communications'
device = 'SFC9120 10G Ethernet Controller (Virtual Function)'
class = network
subclass = ethernetDetailed dump of both interfaces (PF passed through for SFXGE0, and VF for SFXGE1):
sfxge0@pci0:1:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x1924 device=0x0903 subvendor=0x1924 subdevice=0x800c
vendor = 'Solarflare Communications'
device = 'SFC9120 10G Ethernet Controller'
class = network
subclass = ethernet
bar [10] = type I/O Port, range 32, base 0x5000, size 256, enabled
bar [18] = type Memory, range 64, base 0xfc800000, size 8388608, enabled
bar [20] = type Memory, range 64, base 0xfd040000, size 16384, enabled
cap 01[40] = powerspec 3 supports D0 D1 D2 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit
cap 10[70] = PCI-Express 2 endpoint max data 128(2048) FLR NS
max read 512
link x8(x8) speed 8.0(8.0) ASPM disabled(L0s/L1)
cap 11[b0] = MSI-X supports 32 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x2000]
cap 03[d0] = VPD
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 3 corrected
ecap 0003[140] = Serial 1 000f53ffff5699e0
ecap 0017[1c0] = TPH Requester 1
ecap 0018[290] = LTR 1
PCI-e errors = Correctable Error Detected
Corrected = Receiver Error
Bad DLLP
Replay Timer Timeout
none0@pci0:2:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x1924 device=0x1903 subvendor=0x1924 subdevice=0x800c
vendor = 'Solarflare Communications'
device = 'SFC9120 10G Ethernet Controller (Virtual Function)'
class = network
subclass = ethernet
bar [10] = type Memory, range 64, base 0xfb800000, size 8388608, enabled
bar [18] = type Memory, range 64, base 0xfc000000, size 16384, enabled
cap 10[70] = PCI-Express 2 endpoint max data 512(2048) FLR NS
max read 512
link x0(x8) speed 0.0(8.0) ASPM disabled(L0s/L1)
cap 11[b0] = MSI-X supports 8 messages
Table in map 0x18[0x0], PBA in map 0x18[0x2000]
ecap 0000[100] = unknown 0
ecap 0017[110] = TPH Requester 1