Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Proxmox SR-IOV VF pass-through to pfSense VM

    Scheduled Pinned Locked Moved Virtualization
    proxmoxhardwaresriovkvm
    11 Posts 2 Posters 2.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      tim4532
      last edited by tim4532

      Hi all,

      I am trying to pass-through a NIC VF to pfSense VM but failed.

      For the VF creation, I followed this article https://docs.virtuozzo.com/virtuozzo_hybrid_server_7_installation_on_asrock_rack/sr-iov/assigning-sr-iov-network.html
      Everything goes smooth until I boot into pfSense. It shows the error (I have 8 VFs enabled and passed-through)

      ixv0: <Intel(R) X550 Virtual Function> mem 0xfeb5c000-0xfeb5ffff,0xfeb60000-0xfeb63fff at device 27.0 on pci0
      ixv0: ...reset_hw() failure: Reset Failed!
      ixv0: IFDI_ATTACH_PRE failed 5
      device_attach: ixv0 attach returned 5

      I am on Proxmox 7 with X550-AT2 NIC. VM is using BIOS instead of UEFI (maybe the problem? not tested and have no idea).

      I wonder if anyone is having the same issue and could give some advice on configuring SR-IOV with pfSense running on Proxmox VM?

      I also found this post on FreeBSD bug-track: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211062
      Is this patch applied on pfSense FreeBSD too? If anyone knows.

      Thank you in advance.

      NollipfSenseN T 2 Replies Last reply Reply Quote 0
      • NollipfSenseN
        NollipfSense @tim4532
        last edited by

        @tim4532 said in Proxmox SR-IOV VF pass-through to pfSense VM:

        VM is using BIOS instead of UEFI (maybe the problem?

        Could be...I always use UEFI...

        pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
        pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

        T 1 Reply Last reply Reply Quote 0
        • T
          tim4532 @NollipfSense
          last edited by tim4532

          @NollipfSense Just tried every combination and swap to OPNsense but still failed. I might have to compile my own Intel driver to support VF. Anyway thank you for the reply.

          REF: https://forum.opnsense.org/index.php?topic=9576.0

          NollipfSenseN 1 Reply Last reply Reply Quote 0
          • NollipfSenseN
            NollipfSense @tim4532
            last edited by NollipfSense

            @tim4532 Have you looked at this: https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html
            or this one: https://www.reddit.com/r/homelab/comments/cm87qr/tutorial_enabling_sriov_for_intel_nic_x550t2_on/?rdt=57094

            pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
            pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

            T 1 Reply Last reply Reply Quote 0
            • T
              tim4532 @NollipfSense
              last edited by

              @NollipfSense Thank you, and yes, the current setup is identical to those tutorials. Not really helpful for me.

              1 Reply Last reply Reply Quote 0
              • T
                tim4532 @tim4532
                last edited by tim4532

                After Netgate announced that pfSense+ is a joke and *ucks around with home users. I have carefully review my steps to enable IXV interface on OPNsense (pfSense still fails).
                I found out that you must have the VF parent interface link up in order to function normally.
                Also VF would not allow VLAN tagging on guest VM else it will drop those packet. Set VLAN on hypervisor E.g.

                ip link set [INT_NAME] vf [VF_NUM] vlan [VID]
                

                To use the VF in internal network. You must have "spoofchk off" on hypervisor. E.g.

                ip link set [INT_NAME] vf [VF_NUM] spoofchk on
                

                Not tested, to allow MAC spoof on guest VM "trust on":

                ip link set [INT_NAME] vf [VF_NUM] trust on
                

                How to make these config persistent?? Use systemd to make a startup service.
                REF (Proxmox persistent VF): https://forum.proxmox.com/threads/enabling-sr-iov-for-intel-nic-x550-t2-on-proxmox-6.56677/
                REF (big thanks, detail VF config): https://forum.opnsense.org/index.php?topic=9576.0
                REF (NVIDIA, VF config): https://enterprise-support.nvidia.com/s/article/howto-configure-mac-anti-spoofing-for-vms-over-sr-iov

                NollipfSenseN 1 Reply Last reply Reply Quote 0
                • NollipfSenseN
                  NollipfSense @tim4532
                  last edited by

                  @tim4532 said in Proxmox SR-IOV VF pass-through to pfSense VM:

                  After Netgate announced that pfSense+ is a joke and *ucks around with home users. I have carefully review my steps to enable IXV interface on OPNsense (pfSense still fails).

                  This is uncalled for and should have left unsaid...glad you found a solution despite not being sure why you would need to passthrough a virtual firewall to a real firewall when there are Linux bridges...

                  pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
                  pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

                  T 1 Reply Last reply Reply Quote 0
                  • T
                    tim4532 @NollipfSense
                    last edited by

                    @NollipfSense Kinda lost my mind there ignore it.. To reduce the overhead. Linux bridge eats a lot CPU power. Also I built the system inside an small AIO server (NAS, remote server, video rendering etc...) which has only 2x 10G port. I need the virtualization to help to split network segment and most important is MAC spoof for my ISP.

                    NollipfSenseN 1 Reply Last reply Reply Quote 0
                    • NollipfSenseN
                      NollipfSense @tim4532
                      last edited by

                      @tim4532 said in Proxmox SR-IOV VF pass-through to pfSense VM:

                      @NollipfSense Kinda lost my mind there ignore it.. To reduce the overhead. Linux bridge eats a lot CPU power. Also I built the system inside an small AIO server (NAS, remote server, video rendering etc...) which has only 2x 10G port. I need the virtualization to help to split network segment and most important is MAC spoof for my ISP.

                      Now I am understanding...I just got the sick joke. Onto your hardware, it's best to have a separate NIC for Proxmox management...can you hardware support another NIC?

                      pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
                      pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

                      T 1 Reply Last reply Reply Quote 0
                      • T
                        tim4532 @NollipfSense
                        last edited by tim4532

                        @NollipfSense All PCIe slots were occupied since I have an MATX board and only have two full-size slots. My Proxmox management is on a Linux bridge (on-band), and I reserved one 1G physical port (out-band, on demand).

                        FYI: My board got 2x 1G and 2x 10G ports.

                        NollipfSenseN 1 Reply Last reply Reply Quote 0
                        • NollipfSenseN
                          NollipfSense @tim4532
                          last edited by

                          @tim4532 said in Proxmox SR-IOV VF pass-through to pfSense VM:

                          FYI: My board got 2x 1G and 2x 10G ports.

                          I would definitely use one of the 10G for pfSense WAN set for vtnet0 connected directly to your ISP modem. Once you assigned say ens2f0 to vtnet0, you don't need to passthrough the entire NIC, just plug the cable from your ISP and it will automatically passthrough...same goes for LAN if you have an external switch, do the same as you did with WAN and connect Proxmox's management port to the switch...you would have one port available for whatever you want.

                          pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
                          pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post
                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.