Can only use certain interfaces based on uio_pci_generic or vfio-pci



  • I have a system running ESXi 6.7 with PCI passthrough. The NIC chipsets are 82599EB and X722. I can only get one set of NICs bound depending on the DPDK UIO driver.

    For 82599EB, uio_pci_generic works

    0000:0b:00.0      8086:10fb   5.0 GT/s x32 uio_pci_generic                                 
    0000:0b:00.1      8086:10fb   5.0 GT/s x32 uio_pci_generic                                 
    0000:0d:00.0      8086:37d2   5.0 GT/s x32                 Example VPD                     RV: 0x d7
    0000:0d:00.1      8086:37d2   5.0 GT/s x32                 Example VPD                     RV: 0x d7
    0000:0d:00.2      8086:37d0   5.0 GT/s x32                 Example VPD                     RV: 0x d7
    0000:0d:00.3      8086:37d0   5.0 GT/s x32                 Example VPD                     RV: 0x d7
    0000:1d:00.0      8086:10fb   5.0 GT/s x32 uio_pci_generic                                 
    0000:1d:00.1      8086:10fb   5.0 GT/s x32 uio_pci_generic                                 
    0000:1f:00.0      15ad:07b0   5.0 GT/s x32 vmxnet3     
    
    TenGigabitEthernet1d/0/0           3    down  TenGigabitEthernet1d/0/0
                                          Link speed: unknown
                                          Ethernet address 00:1b:21:bc:a2:ba
                                          Intel 82599
                                            carrier down 
                                            flags: pmd maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum
                                            rx: queues 1 (max 128), desc 1024 (min 32 max 4096 align 8)
                                            tx: queues 1 (max 64), desc 1024 (min 32 max 4096 align 8)
                                            pci: device 8086:10fb subsystem 8086:000c address 0000:1d:00.00 numa 0
                                            module: unknown
                                            max rx packet len: 15872
                                            promiscuous: unicast off all-multicast off
                                            vlan offload: strip off filter off qinq off
                                            rx offload avail:  vlan-strip ipv4-cksum udp-cksum tcp-cksum tcp-lro 
                                                               macsec-strip vlan-filter vlan-extend jumbo-frame scatter 
                                                               security keep-crc 
                                            rx offload active: ipv4-cksum jumbo-frame scatter 
                                            tx offload avail:  vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum 
                                                               tcp-tso macsec-insert multi-segs security 
                                            tx offload active: udp-cksum tcp-cksum multi-segs 
                                            rss avail:         ipv4-tcp ipv4-udp ipv4 ipv6-tcp-ex ipv6-udp-ex ipv6-tcp 
                                                               ipv6-udp ipv6-ex ipv6 
                                            rss active:        none
                                            tx burst function: ixgbe_xmit_pkts
                                            rx burst function: ixgbe_recv_pkts
                                        
    TenGigabitEthernet1d/0/1           4    down  TenGigabitEthernet1d/0/1
                                          Link speed: unknown
                                          Ethernet address 00:1b:21:bc:a2:bb
                                          Intel 82599
                                            carrier down 
                                            flags: pmd maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum
                                            rx: queues 1 (max 128), desc 1024 (min 32 max 4096 align 8)
                                            tx: queues 1 (max 64), desc 1024 (min 32 max 4096 align 8)
                                            pci: device 8086:10fb subsystem 8086:000c address 0000:1d:00.01 numa 0
                                            module: id SFP/SFP+/SFP28, compatibility:
                                                    vendor: CISCO-EQUIV, part SFP-10G-LRM-OEM
                                                    revision: B2, serial: THCLRM010206, date code: 150202
                                            max rx packet len: 15872
                                            promiscuous: unicast off all-multicast off
                                            vlan offload: strip off filter off qinq off
                                            rx offload avail:  vlan-strip ipv4-cksum udp-cksum tcp-cksum tcp-lro 
                                                               macsec-strip vlan-filter vlan-extend jumbo-frame scatter 
                                                               security keep-crc 
                                            rx offload active: ipv4-cksum jumbo-frame scatter 
                                            tx offload avail:  vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum 
                                                               tcp-tso macsec-insert multi-segs security 
                                            tx offload active: udp-cksum tcp-cksum multi-segs 
                                            rss avail:         ipv4-tcp ipv4-udp ipv4 ipv6-tcp-ex ipv6-udp-ex ipv6-tcp 
                                                               ipv6-udp ipv6-ex ipv6 
                                            rss active:        none
                                            tx burst function: ixgbe_xmit_pkts
                                            rx burst function: ixgbe_recv_pkts
                                        
    TenGigabitEthernetb/0/0            1    down  TenGigabitEthernetb/0/0
                                          Link speed: unknown
                                          Ethernet address 00:1b:21:80:69:78
                                          Intel 82599
                                            carrier down 
                                            flags: pmd maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum
                                            rx: queues 1 (max 128), desc 1024 (min 32 max 4096 align 8)
                                            tx: queues 1 (max 64), desc 1024 (min 32 max 4096 align 8)
                                            pci: device 8086:10fb subsystem 8086:7a11 address 0000:0b:00.00 numa 0
                                            module: unknown
                                            max rx packet len: 15872
                                            promiscuous: unicast off all-multicast off
                                            vlan offload: strip off filter off qinq off
                                            rx offload avail:  vlan-strip ipv4-cksum udp-cksum tcp-cksum tcp-lro 
                                                               macsec-strip vlan-filter vlan-extend jumbo-frame scatter 
                                                               security keep-crc 
                                            rx offload active: ipv4-cksum jumbo-frame scatter 
                                            tx offload avail:  vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum 
                                                               tcp-tso macsec-insert multi-segs security 
                                            tx offload active: udp-cksum tcp-cksum multi-segs 
                                            rss avail:         ipv4-tcp ipv4-udp ipv4 ipv6-tcp-ex ipv6-udp-ex ipv6-tcp 
                                                               ipv6-udp ipv6-ex ipv6 
                                            rss active:        none
                                            tx burst function: ixgbe_xmit_pkts
                                            rx burst function: ixgbe_recv_pkts
                                       
    TenGigabitEthernetb/0/1            2    down  TenGigabitEthernetb/0/1
                                          Link speed: unknown
                                          Ethernet address 00:1b:21:80:69:79
                                          Intel 82599
                                            carrier down 
                                            flags: pmd maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum
                                            rx: queues 1 (max 128), desc 1024 (min 32 max 4096 align 8)
                                            tx: queues 1 (max 64), desc 1024 (min 32 max 4096 align 8)
                                            pci: device 8086:10fb subsystem 8086:7a11 address 0000:0b:00.01 numa 0
                                            module: unknown
                                            max rx packet len: 15872
                                            promiscuous: unicast off all-multicast off
                                            vlan offload: strip off filter off qinq off
                                            rx offload avail:  vlan-strip ipv4-cksum udp-cksum tcp-cksum tcp-lro 
                                                               macsec-strip vlan-filter vlan-extend jumbo-frame scatter 
                                                               security keep-crc 
                                            rx offload active: ipv4-cksum jumbo-frame scatter 
                                            tx offload avail:  vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum 
                                                               tcp-tso macsec-insert multi-segs security 
                                            tx offload active: udp-cksum tcp-cksum multi-segs 
                                            rss avail:         ipv4-tcp ipv4-udp ipv4 ipv6-tcp-ex ipv6-udp-ex ipv6-tcp 
                                                               ipv6-udp ipv6-ex ipv6 
                                            rss active:        none
                                            tx burst function: ixgbe_xmit_pkts
                                            rx burst function: ixgbe_recv_pkts
                                        
                                        local0                             0    down  local0
                                          Link speed: unknown
                                          local                                    
    

    For X722, vfio-pci works

    Address      Sock VID:PID     Link Speed   Driver          Product Name                    Vital Product Data
    0000:0b:00.0      8086:10fb   5.0 GT/s x32 vfio-pci                                        
    0000:0b:00.1      8086:10fb   5.0 GT/s x32 vfio-pci                                        
    0000:0d:00.0      8086:37d2   5.0 GT/s x32 vfio-pci        Example VPD                     RV: 0x d7
    0000:0d:00.1      8086:37d2   5.0 GT/s x32 vfio-pci        Example VPD                     RV: 0x d7
    0000:0d:00.2      8086:37d0   5.0 GT/s x32 vfio-pci        Example VPD                     RV: 0x d7
    0000:0d:00.3      8086:37d0   5.0 GT/s x32 vfio-pci        Example VPD                     RV: 0x d7
    0000:1d:00.0      8086:10fb   5.0 GT/s x32 vfio-pci                                        
    0000:1d:00.1      8086:10fb   5.0 GT/s x32 vfio-pci                                        
    0000:1f:00.0      15ad:07b0   5.0 GT/s x32 vmxnet3                                         
    
    TenGigabitEthernetd/0/0            1     up   TenGigabitEthernetd/0/0
      Link speed: 10 Gbps
      Ethernet address 00:25:90:ba:2a:70
      Intel X710/XL710 Family
        carrier up full duplex mtu 9206 
        flags: pmd maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum
        rx: queues 1 (max 192), desc 1024 (min 64 max 4096 align 32)
        tx: queues 1 (max 192), desc 1024 (min 64 max 4096 align 32)
        pci: device 8086:37d2 subsystem 15d9:37d2 address 0000:0d:00.00 numa 0
        module: unknown
        max rx packet len: 9728
        promiscuous: unicast off all-multicast off
        vlan offload: strip off filter off qinq off
        rx offload avail:  vlan-strip ipv4-cksum udp-cksum tcp-cksum qinq-strip 
                           outer-ipv4-cksum vlan-filter vlan-extend jumbo-frame 
                           scatter keep-crc 
        rx offload active: ipv4-cksum jumbo-frame scatter 
        tx offload avail:  vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum 
                           tcp-tso outer-ipv4-cksum qinq-insert vxlan-tnl-tso 
                           gre-tnl-tso ipip-tnl-tso geneve-tnl-tso multi-segs 
                           mbuf-fast-free 
        tx offload active: udp-cksum tcp-cksum multi-segs 
        rss avail:         ipv4-frag ipv4-tcp ipv4-udp ipv4-sctp ipv4-other ipv6-frag 
                           ipv6-tcp ipv6-udp ipv6-sctp ipv6-other l2-payload 
        rss active:        none
        tx burst function: i40e_xmit_pkts
        rx burst function: i40e_recv_pkts
    
        rx frames ok                                          40
        rx bytes ok                                         2400
    TenGigabitEthernetd/0/1            2    down  TenGigabitEthernetd/0/1
      Link speed: unknown
      Ethernet address 00:25:90:ba:2a:71
      Intel X710/XL710 Family
        carrier down 
        flags: pmd maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum
        rx: queues 1 (max 192), desc 1024 (min 64 max 4096 align 32)
        tx: queues 1 (max 192), desc 1024 (min 64 max 4096 align 32)
        pci: device 8086:37d2 subsystem 15d9:37d2 address 0000:0d:00.01 numa 0
        module: unknown
        max rx packet len: 9728
        promiscuous: unicast off all-multicast off
        vlan offload: strip off filter off qinq off
        rx offload avail:  vlan-strip ipv4-cksum udp-cksum tcp-cksum qinq-strip 
                           outer-ipv4-cksum vlan-filter vlan-extend jumbo-frame 
                           scatter keep-crc 
        rx offload active: ipv4-cksum jumbo-frame scatter 
        tx offload avail:  vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum 
                           tcp-tso outer-ipv4-cksum qinq-insert vxlan-tnl-tso 
                           gre-tnl-tso ipip-tnl-tso geneve-tnl-tso multi-segs 
                           mbuf-fast-free 
        tx offload active: udp-cksum tcp-cksum multi-segs 
        rss avail:         ipv4-frag ipv4-tcp ipv4-udp ipv4-sctp ipv4-other ipv6-frag 
                           ipv6-tcp ipv6-udp ipv6-sctp ipv6-other l2-payload 
        rss active:        none
        tx burst function: i40e_xmit_pkts
        rx burst function: i40e_recv_pkts
    
    TenGigabitEthernetd/0/2            3    down  TenGigabitEthernetd/0/2
      Link speed: unknown
      Ethernet address 00:25:90:ba:2a:72
      Intel X710/XL710 Family
        carrier down 
        flags: pmd maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum
        rx: queues 1 (max 192), desc 1024 (min 64 max 4096 align 32)
        tx: queues 1 (max 192), desc 1024 (min 64 max 4096 align 32)
        pci: device 8086:37d0 subsystem 15d9:37d0 address 0000:0d:00.02 numa 0
        module: unknown
        max rx packet len: 9728
        promiscuous: unicast off all-multicast off
        vlan offload: strip off filter off qinq off
        rx offload avail:  vlan-strip ipv4-cksum udp-cksum tcp-cksum qinq-strip 
                           outer-ipv4-cksum vlan-filter vlan-extend jumbo-frame 
                           scatter keep-crc 
        rx offload active: ipv4-cksum jumbo-frame scatter 
        tx offload avail:  vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum 
                           tcp-tso outer-ipv4-cksum qinq-insert vxlan-tnl-tso 
                           gre-tnl-tso ipip-tnl-tso geneve-tnl-tso multi-segs 
                           mbuf-fast-free 
        tx offload active: udp-cksum tcp-cksum multi-segs 
        rss avail:         ipv4-frag ipv4-tcp ipv4-udp ipv4-sctp ipv4-other ipv6-frag 
                           ipv6-tcp ipv6-udp ipv6-sctp ipv6-other l2-payload 
        rss active:        none
        tx burst function: i40e_xmit_pkts
        rx burst function: i40e_recv_pkts
    
    TenGigabitEthernetd/0/3            4     up   TenGigabitEthernetd/0/3
      Link speed: 10 Gbps
      Ethernet address 00:25:90:ba:2a:73
      Intel X710/XL710 Family
        carrier up full duplex mtu 9206 
        flags: pmd maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum
        rx: queues 1 (max 192), desc 1024 (min 64 max 4096 align 32)
        tx: queues 1 (max 192), desc 1024 (min 64 max 4096 align 32)
        pci: device 8086:37d0 subsystem 15d9:37d0 address 0000:0d:00.03 numa 0
        module: unknown
        max rx packet len: 9728
        promiscuous: unicast off all-multicast off
        vlan offload: strip off filter off qinq off
        rx offload avail:  vlan-strip ipv4-cksum udp-cksum tcp-cksum qinq-strip 
                           outer-ipv4-cksum vlan-filter vlan-extend jumbo-frame 
                           scatter keep-crc 
        rx offload active: ipv4-cksum jumbo-frame scatter 
        tx offload avail:  vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum 
                           tcp-tso outer-ipv4-cksum qinq-insert vxlan-tnl-tso 
                           gre-tnl-tso ipip-tnl-tso geneve-tnl-tso multi-segs 
                           mbuf-fast-free 
        tx offload active: udp-cksum tcp-cksum multi-segs 
        rss avail:         ipv4-frag ipv4-tcp ipv4-udp ipv4-sctp ipv4-other ipv6-frag 
                           ipv6-tcp ipv6-udp ipv6-sctp ipv6-other l2-payload 
        rss active:        none
        tx burst function: i40e_xmit_pkts
        rx burst function: i40e_recv_pkts
    
        rx frames ok                                           5
        rx bytes ok                                          300
    local0                             0    down  local0
      Link speed: unknown
      local
    

    Any clue how to get both chipsets bound simultaneously?

    Version: tnsr-v19.08.1-1
    Build timestamp: Mon Sep 9 12:26:44 2019 CDT



  • So, I did a package update and after reboot, it immediately Kernel panics. Since this is isn't in production, I just did a fresh install from the latest version ISO. Now all the interfaces populate as expected.


Log in to reply