Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Issue with more than one SR-IOV interface

    Scheduled Pinned Locked Moved
    TNSR
    2
    7
    2.4k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      SANEPLJ
      last edited by SANEPLJ

      I have an TNSR router VM on a KVM host lab and i have passed though 2x SR-IOV nics off my intel x520 of which the first is working fine on the second nic on the TNSR seems to transmit traffic i have confirmed this doing a packet capture,
      but Receive on the otherhand it does not. jumping into the host shell and into vpp cli doing a show interface.

      You can see below nic 6/0/0 there is something wrong if you compare to 5/0/0 which is working it has transmit and recive packets on both interfaces. maybe issue with punt redirect?

      Working interface in green Non working one in Red

      vpp# show interface
      Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count
      VirtualFunctionEthernet5/0/0 1 up 1500/0/0/0
      rx packets 15
      rx bytes 2376
      tx packets 32
      tx bytes 2656
      drops 12
      ip4 10
      tx-error 6
      VirtualFunctionEthernet5/0/0.50 7 up 1500/0/0/0
      rx packets 3
      rx bytes 612
      tx packets 11
      tx bytes 942
      ip4 3
      VirtualFunctionEthernet5/0/0.60 9 up 1500/0/0/0
      rx packets 5794147
      rx bytes 8012565552
      tx packets 13544268
      tx bytes 20238759100
      drops 11
      ip4 5794135
      VirtualFunctionEthernet6/0/0 2 up 1500/0/0/0
      tx packets 62
      tx bytes 4012
      tx-error 1
      VirtualFunctionEthernet6/0/0.10 11 up 1500/0/0/0
      tx packets 47
      tx bytes 2758
      local0 0 down 0/0/0/0 drops 6
      tap1 4 up 1500/0/0/0
      rx packets 24
      rx bytes 1944
      tx packets 3
      tx bytes 612
      ip6 8
      tap1.50 8 up 0/0/0/0
      rx packets 8
      rx bytes 672
      tx packets 3
      tx bytes 612
      ip6 8
      tap1.60 10 up 0/0/0/0
      rx packets 8
      rx bytes 652
      ip6 8
      tap2 5 up 1500/0/0/0
      rx packets 55
      rx bytes 3310
      drops 1
      ip6 12
      tap2.10 12 up 0/0/0/0
      rx packets 43
      rx bytes 2354
      ip6 10

      I can not see any difference between them in the punt redirect config either.

      vpp# show ip punt redirect
      rx VirtualFunctionEthernet5/0/0 via:
      path-list:[13] locks:1 flags:no-uRPF, uRPF-list: None
      path:[13] pl-index:13 ip4 weight=1 pref=0 dvr: oper-flags:resolved,
      tap1
      forwarding
      [@3]: ip4-dvr-tap1-dpo l3
      rx VirtualFunctionEthernet6/0/0 via:
      path-list:[15] locks:1 flags:no-uRPF, uRPF-list: None
      path:[15] pl-index:15 ip4 weight=1 pref=0 dvr: oper-flags:resolved,
      tap2
      forwarding
      [@3]: ip4-dvr-tap2-dpo l3
      rx VirtualFunctionEthernet5/0/0.50 via:
      path-list:[32] locks:1 flags:no-uRPF, uRPF-list: None
      path:[32] pl-index:32 ip4 weight=1 pref=0 dvr: oper-flags:resolved,
      tap1.50
      forwarding
      [@3]: ip4-dvr-tap1.50-dpo l3
      rx VirtualFunctionEthernet5/0/0.60 via:
      path-list:[34] locks:1 flags:no-uRPF, uRPF-list: None
      path:[34] pl-index:34 ip4 weight=1 pref=0 dvr: oper-flags:resolved,
      tap1.60
      forwarding
      [@3]: ip4-dvr-tap1.60-dpo l3
      rx VirtualFunctionEthernet6/0/0.10 via:
      path-list:[60] locks:1 flags:no-uRPF, uRPF-list: None
      path:[86] pl-index:60 ip4 weight=1 pref=0 dvr: oper-flags:resolved,
      tap2.10
      forwarding
      [@3]: ip4-dvr-tap2.10-dpo l3

      Any clues of the issue here or has anyone else had a similer issue?

      Below is the TNSR config file out and various others

      localhost tnsr# show configuration running cli
      nacm enable
      nacm read-default deny
      nacm write-default deny
      nacm exec-default deny
      nacm group admin
      member root
      member tnsr
      exit
      nacm rule-list admin-rules
      group admin
      rule permit-all
      module *
      access-operations "*"
      action permit
      exit
      exit

      dataplane cpu workers 6
      dataplane ethernet default-mtu 1500
      dataplane dpdk dev 0000:05:00.0 network tso on
      dataplane dpdk dev 0000:05:00.0 network num-rx-queues 2
      dataplane dpdk dev 0000:05:00.0 network num-tx-queues 2
      dataplane dpdk dev 0000:06:00.0 network tso on
      dataplane dpdk uio-driver igb_uio
      dataplane buffers buffers-per-numa 32768
      dataplane statseg heap-size 1G

      nat global-options nat44 enabled false

      route table tet
      id 1
      exit

      interface subif VirtualFunctionEthernet5/0/0 50
      exact-match
      outer-dot1q 50
      exit
      interface subif VirtualFunctionEthernet5/0/0 60
      exact-match
      outer-dot1q 60
      exit
      interface subif VirtualFunctionEthernet6/0/0 10
      exact-match
      outer-dot1q 10
      exit

      interface VirtualFunctionEthernet5/0/0
      enable
      exit
      interface VirtualFunctionEthernet5/0/0.50
      enable
      ip address 10.10.10.1/24
      exit
      interface VirtualFunctionEthernet5/0/0.60
      enable
      ip address 10.11.11.1/24
      exit
      interface VirtualFunctionEthernet6/0/0
      enable
      exit
      interface VirtualFunctionEthernet6/0/0.10
      enable
      ip address 172.81.81.231/24
      exit
      interface gigatbit
      disable
      exit

      nat ipfix logging domain 1
      nat ipfix logging src-port 4739
      nat nat64 map parameters
      security-check enable
      exit

      route dynamic manager
      exit

      route dynamic ospf6
      exit

      route dynamic bgp
      disable
      exit

      route dynamic ospf
      exit

      route dynamic rip
      exit

      dhcp4 server
      lease persist true
      lease lfc-interval 0
      interface socket raw
      exit

      unbound server
      enable ip4
      enable tcp
      enable udp
      enable harden glue
      enable hide identity
      port outgoing range 4096
      exit

      snmp host disable

      ssh dataplane enable

      localhost tnsr# show interface
      Interface: VirtualFunctionEthernet5/0/0
      Admin status: up
      Link up, link-speed 10 Gbps, full duplex
      Link MTU: 1500 bytes
      MAC address: 02:09:c0:09:83:4a
      IPv4 MTU: 0 bytes
      IPv4 Route Table: ipv4-VRF:0
      IPv6 MTU: 0 bytes
      IPv6 Route Table: ipv6-VRF:0
      IPv6 addresses:
      fe80::9:c0ff:fe09:834a/64
      VLAN tag rewrite: disable
      Rx-queues
      queue-id 0 : cpu-id 2
      queue-id 1 : cpu-id 3
      counters:
      received: 28251366540 bytes, 19338631 packets, 0 errors
      transmitted: 28251329946 bytes, 19338480 packets, 6 errors
      protocols: 190 IPv4, 0 IPv6
      214 drops, 0 punts, 0 rx miss, 0 rx no buffer

      Interface: VirtualFunctionEthernet5/0/0.50
      Admin status: up
      Link up, link-speed 10 Gbps, full duplex
      Link MTU: 1500 bytes
      IPv4 MTU: 0 bytes
      IPv4 Route Table: ipv4-VRF:0
      IPv4 addresses:
      10.10.10.1/24
      IPv6 MTU: 0 bytes
      IPv6 Route Table: ipv6-VRF:0
      IPv6 addresses:
      fe80::9:c0ff:fe09:834a/64
      VLAN tag rewrite: disable
      counters:
      received: 20238768684 bytes, 13544292 packets, 0 errors
      transmitted: 8012567788 bytes, 5794175 packets, 0 errors
      protocols: 13544281 IPv4, 0 IPv6
      11 drops, 0 punts, 0 rx miss, 0 rx no buffer

      Interface: VirtualFunctionEthernet5/0/0.60
      Admin status: up
      Link up, link-speed 10 Gbps, full duplex
      Link MTU: 1500 bytes
      IPv4 MTU: 0 bytes
      IPv4 Route Table: ipv4-VRF:0
      IPv4 addresses:
      10.11.11.1/24
      IPv6 MTU: 0 bytes
      IPv6 Route Table: ipv6-VRF:0
      IPv6 addresses:
      fe80::9:c0ff:fe09:834a/64
      VLAN tag rewrite: disable
      counters:
      received: 8012565552 bytes, 5794147 packets, 0 errors
      transmitted: 20238759280 bytes, 13544270 packets, 0 errors
      protocols: 5794135 IPv4, 0 IPv6
      11 drops, 0 punts, 0 rx miss, 0 rx no buffer

      Interface: VirtualFunctionEthernet6/0/0
      Admin status: up
      Link up, link-speed 10 Gbps, full duplex
      Link MTU: 1500 bytes
      MAC address: 02:09:c0:94:92:12
      IPv4 MTU: 0 bytes
      IPv4 Route Table: ipv4-VRF:0
      IPv6 MTU: 0 bytes
      IPv6 Route Table: ipv6-VRF:0
      IPv6 addresses:
      fe80::9:c0ff:fe94:9212/64
      VLAN tag rewrite: disable
      Rx-queues
      queue-id 0 : cpu-id 4
      counters:
      received: 0 bytes, 0 packets, 0 errors
      transmitted: 8268 bytes, 112 packets, 1 errors
      protocols: 0 IPv4, 0 IPv6
      0 drops, 0 punts, 0 rx miss, 0 rx no buffer

      Interface: VirtualFunctionEthernet6/0/0.10
      Admin status: up
      Link up, link-speed 10 Gbps, full duplex
      Link MTU: 1500 bytes
      IPv4 MTU: 0 bytes
      IPv4 Route Table: ipv4-VRF:0
      IPv4 addresses:
      172.81.81.231/24
      IPv6 MTU: 0 bytes
      IPv6 Route Table: ipv6-VRF:0
      IPv6 addresses:
      fe80::9:c0ff:fe94:9212/64
      VLAN tag rewrite: disable
      counters:
      received: 0 bytes, 0 packets, 0 errors
      transmitted: 4928 bytes, 72 packets, 0 errors
      protocols: 0 IPv4, 0 IPv6
      0 drops, 0 punts, 0 rx miss, 0 rx no buffer

      1 Reply Last reply Reply Quote 0
      • S
        SANEPLJ
        last edited by SANEPLJ

        Diagram Of Lab

        af9d341f-05b3-4849-a7a9-3a90155b9de3-image.png

        DerelictD 1 Reply Last reply Reply Quote 0
        • DerelictD
          Derelict LAYER 8 Netgate @SANEPLJ
          last edited by

          @saneplj

          The first thing I would do is remove these and see if anything changes:

          dataplane cpu workers 6
dataplane dpdk dev 0000:05:00.0 network tso on
dataplane dpdk dev 0000:05:00.0 network num-rx-queues 2
dataplane dpdk dev 0000:05:00.0 network num-tx-queues 2
dataplane dpdk dev 0000:06:00.0 network tso on
dataplane statseg heap-size 1G

          And why are you using outer-dot1q instead of just dot1q to set the VLAN tags?

          Chattanooga, Tennessee, USA
          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
          Do Not Chat For Help! NO_WAN_EGRESS(TM)

          S 1 Reply Last reply Reply Quote 0
          • S
            SANEPLJ @Derelict
            last edited by

            @derelict Hi and thanks for replying, In relation to you point outer-dot1q, that is a good point, in the TNSR cli I do actually enter dot1q not outer-dot1q but it always seems to apply outer-dot1q even looking at the help output look at below output.

            17e38223-2100-400f-abd3-c372a60b7f3e-image.png

            018443a1-4a56-48d9-b7ef-7459fb46d9d4-image.png

            I have removed all the config you have suggested even though the issue was present before i added them, but sadly nothing has changed still having the same issue.

            As a test i will try removing the subinterface config within TNSR cli and will try configuring directly within VPP CLI. I will update you once done.

            Mean while if you have any other surgestions please let me know.

            Thanks again.

            DerelictD 1 Reply Last reply Reply Quote 0
            • DerelictD
              Derelict LAYER 8 Netgate @SANEPLJ
              last edited by

              @saneplj I would also be working with 21.03.1-1 that was released last week even though there is not anything I can think of that addresses things like this.

              Chattanooga, Tennessee, USA
              A comprehensive network diagram is worth 10,000 words and 15 conference calls.
              DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
              Do Not Chat For Help! NO_WAN_EGRESS(TM)

              S 2 Replies Last reply Reply Quote 0
              • S
                SANEPLJ @Derelict
                last edited by SANEPLJ

                @derelict Hi, I Moved to the latest version 21.03.1-1 but same result. Tried configuring directly within the vpp cli still have the same issues on reciving packets on secound SR-IOV interface.

                I disabled SR-IOV and passed though both physical nic ports same config using same driver igb_uio and they both function as expected, both receiving and transmiting data.

                I can confirm passing multiple SR-IOV interfaces though any other VM's on platforms Linux, Windows, Cisco CSRv and ASAv all interfaces work just fine.

                This seems to be a specific issue with TNSR/VPP with multiple SR-IOV interfaces off an Intel x520 as far as i can determine.

                1 Reply Last reply Reply Quote 0
                • S
                  SANEPLJ @Derelict
                  last edited by SANEPLJ

                  @derelict Thought i would give this another shot on the latest code v21.01-588~tnsr_v21.07_1~g31b3e823e, but to no avail, but i did find somting interesting using the show hardware-interface command in vppctl it is seeing the non working interface VirtualFunctionEthernet6/0/0 MTU at 9206 and the admin up flag is not present and this error [rte_eth_dev_start[port:1, errno:-22]: Unknown error -22]. any Ideas how to look/debug further into this or do yo know what this error is?

                  I have also included working interface VirtualFunctionEthernet5/0/0 for comparison

                  vpp# sh hardware-interfaces
                  Name Idx Link Hardware
                  VirtualFunctionEthernet5/0/0 1 up VirtualFunctionEthernet5/0/0
                  Link speed: 10 Gbps
                  RX Queues:
                  queue thread mode
                  0 main (0) polling
                  Ethernet address 02:09:c0:99:4a:be
                  Intel 82599 VF
                  carrier up full duplex mtu 1500
                  flags: admin-up pmd maybe-multiseg subif tx-offload intel-phdr-cksum rx-ip4-cksum
                  rx: queues 1 (max 2), desc 1024 (min 32 max 4096 align 8)
                  tx: queues 1 (max 2), desc 1024 (min 32 max 4096 align 8)
                  pci: device 8086:10ed subsystem 8086:000c address 0000:05:00.00 numa 0
                  max rx packet len: 9728
                  promiscuous: unicast off all-multicast on
                  vlan offload: strip off filter off qinq off
                  rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum vlan-filter
                  jumbo-frame scatter keep-crc rss-hash
                  rx offload active: ipv4-cksum jumbo-frame scatter
                  tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum
                  tcp-tso multi-segs
                  tx offload active: udp-cksum tcp-cksum multi-segs
                  rss avail: ipv4-tcp ipv4-udp ipv4 ipv6-tcp-ex ipv6-udp-ex ipv6-tcp
                  ipv6-udp ipv6-ex ipv6
                  rss active: ipv4-tcp ipv6-tcp-ex ipv6-tcp ipv6-udp ipv6-ex
                  tx burst function: ixgbe_xmit_pkts
                  rx burst function: ixgbe_recv_scattered_pkts_vec

                  tx frames ok                                          86
tx bytes ok                                         7258
rx frames ok                                        1325
rx bytes ok                                       191694
extended stats:
  rx_good_packets                                   1325
  tx_good_packets                                     86
  rx_good_bytes                                   191694
  tx_good_bytes                                     7258
  rx_multicast_packets                              1550

                  VirtualFunctionEthernet6/0/0 2 up VirtualFunctionEthernet6/0/0
                  Link speed: 10 Gbps
                  RX Queues:
                  queue thread mode
                  0 main (0) polling
                  Ethernet address 02:09:c0:c1:84:fc
                  Intel 82599 VF
                  carrier up full duplex mtu 9206
                  flags: pmd maybe-multiseg subif tx-offload intel-phdr-cksum rx-ip4-cksum
                  rx: queues 1 (max 2), desc 1024 (min 32 max 4096 align 8)
                  tx: queues 1 (max 2), desc 1024 (min 32 max 4096 align 8)
                  pci: device 8086:10ed subsystem 8086:000c address 0000:06:00.00 numa 0
                  max rx packet len: 9728
                  promiscuous: unicast off all-multicast off
                  vlan offload: strip off filter on qinq off
                  rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum vlan-filter
                  jumbo-frame scatter keep-crc rss-hash
                  rx offload active: ipv4-cksum jumbo-frame scatter
                  tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum
                  tcp-tso multi-segs
                  tx offload active: udp-cksum tcp-cksum multi-segs
                  rss avail: ipv4-tcp ipv4-udp ipv4 ipv6-tcp-ex ipv6-udp-ex ipv6-tcp
                  ipv6-udp ipv6-ex ipv6
                  rss active: ipv4-tcp ipv6-tcp-ex ipv6-tcp ipv6-udp ipv6-ex
                  tx burst function: ixgbe_xmit_pkts
                  rx burst function: ixgbe_recv_pkts

                  rx frames ok                                  4294966922
rx bytes ok                                  68719442425  Errors:
rte_eth_dev_start[port:1, errno:-22: Unknown error -22

                  Thanks.

                  1 Reply Last reply Reply Quote 0
                  • First post
                    Last post