Issue with more than one SR-IOV interface

SANEPLJ · May 8, 2021, 2:10 PM

I have an TNSR router VM on a KVM host lab and i have passed though 2x SR-IOV nics off my intel x520 of which the first is working fine on the second nic on the TNSR seems to transmit traffic i have confirmed this doing a packet capture,
but Receive on the otherhand it does not. jumping into the host shell and into vpp cli doing a show interface.

You can see below nic 6/0/0 there is something wrong if you compare to 5/0/0 which is working it has transmit and recive packets on both interfaces. maybe issue with punt redirect?

Working interface in green Non working one in Red

vpp# show interface
Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count
VirtualFunctionEthernet5/0/0 1 up 1500/0/0/0
rx packets 15
rx bytes 2376
tx packets 32
tx bytes 2656
drops 12
ip4 10
tx-error 6
VirtualFunctionEthernet5/0/0.50 7 up 1500/0/0/0
rx packets 3
rx bytes 612
tx packets 11
tx bytes 942
ip4 3
VirtualFunctionEthernet5/0/0.60 9 up 1500/0/0/0
rx packets 5794147
rx bytes 8012565552
tx packets 13544268
tx bytes 20238759100
drops 11
ip4 5794135
VirtualFunctionEthernet6/0/0 2 up 1500/0/0/0
tx packets 62
tx bytes 4012
tx-error 1
VirtualFunctionEthernet6/0/0.10 11 up 1500/0/0/0
tx packets 47
tx bytes 2758
local0 0 down 0/0/0/0 drops 6
tap1 4 up 1500/0/0/0
rx packets 24
rx bytes 1944
tx packets 3
tx bytes 612
ip6 8
tap1.50 8 up 0/0/0/0
rx packets 8
rx bytes 672
tx packets 3
tx bytes 612
ip6 8
tap1.60 10 up 0/0/0/0
rx packets 8
rx bytes 652
ip6 8
tap2 5 up 1500/0/0/0
rx packets 55
rx bytes 3310
drops 1
ip6 12
tap2.10 12 up 0/0/0/0
rx packets 43
rx bytes 2354
ip6 10

I can not see any difference between them in the punt redirect config either.

vpp# show ip punt redirect
rx VirtualFunctionEthernet5/0/0 via:
path-list:[13] locks:1 flags:no-uRPF, uRPF-list: None
path:[13] pl-index:13 ip4 weight=1 pref=0 dvr: oper-flags:resolved,
tap1
forwarding
[@3]: ip4-dvr-tap1-dpo l3
rx VirtualFunctionEthernet6/0/0 via:
path-list:[15] locks:1 flags:no-uRPF, uRPF-list: None
path:[15] pl-index:15 ip4 weight=1 pref=0 dvr: oper-flags:resolved,
tap2
forwarding
[@3]: ip4-dvr-tap2-dpo l3
rx VirtualFunctionEthernet5/0/0.50 via:
path-list:[32] locks:1 flags:no-uRPF, uRPF-list: None
path:[32] pl-index:32 ip4 weight=1 pref=0 dvr: oper-flags:resolved,
tap1.50
forwarding
[@3]: ip4-dvr-tap1.50-dpo l3
rx VirtualFunctionEthernet5/0/0.60 via:
path-list:[34] locks:1 flags:no-uRPF, uRPF-list: None
path:[34] pl-index:34 ip4 weight=1 pref=0 dvr: oper-flags:resolved,
tap1.60
forwarding
[@3]: ip4-dvr-tap1.60-dpo l3
rx VirtualFunctionEthernet6/0/0.10 via:
path-list:[60] locks:1 flags:no-uRPF, uRPF-list: None
path:[86] pl-index:60 ip4 weight=1 pref=0 dvr: oper-flags:resolved,
tap2.10
forwarding
[@3]: ip4-dvr-tap2.10-dpo l3

Any clues of the issue here or has anyone else had a similer issue?

Below is the TNSR config file out and various others

localhost tnsr# show configuration running cli
nacm enable
nacm read-default deny
nacm write-default deny
nacm exec-default deny
nacm group admin
member root
member tnsr
exit
nacm rule-list admin-rules
group admin
rule permit-all
module *
access-operations "*"
action permit
exit
exit

dataplane cpu workers 6
dataplane ethernet default-mtu 1500
dataplane dpdk dev 0000:05:00.0 network tso on
dataplane dpdk dev 0000:05:00.0 network num-rx-queues 2
dataplane dpdk dev 0000:05:00.0 network num-tx-queues 2
dataplane dpdk dev 0000:06:00.0 network tso on
dataplane dpdk uio-driver igb_uio
dataplane buffers buffers-per-numa 32768
dataplane statseg heap-size 1G

nat global-options nat44 enabled false

route table tet
id 1
exit

interface subif VirtualFunctionEthernet5/0/0 50
exact-match
outer-dot1q 50
exit
interface subif VirtualFunctionEthernet5/0/0 60
exact-match
outer-dot1q 60
exit
interface subif VirtualFunctionEthernet6/0/0 10
exact-match
outer-dot1q 10
exit

interface VirtualFunctionEthernet5/0/0
enable
exit
interface VirtualFunctionEthernet5/0/0.50
enable
ip address 10.10.10.1/24
exit
interface VirtualFunctionEthernet5/0/0.60
enable
ip address 10.11.11.1/24
exit
interface VirtualFunctionEthernet6/0/0
enable
exit
interface VirtualFunctionEthernet6/0/0.10
enable
ip address 172.81.81.231/24
exit
interface gigatbit
disable
exit

nat ipfix logging domain 1
nat ipfix logging src-port 4739
nat nat64 map parameters
security-check enable
exit

route dynamic manager
exit

route dynamic ospf6
exit

route dynamic bgp
disable
exit

route dynamic ospf
exit

route dynamic rip
exit

dhcp4 server
lease persist true
lease lfc-interval 0
interface socket raw
exit

unbound server
enable ip4
enable tcp
enable udp
enable harden glue
enable hide identity
port outgoing range 4096
exit

snmp host disable

ssh dataplane enable

localhost tnsr# show interface
Interface: VirtualFunctionEthernet5/0/0
Admin status: up
Link up, link-speed 10 Gbps, full duplex
Link MTU: 1500 bytes
MAC address: 02:09:c0:09:83:4a
IPv4 MTU: 0 bytes
IPv4 Route Table: ipv4-VRF:0
IPv6 MTU: 0 bytes
IPv6 Route Table: ipv6-VRF:0
IPv6 addresses:
fe80::9:c0ff:fe09:834a/64
VLAN tag rewrite: disable
Rx-queues
queue-id 0 : cpu-id 2
queue-id 1 : cpu-id 3
counters:
received: 28251366540 bytes, 19338631 packets, 0 errors
transmitted: 28251329946 bytes, 19338480 packets, 6 errors
protocols: 190 IPv4, 0 IPv6
214 drops, 0 punts, 0 rx miss, 0 rx no buffer

Interface: VirtualFunctionEthernet5/0/0.50
Admin status: up
Link up, link-speed 10 Gbps, full duplex
Link MTU: 1500 bytes
IPv4 MTU: 0 bytes
IPv4 Route Table: ipv4-VRF:0
IPv4 addresses:
10.10.10.1/24
IPv6 MTU: 0 bytes
IPv6 Route Table: ipv6-VRF:0
IPv6 addresses:
fe80::9:c0ff:fe09:834a/64
VLAN tag rewrite: disable
counters:
received: 20238768684 bytes, 13544292 packets, 0 errors
transmitted: 8012567788 bytes, 5794175 packets, 0 errors
protocols: 13544281 IPv4, 0 IPv6
11 drops, 0 punts, 0 rx miss, 0 rx no buffer

Interface: VirtualFunctionEthernet5/0/0.60
Admin status: up
Link up, link-speed 10 Gbps, full duplex
Link MTU: 1500 bytes
IPv4 MTU: 0 bytes
IPv4 Route Table: ipv4-VRF:0
IPv4 addresses:
10.11.11.1/24
IPv6 MTU: 0 bytes
IPv6 Route Table: ipv6-VRF:0
IPv6 addresses:
fe80::9:c0ff:fe09:834a/64
VLAN tag rewrite: disable
counters:
received: 8012565552 bytes, 5794147 packets, 0 errors
transmitted: 20238759280 bytes, 13544270 packets, 0 errors
protocols: 5794135 IPv4, 0 IPv6
11 drops, 0 punts, 0 rx miss, 0 rx no buffer

Interface: VirtualFunctionEthernet6/0/0
Admin status: up
Link up, link-speed 10 Gbps, full duplex
Link MTU: 1500 bytes
MAC address: 02:09:c0:94:92:12
IPv4 MTU: 0 bytes
IPv4 Route Table: ipv4-VRF:0
IPv6 MTU: 0 bytes
IPv6 Route Table: ipv6-VRF:0
IPv6 addresses:
fe80::9:c0ff:fe94:9212/64
VLAN tag rewrite: disable
Rx-queues
queue-id 0 : cpu-id 4
counters:
received: 0 bytes, 0 packets, 0 errors
transmitted: 8268 bytes, 112 packets, 1 errors
protocols: 0 IPv4, 0 IPv6
0 drops, 0 punts, 0 rx miss, 0 rx no buffer

Interface: VirtualFunctionEthernet6/0/0.10
Admin status: up
Link up, link-speed 10 Gbps, full duplex
Link MTU: 1500 bytes
IPv4 MTU: 0 bytes
IPv4 Route Table: ipv4-VRF:0
IPv4 addresses:
172.81.81.231/24
IPv6 MTU: 0 bytes
IPv6 Route Table: ipv6-VRF:0
IPv6 addresses:
fe80::9:c0ff:fe94:9212/64
VLAN tag rewrite: disable
counters:
received: 0 bytes, 0 packets, 0 errors
transmitted: 4928 bytes, 72 packets, 0 errors
protocols: 0 IPv4, 0 IPv6
0 drops, 0 punts, 0 rx miss, 0 rx no buffer

SANEPLJ · May 8, 2021, 7:46 PM

Diagram Of Lab

Derelict · May 9, 2021, 2:33 PM

@saneplj

The first thing I would do is remove these and see if anything changes:

dataplane cpu workers 6
dataplane dpdk dev 0000:05:00.0 network tso on
dataplane dpdk dev 0000:05:00.0 network num-rx-queues 2
dataplane dpdk dev 0000:05:00.0 network num-tx-queues 2
dataplane dpdk dev 0000:06:00.0 network tso on
dataplane statseg heap-size 1G

And why are you using outer-dot1q instead of just dot1q to set the VLAN tags?

SANEPLJ · May 9, 2021, 6:51 PM

@derelict Hi and thanks for replying, In relation to you point outer-dot1q, that is a good point, in the TNSR cli I do actually enter dot1q not outer-dot1q but it always seems to apply outer-dot1q even looking at the help output look at below output.

I have removed all the config you have suggested even though the issue was present before i added them, but sadly nothing has changed still having the same issue.

As a test i will try removing the subinterface config within TNSR cli and will try configuring directly within VPP CLI. I will update you once done.

Mean while if you have any other surgestions please let me know.

Thanks again.

Derelict · May 12, 2021, 9:32 PM

@saneplj I would also be working with 21.03.1-1 that was released last week even though there is not anything I can think of that addresses things like this.

SANEPLJ · May 12, 2021, 9:32 PM

@derelict Hi, I Moved to the latest version 21.03.1-1 but same result. Tried configuring directly within the vpp cli still have the same issues on reciving packets on secound SR-IOV interface.

I disabled SR-IOV and passed though both physical nic ports same config using same driver igb_uio and they both function as expected, both receiving and transmiting data.

I can confirm passing multiple SR-IOV interfaces though any other VM's on platforms Linux, Windows, Cisco CSRv and ASAv all interfaces work just fine.

This seems to be a specific issue with TNSR/VPP with multiple SR-IOV interfaces off an Intel x520 as far as i can determine.

SANEPLJ · Oct 28, 2021, 9:47 PM

@derelict Thought i would give this another shot on the latest code v21.01-588~tnsr_v21.07_1~g31b3e823e, but to no avail, but i did find somting interesting using the show hardware-interface command in vppctl it is seeing the non working interface VirtualFunctionEthernet6/0/0 MTU at 9206 and the admin up flag is not present and this error [rte_eth_dev_start[port:1, errno:-22]: Unknown error -22]. any Ideas how to look/debug further into this or do yo know what this error is?

I have also included working interface VirtualFunctionEthernet5/0/0 for comparison

vpp# sh hardware-interfaces
Name Idx Link Hardware
VirtualFunctionEthernet5/0/0 1 up VirtualFunctionEthernet5/0/0
Link speed: 10 Gbps
RX Queues:
queue thread mode
0 main (0) polling
Ethernet address 02:09:c0:99:4a:be
Intel 82599 VF
carrier up full duplex mtu 1500
flags: admin-up pmd maybe-multiseg subif tx-offload intel-phdr-cksum rx-ip4-cksum
rx: queues 1 (max 2), desc 1024 (min 32 max 4096 align 8)
tx: queues 1 (max 2), desc 1024 (min 32 max 4096 align 8)
pci: device 8086:10ed subsystem 8086:000c address 0000:05:00.00 numa 0
max rx packet len: 9728
promiscuous: unicast off all-multicast on
vlan offload: strip off filter off qinq off
rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum vlan-filter
jumbo-frame scatter keep-crc rss-hash
rx offload active: ipv4-cksum jumbo-frame scatter
tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum
tcp-tso multi-segs
tx offload active: udp-cksum tcp-cksum multi-segs
rss avail: ipv4-tcp ipv4-udp ipv4 ipv6-tcp-ex ipv6-udp-ex ipv6-tcp
ipv6-udp ipv6-ex ipv6
rss active: ipv4-tcp ipv6-tcp-ex ipv6-tcp ipv6-udp ipv6-ex
tx burst function: ixgbe_xmit_pkts
rx burst function: ixgbe_recv_scattered_pkts_vec

tx frames ok                                          86
tx bytes ok                                         7258
rx frames ok                                        1325
rx bytes ok                                       191694
extended stats:
  rx_good_packets                                   1325
  tx_good_packets                                     86
  rx_good_bytes                                   191694
  tx_good_bytes                                     7258
  rx_multicast_packets                              1550

VirtualFunctionEthernet6/0/0 2 up VirtualFunctionEthernet6/0/0
Link speed: 10 Gbps
RX Queues:
queue thread mode
0 main (0) polling
Ethernet address 02:09:c0:c1:84:fc
Intel 82599 VF
carrier up full duplex mtu 9206
flags: pmd maybe-multiseg subif tx-offload intel-phdr-cksum rx-ip4-cksum
rx: queues 1 (max 2), desc 1024 (min 32 max 4096 align 8)
tx: queues 1 (max 2), desc 1024 (min 32 max 4096 align 8)
pci: device 8086:10ed subsystem 8086:000c address 0000:06:00.00 numa 0
max rx packet len: 9728
promiscuous: unicast off all-multicast off
vlan offload: strip off filter on qinq off
rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum vlan-filter
jumbo-frame scatter keep-crc rss-hash
rx offload active: ipv4-cksum jumbo-frame scatter
tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum
tcp-tso multi-segs
tx offload active: udp-cksum tcp-cksum multi-segs
rss avail: ipv4-tcp ipv4-udp ipv4 ipv6-tcp-ex ipv6-udp-ex ipv6-tcp
ipv6-udp ipv6-ex ipv6
rss active: ipv4-tcp ipv6-tcp-ex ipv6-tcp ipv6-udp ipv6-ex
tx burst function: ixgbe_xmit_pkts
rx burst function: ixgbe_recv_pkts

rx frames ok                                  4294966922
rx bytes ok                                  68719442425  Errors:
rte_eth_dev_start[port:1, errno:-22: Unknown error -22

Thanks.