I'm looking for help with determining how to better troubleshoot MAP on TNSR and see where traffic is dropping. It seems to be reaching the proper interface from both the v4 and v6 sides.

Also, what needs to be done to get MAP to properly pick up and translate traffic? It seems broken ...

Troubleshooting

High Level Checks done

• Routing Checked to/from the v4 and v6 sides, respectively for MAP v4 and v6 Prefixes - it's working
  • this is verified with tcpdump v4 inbound flows, and v6 reachability (i.e. ping) to our DMR (Default Mapping Rule aka BR IPv6 interface address), and show route table default ADDR checks for routes on upstream / downstream routers)
• toggling MAP enable / disable on the interface in case MAP didn't apply properly
• verified proper bits for the MAP-T config (e.g. EA bits, PSID offset, PSID length, Prefix, etc)
  • Related: also verified some of the addresses seen on the wire with https://github.com/ejordangottlieb/pyswmap code and manual calculations

With that said ... is there better debugging logs or counters for errors or "non-security checked" traffic hitting the MAP flows in VPP? Troubleshooting seems extremely limited from what the docs say, especially if v6 flows aren't being shown in tcpdump (which seems to be the case, see below troubleshooting)

NDPs showing up for IPv6 Destination addresses from CPE

What's interesting is we see NDPs going OUT from the BR DMR.

This is unexpected.

These addresses are inbound from the CPE and should be ingested by MAP, not sent anywhere else. We verified the SRC/DST v6 addresses on the WAN side via wireshark as an additional sanity check to ensure they are properly formatted for the given IPv6 Prefix, Destination DMR, Destination v4 host, PSID, etc.

An example tcpdump snippet on the BR interface showing unexpected NDPs and no other ICMPv6, ICMP, HTTP traffic (test traffic from the CPEs bound for the v4 Internet was ICMP and/or HTTP):

02:23:51.094333 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::20c:29ff:feca:40bb > ff02::1:ff00:0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:64:0:1:101:100:0
	  source link-address option (1), length 8 (1): 00:0c:29:ca:40:bb
	    0x0000:  000c 29ca 40bb
02:23:52.098581 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::20c:29ff:feca:40bb > ff02::1:ff00:0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:64:0:1:101:100:0
	  source link-address option (1), length 8 (1): 00:0c:29:ca:40:bb
	    0x0000:  000c 29ca 40bb
02:23:52.975038 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::20c:29ff:feca:40bb > ff02::1:ff00:0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:64:0:68:1223:f00:0
	  source link-address option (1), length 8 (1): 00:0c:29:ca:40:bb
	    0x0000:  000c 29ca 40bb
02:23:53.104128 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::20c:29ff:feca:40bb > ff02::1:ff00:0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:64:0:1:101:100:0
	  source link-address option (1), length 8 (1): 00:0c:29:ca:40:bb
	    0x0000:  000c 29ca 40bb
02:23:53.977166 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::20c:29ff:feca:40bb > ff02::1:ff00:0: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8:64:0:68:1223:f00:0
	  source link-address option (1), length 8 (1): 00:0c:29:ca:40:bb

Specifically, what's interesting:

• tcpdumps don't show the incoming packets destined to the DMR interface from CEs
  • this could be ok depending on how vpp input graphs are laid out, but we would need confirmation from devs
• To the above point, DMR-destined traffic from CEs is getting to the BR somehow and being processed.
  • Those NDP v6 addresses seen above in the tcpdump (2001:db8:64:0:1:101:100:0, 2001:db8:64:0:68:1223:f00:0) are the proper translated destination packets for cloudflare DNS (1.1.1.1) and a cloudflare web site (104.18.35.15)

It almost seems that the v6 inbound packets are getting processed for routing on the local subnet (thus the NDP), but we would expect MAP to pick this up, translate it to v4, and dump it out toward the next-hop with the proper v4 public address (A.B.C.128 in this case since PSIDs are 0 and 1 with a sharing ratio of 8 on this MAP domain)

Additional Troubleshooting

To further investigate, we dumped the VPP map stats (in vppctrl) and they are:

• Very one-sided (Encapsulated packets > 0, Decapsulated = 0)
  • But also imbalanced. Organic Internet traffic toward v4 addresses in the MAP domain + other scapy crafted packets destined for our CPE number in the hundreds and thousands of packets per second, but watching the show map stats output, we'll see 10-20 pps on the Encapsulated packets counter. It's inconclusive if these are the traffic flows from the Internet, but we strongly suspect no due to the counts being off.
• tcpdump shows inbound flows (v4 internet to v6 CPE) which should get translated along with our outbound flows (v6 translated CPE traffic to v4 Internet), so we'd expect processing counts for both

This implies MAP is not being applied / activated properly. We even toggled MAP on the interface (map disable; exit followed by map enable; exit) to "repush" config, but that doesn't change any of the above observed behavior; traffic still seems broken through MAP.

Are there other counters / logs / taps that can be created to specifically troubleshoot MAP flows?

Reference Output:

vpp# show map stats
MAP domains structure: 64
MAP domains: 1 (64 bytes)
MAP rules: 0 (0 bytes)
Total: 64 bytes)
MAP pre-resolve: IP6 next-hop: un-set, IP4 next-hop: un-set
MAP traffic-class: copy
MAP IPv6 inbound security check: enabled, fragmented packet security check: disabled
ICMP-relay IPv4 source address: A.B.C.D
ICMP6 unreachables sent for unmatched packets: enabled
Inner fragmentation: disabled
Fragment packets regardless of DF flag: disabled
Encapsulated packets: 4035 bytes: 180880
Decapsulated packets: 0 bytes: 0
ICMP relayed packets: 0

Relevant TNSR MAP config

nat nat64 map parameters
    icmp source-address A.B.C.D
    icmp6 unreachable-msgs enable
    security-check enable
exit
nat nat64 map br
    ipv4 prefix A.B.C.128/25
    ipv6 prefix 2001:db8:ce1::/48
    ipv6 source 2001:db8:64::/64
    embedded-address bit-length 10
    port-set offset 3
    port-set length 3
    mtu 1500
exit

interface br
    enable
    ip address 172.31.252.29/31
    ipv6 address 2001:db8:64::1/64
    ipv6 router-advertisements
        send-advertisements false
        max-rtr-adv-interval 600
        managed-flag false
        other-config-flag false
    exit
    map enable
    map translate
exit