Trying vrrp TNSR + Juniper MX80 = bad packet checksum
-
I have issue with VRRP on TNSR.
I have primary router Juniper MX80 and hardware server with TNSR.
Server with TNSR have Mellanox ConnectX4-Lx (MCX4111A-ACAT), so it is basically compatible with vrrp.Between routers I have two Juniper qfx3500. Ports that connected to routers configured as access, uplink between qfx3500 configured as trunk. Storm control is disabled on all ports between routers.
Juniper MX80 config
#show interfaces ge-1/1/6 unit 0 { description VRRP-test; family inet { address 192.168.222.3/24 { vrrp-group 240 { virtual-address 192.168.222.1; priority 250; accept-data; } } } }TNSR config
# show configuration running js "name": "LocalVM", "description": "VRRP-TEST", "enabled": true, "ipv4": { "address": { "ip": "192.168.222.2/24" }, "netgate-vrrp:vrrp": { "vrrp-instance": [ { "vrid": 240, "version": "netgate-interface:vrrp-v3", "preempt": { "enabled": false }, "priority": 100, "accept-mode": true, "advertise-interval-centi-sec": 100, "virtual-ipv4-addresses": { "virtual-ipv4-address": [ { "ipv4-address": "192.168.222.1" } ] } } ] } }The issue is that both routers became masters.
Juniper MX80 status# run show vrrp summary Interface State Group VR state VR Mode Type Address ge-1/1/6.0 up 240 master Active lcl 192.168.222.3 vip 192.168.222.1TNSR status
# show interface ip vrrp-virtual-router Interface: LocalVM IPv4 VRRP: VR: 240 State: master, Priority: 100, Flags: Accept_Mode Addresses: 192.168.222.1 Timers: Adv 100cs, Master down 360cs, Skew 60csAfter few days of digging I found the problem.
On both sides tcpdump shows that vrrp packets have "bad checksum".Juniper MX80 tcpdump
# run monitor traffic interface ge-1/1/6 no-resolve count 10 detail Address resolution is OFF. Listening on ge-1/1/6, capture size 1514 bytes 20:08:20.701260 In IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto: VRRP (112), length: 32) 192.168.222.2 > 224.0.0.18: VRRPv3-advertisement 12: vrid=240 prio=100 intvl=100(centisec) (bad vrrp cksum 4bc5!) addrs: 192.168.222.1 20:08:20.945022 Out IP (tos 0xc0, ttl 255, id 17, offset 0, flags [none], proto: VRRP (112), length: 32) 192.168.222.3 > 224.0.0.18: VRRPv3-advertisement 12: vrid=240 prio=250 intvl=100(centisec) addrs: 192.168.222.1 20:08:21.705253 In IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto: VRRP (112), length: 32) 192.168.222.2 > 224.0.0.18: VRRPv3-advertisement 12: vrid=240 prio=100 intvl=100(centisec) (bad vrrp cksum 4bc5!) addrs: 192.168.222.1 20:08:21.824882 Out IP (tos 0xc0, ttl 255, id 17, offset 0, flags [none], proto: VRRP (112), length: 32) 192.168.222.3 > 224.0.0.18: VRRPv3-advertisement 12: vrid=240 prio=250 intvl=100(centisec) addrs: 192.168.222.1 20:08:22.618815 Out IP (tos 0xc0, ttl 255, id 17, offset 0, flags [none], proto: VRRP (112), length: 32) 192.168.222.3 > 224.0.0.18: VRRPv3-advertisement 12: vrid=240 prio=250 intvl=100(centisec) addrs: 192.168.222.1 20:08:22.698251 In IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto: VRRP (112), length: 32) 192.168.222.2 > 224.0.0.18: VRRPv3-advertisement 12: vrid=240 prio=100 intvl=100(centisec) (bad vrrp cksum 4bc5!) addrs: 192.168.222.1 20:08:23.385979 Out IP (tos 0xc0, ttl 255, id 17, offset 0, flags [none], proto: VRRP (112), length: 32) 192.168.222.3 > 224.0.0.18: VRRPv3-advertisement 12: vrid=240 prio=250 intvl=100(centisec) addrs: 192.168.222.1TNSR tcpdump
$ sudo dp-exec tcpdump -XXX -vvv -nei capture "src host 192.168.222.2 or dst host 192.168.222.2 or src host 192.168.222.3 or dst host 192.168.222.3 or src host 192.168.222.1 or dst host 192.168.222.1" 19:12:01.166429 00:00:5e:00:01:f0 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 60: (tos 0xc0, ttl 255, id 17, offset 0, flags [none], proto VRRP (112), length 32) 192.168.222.3 > 224.0.0.18: vrrp 192.168.222.3 > 224.0.0.18: VRRPv3, Advertisement, vrid 240, prio 250, intvl 100cs, length 12, (bad vrrp cksum 34ff), addrs: 192.168.222.1 0x0000: 0100 5e00 0012 0000 5e00 01f0 0800 45c0 ..^.....^.....E. 0x0010: 0020 0011 0000 ff70 3bde c0a8 de03 e000 .......p;....... 0x0020: 0012 31f0 fa01 0064 34ff c0a8 de01 0000 ..1....d4....... 0x0030: 0000 0000 0000 0000 0000 0000 ............ 19:12:02.227469 00:00:5e:00:01:f0 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 60: (tos 0xc0, ttl 255, id 17, offset 0, flags [none], proto VRRP (112), length 32) 192.168.222.3 > 224.0.0.18: vrrp 192.168.222.3 > 224.0.0.18: VRRPv3, Advertisement, vrid 240, prio 250, intvl 100cs, length 12, (bad vrrp cksum 34ff), addrs: 192.168.222.1 0x0000: 0100 5e00 0012 0000 5e00 01f0 0800 45c0 ..^.....^.....E. 0x0010: 0020 0011 0000 ff70 3bde c0a8 de03 e000 .......p;....... 0x0020: 0012 31f0 fa01 0064 34ff c0a8 de01 0000 ..1....d4....... 0x0030: 0000 0000 0000 0000 0000 0000 ............ 19:12:02.920129 00:00:5e:00:01:f0 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 60: (tos 0xc0, ttl 255, id 17, offset 0, flags [none], proto VRRP (112), length 32) 192.168.222.3 > 224.0.0.18: vrrp 192.168.222.3 > 224.0.0.18: VRRPv3, Advertisement, vrid 240, prio 250, intvl 100cs, length 12, (bad vrrp cksum 34ff), addrs: 192.168.222.1 0x0000: 0100 5e00 0012 0000 5e00 01f0 0800 45c0 ..^.....^.....E. 0x0010: 0020 0011 0000 ff70 3bde c0a8 de03 e000 .......p;....... 0x0020: 0012 31f0 fa01 0064 34ff c0a8 de01 0000 ..1....d4....... 0x0030: 0000 0000 0000 0000 0000 0000 ............ 19:12:03.322410 1c:34:da:4c:02:1f > 01:00:5e:00:00:16, ethertype IPv4 (0x0800), length 54: (tos 0xc0, ttl 1, id 0, offset 0, flags [none], proto IGMP (2), length 40, options (RA)) 192.168.222.2 > 224.0.0.22: igmp v3 report, 1 group record(s) [gaddr 224.0.0.18 to_ex { }] 0x0000: 0100 5e00 0016 1c34 da4c 021f 0800 46c0 ..^....4.L....F. 0x0010: 0028 0000 0000 0102 a54e c0a8 de02 e000 .(.......N...... 0x0020: 0016 9404 0000 2200 f9eb 0000 0001 0400 ......"......... 0x0030: 0000 e000 0012 ...... 19:12:03.322414 00:00:5e:00:01:f0 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 46: (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto VRRP (112), length 32) 192.168.222.2 > 224.0.0.18: vrrp 192.168.222.2 > 224.0.0.18: VRRPv3, Advertisement, vrid 240, prio 100, intvl 100cs, length 12, addrs: 192.168.222.1 0x0000: 0100 5e00 0012 0000 5e00 01f0 0800 4500 ..^.....^.....E. 0x0010: 0020 0000 0000 ff70 3cb0 c0a8 de02 e000 .......p<....... 0x0020: 0012 31f0 6401 0064 4bc5 c0a8 de01 ..1.d..dK..... 19:12:03.322416 1c:34:da:4c:02:1f > Broadcast, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.222.1 (Broadcast) tell 192.168.222.1, length 28 0x0000: ffff ffff ffff 1c34 da4c 021f 0806 0001 .......4.L...... 0x0010: 0800 0604 0001 0000 5e00 01f0 c0a8 de01 ........^....... 0x0020: ffff ffff ffff c0a8 de01 .......... 19:12:03.799482 00:00:5e:00:01:f0 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 60: (tos 0xc0, ttl 255, id 17, offset 0, flags [none], proto VRRP (112), length 32) 192.168.222.3 > 224.0.0.18: vrrp 192.168.222.3 > 224.0.0.18: VRRPv3, Advertisement, vrid 240, prio 250, intvl 100cs, length 12, (bad vrrp cksum 34ff), addrs: 192.168.222.1 0x0000: 0100 5e00 0012 0000 5e00 01f0 0800 45c0 ..^.....^.....E. 0x0010: 0020 0011 0000 ff70 3bde c0a8 de03 e000 .......p;....... 0x0020: 0012 31f0 fa01 0064 34ff c0a8 de01 0000 ..1....d4....... 0x0030: 0000 0000 0000 0000 0000 0000 ............ 19:12:03.839908 00:00:5e:00:01:f0 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 46: (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto VRRP (112), length 32) 192.168.222.2 > 224.0.0.18: vrrp 192.168.222.2 > 224.0.0.18: VRRPv3, Advertisement, vrid 240, prio 100, intvl 100cs, length 12, addrs: 192.168.222.1 0x0000: 0100 5e00 0012 0000 5e00 01f0 0800 4500 ..^.....^.....E. 0x0010: 0020 0000 0000 ff70 3cb0 c0a8 de02 e000 .......p<....... 0x0020: 0012 31f0 6401 0064 4bc5 c0a8 de01 ..1.d..dK.....As you can see Juniper receive packets with bad checksum from TNSR and TNSR receive packets with bad checksum from Juniper.
According to VRRP RFC rfc5798 section 7.1 "MUST verify the VRRP checksum" and if it is bad the packet will dropped.
Does anybody have setup with working VRRP?
Also I can not find any information about which packet is using to run VRRP on TNSR.