Switches are not learning CARP HA MAC
-
First off, I've set up the canonical CARP HA architecture many, many times using Cisco or Foundry/Brocade switches but this time I'm using Penguin Arcticas (4804iq 1G and 4806xp 10G) with Cumulus Linux on the management side. The switches are running purely in L2 mode.
The symptom that I first encountered was that traffic was appearing on host ports that it shouldn't be, which was tracked down by Cumulus support to being caused by the switches not learning the mac address of the CARP VIP and thus traffic bound for the gateway being flooded out all ports on the VLAN.
I'm running some very large VM-hosting servers on this network, and 8 dedicated cores for handling network interrupts are needed to avoid performance issues with the load under normal circumstances, dedicating 4x that many to handle all this extra traffic is not an option.
Any possibility this is an IGMP issue? Any suggestions on what to investigate?
-
Switch problem. The switch should learn the CARP MAC from the CARP advertisements (which are multicast to 224.0.0.18, same as VRRP).
If the switches are somehow blocking multicast they might not be learning the CARP MAC.
https://doc.pfsense.org/index.php/CARP_Configuration_Troubleshooting#Switch.2FLayer_2_Issues
-
Already been there, turned off igmp snooping to no effect (the switches have no L3 configuration on the default VRF so doubt that would solve it).
We were able to force the mac into the switches with the use of arping and eliminate the packet flooding, but getting this to work right in the event of a failover might be tricky.
-
Use another (not broken) switch.
-
OK, we'll get right on throwing out $75,000 worth of switches. Thanks…
-
Well then they need to fix it. Broken is broken.
-
We've got a ticket open with the switch-OS vendor, had already tracked it down to the mac not getting learned, trying to figure out WHY its not being learned.
-
Switch problem. The switch should learn the CARP MAC from the CARP advertisements (which are multicast to 224.0.0.18, same as VRRP).
If the switches are somehow blocking multicast they might not be learning the CARP MAC.
https://doc.pfsense.org/index.php/CARP_Configuration_Troubleshooting#Switch.2FLayer_2_Issues
The actual CARP protocol is kind of underdocumented, lacking an RFC-like packet flow definition, but I was lead to understand that the mac would be learned not from the actual advertisements but from a gratuitous arp sent from the master after the election just as it was done in VRRP. And in our case the election does seem to be carried out correctly.
-
The only time (that I know of) the CARP MAC address itself goes on the wire in the frame header is in the CARP advertisements. It is up to the switch to add it to the table since it received traffic from that MAC address on that port.
The ARP response to a WHO HAS for the CARP VIP is sourced from the interface MAC address, but contains the CARP VIP in the IS AT response.
Same with the gratuitous ARP. Sourced from the interface MAC, contains the CARP MAC as the sender MAC.
-
The only time (that I know of) the CARP MAC address itself goes on the wire in the frame header is in the CARP advertisements. It is up to the switch to add it to the table since it received traffic from that MAC address on that port.
This is actually codified in RFC5798 7.2 for VRRP.
And here is what is strange here: I'm looking at tcpdumps in wireshark and seeing the MAC of the physical interface in the header, not the VIP's MAC:
Frame 53: 70 bytes on wire (560 bits), 70 bytes captured (560 bits) Ethernet II, Src: SuperMic_ea:4b:70 (0c:c4:7a:ea:4b:70), Dst: IPv4mcast_12 (01:00:5e:00:00:12) Destination: IPv4mcast_12 (01:00:5e:00:00:12) Address: IPv4mcast_12 (01:00:5e:00:00:12) .... ..0\. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast) Source: SuperMic_ea:4b:70 (0c:c4:7a:ea:4b:70) Address: SuperMic_ea:4b:70 (0c:c4:7a:ea:4b:70) .... ..0\. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: IPv4 (0x0800) Internet Protocol Version 4, Src: X.X.X.X, Dst: 224.0.0.18 Common Address Redundancy Protocol -
Where is that capture taken? That is not what I see (or ever have seen).
-
Anything else? Virtualization of pfSense?
-
All real hardware (Supermicro Xeon E3-1270 machines, Supermicro-branded Intel X520-SR2 10G cards being used for WAN/LAN, there's i210 onboard that's used for PFSYNC).
Captures were taken at one of the switches, I believe the one that the carp master is connected to (I have so many dumpfiles, I've been staring at Wireshark and RFCs all day)
-
What does a capture filtered on CARP taken on pfSense show? What is the source MAC there?
Diagnostics > Packet Capture, Select interface, Protocol: CARP on the current master.
I'm starting to smell something like shared IPMI maybe. Just a hunch. Probably wrong.
-
Should not be IPMI related, the first thing we do after powering on Supermicro boxes is to set the IPMI to dedicated.
Packet capture on the firewall also indicates carp packets sourced on the physical address:
Frame 1: 70 bytes on wire (560 bits), 70 bytes captured (560 bits) Ethernet II, Src: SuperMic_ea:4b:70 (0c:c4:7a:ea:4b:70), Dst: IPv4mcast_12 (01:00:5e:00:00:12) Destination: IPv4mcast_12 (01:00:5e:00:00:12) Source: SuperMic_ea:4b:70 (0c:c4:7a:ea:4b:70) Type: IPv4 (0x0800) Internet Protocol Version 4, Src: X.X.X.X, Dst: 224.0.0.18 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x10 (DSCP: Unknown, ECN: Not-ECT) Total Length: 56 Identification: 0x984a (38986) Flags: 0x02 (Don't Fragment) Fragment offset: 0 Time to live: 255 Protocol: VRRP (112) Header checksum: 0x0000 [validation disabled] [Header checksum status: Unverified] Source: X.X.X.X Destination: 224.0.0.18 [Source GeoIP: Unknown] [Destination GeoIP: Unknown] Common Address Redundancy Protocol Version 2, Packet type 1 (Advertisement) Virtual Host ID: 102 Advertisment Skew: 100 Auth Len: 7 Demotion indicator: 0 Adver Int: 1 Checksum: 0x5e00 [correct] Counter: 17876480869048372106 HMAC: 8700fa73033777bc3da9ed687db21e9ca7ede6e4 -
OK, worked out the cause.
We are SPANning the LAN port to call recording servers (this installation is a massive-scale asterisk farm, 800 instances of asterisk both virtual and bare metal in 3 racks). Something in the bridge is rewriting the source mac. When I removed the SPAN ports the carp announcements became correct.
The "broken" switch is totally innocent.
This is a pfsense bug, but might be hard to fix.
I've logged this as bug 7648 in redmine.
-
Right. Silly place for a SPAN port with a $75,000 switch at-the-ready.
When you said the switch wasn't learning the CARP MACs I (incorrectly) assumed you had verified the MACs were arriving at the switch and not being learned. I'll try to do better next time.
-
There's some odd quirks in the Broadcom Trident II ASIC on the Arcticas that make it impractical to SPAN on the switches (they insert dotq tags on SPANned packets originating on untagged ports but destined for tagged ports - you wind up with a mix of tagged and untagged packets coming out your SPAN port).
We do have a workaround for the moment: running a script on the firewalls that periodically does an arping (using the CARP mac) if that firewall is currently the carp master, and a trigger on a CARP state change that does an immediate arping on a BACKUP->MASTER transition.