COX and the CARP mac
I've been searching the forums for this problem and it seems that we might just need a new ISP but for the time being my client is stuck with Cox's "Business" cable modem service.
We have (3) public (static) IP addresses and (2) SG-4860.
The lowest address .53 is the Carp address.
The .54 address is on FW1 WAN
The .55 address is on FW2 WAN
I've changed my NAT rules so outbound traffic is nat'd to the HA address on .53.
If I shutdown FW1, the CARP address moves over to FW2 where it becomes MASTER as expected. However, the .53 address is unreachable from the Internet and users on the LAN can't surf the Internet ... YES, there's a LAN CARP too and it's the default gateway for LAN devices.
The only way to get the .53 CARP address to work on FW2 is to power cycle the cable modem. The problem with this is that if FW1 comes back up, even though the CARP address moves to FW1, the cable modem still thinks it's on FW2 so again we can't pass traffic.
I was able to escalate this all the way up into their NOC and spoke to an Engineer. He explained they don't allow MAC address changes from the customer's end to prevent IP spoofing and told us it was for our own security. I think it's to make up for their weird provisioning scheme since they put us in a /25 segment and only allocated 3 IPs to us.
The engineer said he saw the MAC of our HA address was the same as our FW1. He suggested we assign a dedicated MAC to the Carp IP which could then move between the two firewalls. He mentioned this is how SonicWall does it. Grrrrrr... spit....
Is there anyway to circumvent their silly no arp flushing MAC control by assigning a static MAC entry?
The MAC address for the CARP VIP is static. It is based on the VHID.
It will look like this: 00-00-5e-00-01-XX where XX = 0xVHID
Is there a switch between the cable modem and the firewalls or are you trying to use a switch built into that device? If the latter, try a switch into one port on the Cable device.
The ISP Layer 2 device will see the CARP MAC as the source MAC in the CARP advertisements. They are sent to the Layer 2 Multicast address 01:00:5e:00:00:12 (all points multicast) to Layer 3 multicast address 22.214.171.124. That MAC address has to be added to the switch port's MAC address table based on those. This MAC address will change ports on a failover event. The ISP device must move the MAC address to the new port as any switch should.
The ISP Layer 3 gear will get the CARP MAC in response to ARP "WHO HAS" requests for the CARP VIP address. Their gear needs to do the right thing with it. The ARP reply from the WAN interface that is currently CARP MASTER will contain the CARP MAC in the ARP "IS AT" response. This ARP response will be sourced from the interface IP and MAC address.
The ISP Layer 3 gear also needs to honor the interface addresses that will ARP as normal. The ISP device will only ever see the interface MAC address on the port connected to that node.
Thanks for the explanation Derelict.
I failed to mention, that we do have an HP Aruba switch in between their cable modem box and our 2 firewalls since their equipment only has 1 ethernet port.
All three ports in the switch are vlan'd as untagged ports on vlan 900 so the traffic is isolated from the LAN. This is the standard way I've done this at well over 30 sites around the country. Just not with COX.
When working with the Cox Engineer they would first see a MAC associate with the CARP address as you described but it would not respond to ping. After we rebooted all of the equipment the CARP IP would respond to ping but the Engineer would have the same MAC for the CARP IP as Firewall 1's WAN Interface and not the 00-00-5e-00-01-XX address that we expect.
Sounds like their gear is not CARP-compatible, unfortunately.
No excuse for that these days.
You can always ask them how to configure it for VRRP instead. VRRP might be something they are more familiar with (and if they do what they need to do for VRRP, CARP should work too.)
Their layer 2 port should have the following MAC addresses in its table:
Primary's WAN interface MAC
Secondary's WAN interface MAC
It should not care at all which node's interface is actually holding the CARP MASTER. It is one switch port with three MAC addresses from their perspective.
Their Layer 3 interface should have three ARP entries:
CARP VIP: CARP MAC
Primary interface IP address: Primary WAN MAC
Secondary interface IP address: Secondary WAN MAC.
When working with the Cox Engineer they would first see a MAC associate with the CARP address
Kind of need to determine what layer you're talking about. There is no association of IP addresses with MAC addresses at Layer 2. It sounds like they are talking about ARP there but it is not clear.
It sounds like their gear is broken.
They also need to not enforce an IP:MAC relationship in such a way that drops traffic if there is a mismatch. For example if they send a packet to a CARP VIP, the destination will be the CARP MAC, but the reply will come from the MAC address of the current master node.
@jimp, you nailed it!