COX and the CARP mac
-
I've been searching the forums for this problem and it seems that we might just need a new ISP but for the time being my client is stuck with Cox's "Business" cable modem service.
We have (3) public (static) IP addresses and (2) SG-4860.
The lowest address .53 is the Carp address.
The .54 address is on FW1 WAN
The .55 address is on FW2 WANI've changed my NAT rules so outbound traffic is nat'd to the HA address on .53.
If I shutdown FW1, the CARP address moves over to FW2 where it becomes MASTER as expected. However, the .53 address is unreachable from the Internet and users on the LAN can't surf the Internet ... YES, there's a LAN CARP too and it's the default gateway for LAN devices.
The only way to get the .53 CARP address to work on FW2 is to power cycle the cable modem. The problem with this is that if FW1 comes back up, even though the CARP address moves to FW1, the cable modem still thinks it's on FW2 so again we can't pass traffic.
I was able to escalate this all the way up into their NOC and spoke to an Engineer. He explained they don't allow MAC address changes from the customer's end to prevent IP spoofing and told us it was for our own security. I think it's to make up for their weird provisioning scheme since they put us in a /25 segment and only allocated 3 IPs to us.
The engineer said he saw the MAC of our HA address was the same as our FW1. He suggested we assign a dedicated MAC to the Carp IP which could then move between the two firewalls. He mentioned this is how SonicWall does it. Grrrrrr... spit....
Is there anyway to circumvent their silly no arp flushing MAC control by assigning a static MAC entry?
~ Brian
-
The MAC address for the CARP VIP is static. It is based on the VHID.
It will look like this: 00-00-5e-00-01-XX where XX = 0xVHID
Is there a switch between the cable modem and the firewalls or are you trying to use a switch built into that device? If the latter, try a switch into one port on the Cable device.
The ISP Layer 2 device will see the CARP MAC as the source MAC in the CARP advertisements. They are sent to the Layer 2 Multicast address 01:00:5e:00:00:12 (all points multicast) to Layer 3 multicast address 224.0.0.18. That MAC address has to be added to the switch port's MAC address table based on those. This MAC address will change ports on a failover event. The ISP device must move the MAC address to the new port as any switch should.
The ISP Layer 3 gear will get the CARP MAC in response to ARP "WHO HAS" requests for the CARP VIP address. Their gear needs to do the right thing with it. The ARP reply from the WAN interface that is currently CARP MASTER will contain the CARP MAC in the ARP "IS AT" response. This ARP response will be sourced from the interface IP and MAC address.
The ISP Layer 3 gear also needs to honor the interface addresses that will ARP as normal. The ISP device will only ever see the interface MAC address on the port connected to that node.
-
Thanks for the explanation Derelict.
I failed to mention, that we do have an HP Aruba switch in between their cable modem box and our 2 firewalls since their equipment only has 1 ethernet port.
All three ports in the switch are vlan'd as untagged ports on vlan 900 so the traffic is isolated from the LAN. This is the standard way I've done this at well over 30 sites around the country. Just not with COX.
When working with the Cox Engineer they would first see a MAC associate with the CARP address as you described but it would not respond to ping. After we rebooted all of the equipment the CARP IP would respond to ping but the Engineer would have the same MAC for the CARP IP as Firewall 1's WAN Interface and not the 00-00-5e-00-01-XX address that we expect.
~ Brian
-
Sounds like their gear is not CARP-compatible, unfortunately.
No excuse for that these days.
You can always ask them how to configure it for VRRP instead. VRRP might be something they are more familiar with (and if they do what they need to do for VRRP, CARP should work too.)
Their layer 2 port should have the following MAC addresses in its table:
CARP MAC
Primary's WAN interface MAC
Secondary's WAN interface MACIt should not care at all which node's interface is actually holding the CARP MASTER. It is one switch port with three MAC addresses from their perspective.
Their Layer 3 interface should have three ARP entries:
CARP VIP: CARP MAC
Primary interface IP address: Primary WAN MAC
Secondary interface IP address: Secondary WAN MAC.When working with the Cox Engineer they would first see a MAC associate with the CARP address
Kind of need to determine what layer you're talking about. There is no association of IP addresses with MAC addresses at Layer 2. It sounds like they are talking about ARP there but it is not clear.
It sounds like their gear is broken.
-
They also need to not enforce an IP:MAC relationship in such a way that drops traffic if there is a mismatch. For example if they send a packet to a CARP VIP, the destination will be the CARP MAC, but the reply will come from the MAC address of the current master node.
-
@jimp, you nailed it!
-
@rhsfit once you figured out this was happening, did you (and Cox) find a solution?
-
Nope. We have just been living with it until the contract is over.
-
@rhsfit shoot, I was really hoping
Having the same problem (I think) but can't get Cox tier 2 support to acknowledge that the problem is on their end, or even really get them to understand the problem.Sounds like you were able to at least get the CARP VIP working for inbound/outbound traffic though, it just didn't switch over to the slave firewall properly when the master went down? My problem is slightly different... outbound CARP VIP traffic from the firewalls just goes into a black hole for me, master or slave.
-
Tell them it's a couple cisco routers using VRRP instead, maybe. VRRP might not seem so scary to them. The basic requirements on their side should be the same.
-
@Derelict do ISPs typically have to make accommodations for CARP to work?
-
Not real ones.
-
I mean you could always approach it like this:
"My business requirements dictate layer 3 redundancy here. What if I switch to Cisco routers and VRRP? Will that work with your gear? Is that supported?"
-
I'll give it a shot and report back
-
I'm finally getting somewhere with Cox. Just got off the phone with tier 3 support, with a tech who was familiar with HSRP and VRRP (but not CARP). He is thinking currently that the CMTS is probably dropping the outbound packets since the outbound source MAC doesn't match the VIP's (VHID) MAC in the ARP table. He says he knows VRRP works, and that they don't have to do any special configuration for it. His thought is that with VRRP the outbound source MAC would be the VIP (VHID) MAC so there would be no ARP mismatch, and therefore the CMTS wouldn't reject the packet.
@Derelict Are you able to confirm or deny whether VRRP and CARP differ on the outbound source MAC?
Cox has scheduled a call with DOCSIS on Monday to do some further investigation, so hopefully I'll get a solid answer whether they can support CARP, and if not why not.
-
@meyerds I have access to a CARP pair talking to a VRRP pair.
When pfSense sends an ICMP echo request to the VRRP address, it goes to the VRRP MAC address 00:00:5e:00:01:01.
The ICMP echo reply is sourced from the router's interface address 44:2b:03:aa:bb:cc.
Just like pfSense/CARP.
-
Just in case anyone is looking for answers, this is what I found out about CARP with a Cox Business cable modem connection: It doesn't work, 100% certain. Cox's CMTS registers the known MAC address for your firewall, which in this case is the VHID MAC, and any outbound traffic that does not match that MAC address will be dropped (as it is for traffic leaving the master firewall in a CARP pair). I finally got to the DOCSIS tech support tier, and the tech was very knowledgeable and was able to change some firmware settings in both the modem and on the CMTS in an effort to remove the limitation. Despite making firmware changes that in theory should've worked, it had no effect. Note he also said the changes he made in the CMTS would've only lasted until the next Cox scheduled maintenance on their end, anyway. He said VRRP/CARP works fine on their fiber connections for sure, so it's a limitation of the CMTS since those networks are meant for residential connections where the possibility of abuse is higher. He also said they have almost no visibility or control over the cable modem firmware, that it's very closely guarded by the modem manufacturers. Bummer.
-
Yeah that's too bad. Thanks for pursuing it further and reporting back.
-
D drees referenced this topic on
