CARP Issues with ISP
I am having an issue with my ISP when I enable CARP. I remember reading some posts long ago where others had issues with their ISPs with CARP, but I can't seem to find those ports or remember all the details, but I think at least some of the posts stated that the ISP was using Proxy ARP.
I will describe my setup and what happens. At this time I am not even testing with the 2nd firewall on, I am just setting everything up in the primary firewall.
Cable modem setup for a /29 netblock and a routed /28. The first usable IP on the /29 is the gateway on their end.
The cable modem is connected to port on a Cisco switch and the firewalls are connected to the same switch. The only devices connected to this vlan are the cable modem and the firewalls. The firewall connection is a trunk.
When I enable a CARP IP on the interface for this ISP, within a couple of minutes the firewall gets a ton of packet loss to the gateway IP and from outside devices I cannot ping the WAN IP on the firewall. I can access the CARP IP and alias IPs without as issue.
My hope is someone has experienced this or a simal issue and has some insight as my ISP can see the problem, but is not sure why it is happening.
My guess is that ISPs system, probably the CMTS, is seeing the massive amount of CARP traffic broadcast from the firewall IP on the interface for this ISP and starts rate limiting the firewall IP, but not the entire connection or other IPs.
I know it is not my switch as I have CARP on all the interfaces. I have a 2x 1GB LACP connection between the firewalls and the switch that is a trunk with multiple VLANs for my internal networks. From the same switch I have a 1GB connection setup as a trunk with the VLANs for my WAN connections. I have tried this without trunking with no relief.
Outside of the static IPs, I believe my cable modem and connection in general is setup like any consumer account and that may be the issue. I know for business accounts they usually provide an SMC device and I have a setup with redundant pfSense routers using CARP with Bright House who provided an SMC and I have not has issues from day 1. The ISP for my house is different, but I believe they also provide SMC routers for business class connection. That may not make a bit of difference as this could be a settings issue and the needed settings may work fine with the cable modem I have.
Thanks for your help!
AFTER your box is up and running (not so well as you write), call the ISP and ask them to clear the cable modem's ARP table. If it starts working (for a little while), then you are seeing the same thing I'm seeing which only started within the last few weeks.
Somehow, the router in the cable modem puts the interface's actual ip in its ARP table with the first VIP mac address. Then, it puts the VIP's mac address in it's table for the pf router's actual interface card ip.
The result is no traffic gets through. Traffic addressed on the link to the vip has the interface address and so it gets dropped. Traffic addressed to the VIP on the link has the interface's IP and that gets dropped. I still don't have an answer.