4 IP addresses not working as CARP VIPs
This is a strange one.
I have a network x.x.x.64/28 network from a cable internet provider. x.x.x.65 is the router provided by ISP. My usable range is x.x.x.66 ~ 78
Have a pfSense 2.1 firewall pair behind it. On x.x.x.67 and x.x.x.68.
On x.x.x.66 there is a pfSense 1.2.3 firewall soon to be decommissioned. That firewall has no Proxy ARP or CARP setup on it. Just its own interface address (x.x.x.66).
Between the two 2.1 firewalls I have x.x.x.69 ~ 72 setup as CARP VIPs. Those VIPs stopped working mysteriously a few days ago. They are no longer reachable from the internet. From any machine except the ISPs router on the x.x.x.64/28 subnet I can ping those VIPs and get to the services NATed/port-forwarded to those VIPs. No problem. From ay of those VIPs I can ping all machines on that x.x.x.64/28 subnet except the ISP's router (x.x.x.65) or any pingable device on the internet.
If I remove those VIPs from the firewall and stand up a good old Linux test machines with IPs in the 69~72 range, I can ping the ISP's router from that machine and I can ping that machine from the internet.
If I setup CARP VIPs on the 2.1 firewalls in the 73 ~ 78 range, all those VIPs work fine. They are reachable from the internet and they can send traffic to the internet. The firewalls interfaces (66, 67, and 68) are also reachable from the internet.
If I re-setup any VIPs between x.x.x.69 ~ 72, they still cannot send any traffic to the internet nor are reachable from the internet. But they are reachable from other machines on the same network. Have completely removed those VIPs from the firewalls, rebooted the3 firewalls and added them back again. Same behavior.
I have checked and double-checked for IP conflicts and there are none.
Doing packet captures I can see that the ISP's router ARPs for all IPs on the x.x.x.64/28 network when it needs to get to them except for 69~72 when they are setup as CARP VIPs. It does however ARP for them when I assign them to regular Linux test machines. The moment I configure those IPs again as VIPs on the firewall, the router no longer ARPs for those IPs when something from the internet needs to get to them. But other machines on the same subnet ARPs from them just fine and start talking to those VIPs.
Unfortuantely, I do not have console access to the ISP router. Being assured by the ISP that there are no stuck arp cache entries in there. Which I do believe because I can give a Linux machine an IP 69~72 and it works. And the router has been rebooted as well.
Luckily, I have 3 IPs to spare so I have moved my VIPs from 69~72 –> 73~76. But now I am completely out of IP addresses on that network.
Using arping from a Linux machine I have tried "arping -S x.x.x.69 -B" in an attempt to get the router to drop whatever it has that is causing it to not arp for the VIPs IP with no results.
Should also mention that setting a CARP VIP on the 1.2.3 firewall in the 69~72 range doesn't work either.
A standard Linux machine with those IPs works fine.
And I just lost 74 as well. It is no longer working either as a CARP VIP. Works if I give the test Linux machine that IP. Doesn't work as a CARP VIP. Not working from 1.2.3 or 2.1.
Found the answer –> http://forum.pfsense.org/index.php/topic,66838.0.html