WAN CARP not working/seems like ISP issue
-
Been working on this on and off for days. Driving me crazy, I've setup HA pfSense boxes with CARP at 3 other locations with no issues, but am having serious issues on this one. Anyone have advice?
Setup as follows
WAN IPs
xxx.xxx.xxx.100 CARP IP
xxx.xxx.xxx.98 Primary
xxx.xxx.xxx.99 SecondaryLAN IPs
192.168.1.1 CARP IP
192.168.1.2 Primary
192.168.1.3 Secondarywe have a block of 5 IPs xxx.xxx.xxx.96/29
xxx.xxx.xxx.97 is the gateway on my ISP
Patch cable between each firewall and modem
Firewalls are identical hardware wise
I am 99.9999% positive my settings are correct. It works, exactly as expected, if I switch the WAN IPs to a LAN with a different upstream(Carp 172.16.0.1, primary 172.16.0.2, backup 172.16.0.3) This leads me to believe it's a problem with my ISP.
When i configure the Firewalls with my public IPs(.98 & .99) and a shared CARP IP(.100) and change outbound NAT to CARP IP, it appears I'm not getting any internet on my Firewalls, both gateways go to 100% loss. I thought my pc wasn't able to access the internet but I realized it is the DNS on the firewall is unable to resolve anything. I switched my PCs DNS to 8.8.8.8. I visited Whatsmyip.org and it appeared as the correct CARP IP address. Failover doesn't work though when I switch off the Primary.
I can delete the CARP IP and both firewalls can connect to the internet, I can set a NAT rule for "this firewall" and point it to the CARP IP and the firewall can connect, but obviously this causes issues on the secondary firewall.
It's like the ISP gateway is accepting from any of my 5 IPs, unless one is a CARP address, then it only accepts from that address and the others become unusable. I attempted to swap around the IPs and make .100 primary .101 secondary and .102 CARP and it does the exact same thing.
I really hope someone can help. Thanks!
-
Probably the ISP device only allowing one MAC address or something.
With CARP there will be a CARP MAC and a MAC address for each interface. You might have to packet capture and look at the traffic in wireshark to see what's really happening.
They will need to honor the MAC address for the CARP VIP from whatever node is the master at the time. This is in addition to the interface address and MAC address from both nodes simultaneously.
-
Probably the ISP device only allowing one MAC address or something.
With CARP there will be a CARP MAC and a MAC address for each interface. You might have to packet capture and look at the traffic in wireshark to see what's really happening.
They will need to honor the MAC address for the CARP VIP from whatever node is the master at the time. This is in addition to the interface address and MAC address from both nodes simultaneously.
I thought this at first, but if that was the issue, wouldn't there be issues with the two firewalls with CARP disabled? I can have one PC point to .2 as it's LAN gateway and go to the internet as .98 and another PC pointed to .3 as it's LAN gateway and go to the internet as .99 and I can do this simultaneously with transfers running on both PCs.
-
Like I said, you will probably need to pcap to see what is really going on.
Are you using your own switch or the ISP device?
