Single Pub-IP CARP - no Internet connectivity after implementing CARP
-
I'm sorry for asking the 10000000th question about doing CARP with only one public IP. I've done "3 public IP" WAN-CARP before and had no problems. On one site, they have one public IP and asked me if it would be possible to still do Firewall HA despite only having one IP. You've heard this a million times by now and again i'm sorry for asking.
They had one ESXi virtual pfSense Firewall VM with public IP A.B.C.D/24 (not DHCP but static assignment from their ISP) on the WAN interface.
I've built a second box, re-configured the WAN interface with a private IP 10.99.99.205 and the original one with 10.99.99.204 (see drawing below).
I made CARP VIP's for the internal VLAN's and the WAN and it all works fine. No dual master issues. Fail-over works from the inside etc. All good. I have no issues with CARP itself.The WAN CARP VIP now has the A.B.C.D/24 public IP address and I changed the outgoing NAT rules of the various internal networks to use that instead of the WAN interface as was the case before I started.
I also changed the firewall rules accordingly and there are no drops. All good.
I removed the "upstream gateway" from the original WAN interfaces and replaced it with a Gateway object (which is the network device at the provider) and making it the default.The problem:
When i'm SSH'ing into the first (original firewall) which is Master for all interfaces, incl. WAN, I can no longer ping the next hop which is the network device (my gateway) at the provider. So i'm A.B.C.D/24 and the provider's device is A.B.C.E/24.
Looking at if-config, it all looks fine. The vmx0 interface now has both the private 10.99.99.204 as well as the public A.B.C.D/24 address.
Even if I would not have any NAT rules at all, the firewall should at least be able to ping it's own next hop/ gateway which sits inside the same /24 network as it's own public IP.If I totally remove the WAN CARP VIP, and re-configure the (former) Master's WAN Interface with that public A.B.C.D addresss, I can ping the providers device (the default gateway) again.
All I really do is:
"remove the WAN CARP (and make sure it's gone from both Firewalls) give the native WAN Interface vmx0 on the "Master" the public IP -> can ping the provider again"
then
"give the native WAN the internal 10.99.99.204 address, create the WAN VIP with the public IP, make sure CARP does the proper Master and Backup roles thing" -> ping is dead.While using the WAN CARP VIP and doing a TCP dump on the WAN interface, I see hosts inside the same /24 provider subnet as me (other customers with static IP's) chatting with each other, arp-requests and replies flying by so I have L2 connectivity to the provider. My MAC-address does not change when switching to a WAN CARP so i'm not running into the issue where the provider still thinks my IP is on a different MAC address etc.
It should work. But it doesn't. I tried rebooting the firewall pair, one by one, the provider's router etc. just in case. No dice.I cannot get it to work and what bothers me is that, on the one hand, every official documentation says "you need 3 public IP's" but multiple forum-posts report success with only one public IP. And I do exactly what they do.
And common sense tells me it cannot work for the simple reason that tcpdump on the FW shows that the source IP of those pings to the provider's device, are the WAN's own local (private) IP and not the CARP IP.TCPDUMP output directly on the Master firewall on Interface vmx0 shows:
IP 10.99.99.204 > A.B.C.E: ICMP echo request, id 46006, seq 305, length 8
Of course that does not work.
What would work if I saw this:
IP A.B.C.D > A.B.C.E: ICMP echo request, id 46006, seq 305, length 8
but I don't.So how can it work for others? Do they have a way to make pfSense send packets really originating from their WAN single Public IP CARP address ?
Does not work:
Ping from A.B.C.D on "FW A" to Provider's network device at A.B.C.E
–--------------------------
| |
| ISP Switching env. |
| |
----------------------------
| IP: A.B.C.E/24
|
|
/ Internet Link
/
(Master) / (Backup)
CARP: A.B.C.D/24 /
------------------- -------------------
| 10.99.99.204/24 | | 10.99.99.205/24 |
| | | |
| FW A | | FW B |
| | | |
------------------- -------------------
|.204 |.205
|.254 (CARP VIP) |(Backup)
| |
------------------------------------------------
Internal LAN 192.168.10.0/24
Default GW on this LAN is .254Works:
Ping from A.B.C.D on "FW A" to Provider's network device at A.B.C.E
----------------------------
| |
| ISP Switching env. |
| |
----------------------------
| IP: A.B.C.E/24
|
|
/ Internet link
/
/
(no CARP) /
------------------- -------------------
| A.B.C.D /24 | | 10.99.99.205/24 |
| | | |
| FW A | | FW B |
| | | |
------------------- -------------------
|.204 |.205
|.254 (CARP VIP) |(Backup)
| |
------------------------------------------------
Internal LAN 192.168.10.0/24
Default GW on this LAN is .254 -
Apologies for just skimming, don't have time to read the whole thing carefully.
Try adding an outbound NAT rule- WAN, any, "this firewall(self)" any [Public CARP VIP] -
Hi,
I tried your suggestion. Same problem. I also tried an outbound NAT rule "WAN, any, "WAN Interface IP/32" (as only networks can be entered) any "Public CARP VIP" (which I think is essentially the same as what you are saying). Both where the top-most outbound NAT rule.
What I see with both NAT rules in TCPdump are icmp-echo-requests leaving as the CARP Public IP (so that's good as it hits the new NAT rule) but nothing comes back.
To my amazement, I also still see icmp requests leaving for the same target (router at provider) with source IP 10.99.99.204 (the WAN interface of the Master). These are not mine and I guess the Gateway pinger sends them every 1 second.
So I see two source-addresses when pinging the provider's device (which is my def.gateway):
- "CARP Public IP" (82.136.xx.yy) to "Provider's router" (82.136.xx.zz) but only when I ping it myself directly from the Firewall's native console (i'll call these "my pings")
- "Master WAN IP (the 10.99. address) to "Provider's router" every second, I guess from the gateway-check-pinger-thingy.
(tcpdump sees no return packets aka icmp-echo replies)
raw output:
IP 82.136.xx.yy > 82.136.xx.zz: ICMP echo request (my pings"
IP 10.99.99.204 > 82.136.xx.zz: ICMP echo request (gateway checker thingy pings, 1 every second)The latter I don't understand. I created that outbound NAT rule and when I do pings (the "my pings"), it translates them and then sends them out. But the gateway-check-pinger seems to ignore NAT rules as it's icmp-echo-request packets have the physical WAN IP of the master as the source IP ????
The end-result is that pings now work. Sometimes…
So pings work for a couple of minutes, then die for a while, then pings work again, die again. The intervals are random.
I can ping the providers router now and some devices inside their network but not beyond.
When I revert to a non-CARP setup for the WAN interface, as described in my OP, all is 100% good again.I'm totally out of ideas guys.
