Is the primary node actually seeing the interface go down? That is what is necessary to trigger a failover. It will fail over just fine with an actual interface failure. Even only one of many.

CARP does not protect against a failure at Layer 2. That is up to you to provide Layer 2 redundancy in addition to Layer 3.

It has zero to do with NAT.

@derelict said in DHCP from Backup Node?:

If you view Status > DHCP Leases you should see normal/normal on both nodes. If not, something is wrong.

Yes, I got it working so far, and yes I gave both gateway and DNs the CARP Virtual IP. After figuring out I have to add the slave IP there it started working.

Unfortunately only on two of the interfaces, but not on the third.
There it says "My state: recover" and "Peer State: unknown state"

This is obvious not "normal", but how do I troubleshoot? I alread stopped both services and removed the dhcp-leases files on both servers, but no change.

I can ping both addresses vice-versa. How can I troubleshoot?

After re-configuring the dhcp service again and again it went finally to "normal/normal". So it is working now and I am fine.

I personally really like the IP Alias VIPs stacked on the CARP VIP.

You only have one "stream" of CARP heartbeats and you can move dozens of VIPs at a time from primary to secondary and back.

The only time I generally make multiple CARP VIPs is for cleanliness in cases where you have VIPs in multiple subnets. I generally make one VIP per subnet and stack the IP Aliases that are also in that subnet on that VIP. This is a personal preference.

If you make all of them CARP, then you need a VHID for each of them and any missed advertisement will result in that VIP swinging to the other node while the rest remain. This is never what you want.

The stacked IP Alias technique reduces the advertisement traffic to that of just the one VIP.

What you possibly can do:

Make 2 VPN tunnel. On from the first pfsense and one from the second pfsense. Then you can still make CARP but you configure to NOT sync the IPSec conig.
When the failover takes place, the vpn tunnel will already be up.

depending on your setup you may run ospf or another routing protocoll with the two vpn tunnel to make changes that are nessessary due to topology change.

Best Regars,
blex

Hi,

thanks for reply.
No, it's a dedicated interface and there is no captive portal in it.
It's on a different Interface and also VLAN.

If you bring up the master again the slaves config will be overwritten by the master.

To avoid that disconnect the masters sync interface and reverse the config sync direction by removing the "Synchronize Config to IP" from the master in System > High Avail. Sync and configuring the XMLRPC Sync on the slave. Ensure that all options you want to sync are checked.

It might work fine using an interface address or an IP Alias VIP but not work CARP (using the identical IP address) because of improper handling of the necessary MAC address behavior by something upstream.

I found that the solution can also lie in the interface settings.

https://forum.pfsense.org/index.php?topic=129871.0

In the OpenVPN Client Protocol dropdown, you probably have selected "UDP IPv4 and Ipv6 on all interfaces (multihome)".
That ignores the selected interface.
Select "UDP on IPv4 only"

This fixed the problem on my end.

I'm doing well this week with answering my own posts…

Got this working by creating the IP Alias on the cache interface, then setting up outbound NAT for "This Router" as the source to the

Yes. preferably a /29

That was it, thanks.

Hi Derelict

I found out that in hyper-v i must activate the option for "mac address spoofing" on the vNics, after that all works fine :)

Spoofing.PNG
Spoofing.PNG_thumb

Not that I know of.

as I now found out I was wrong that CARP MAC is randonly calculated every reboot.
The CARD MAC always is "00:00:5E:00:01:<vhid>"

We are going to check with global admin whether we can get a static VHID on WAN and therefore register the resulting CARP MAC.</vhid>

Have to probably see a network diagram, complete will full IP addressing, subnets, and gateways and a complete description of the traffic that is causing the "storm" preferably with a packet capture.

There should be at least four addresses on the WAN network:

Upstream gateway
CARP VIP
Primary WAN interface
Secondary WAN interface

Traffic should be able to freely flow between any of those interfaces without issue.

No, it is not normal to use Automatic outbound NAT in an HA configuration but it should not, in and of itself, cause the issues you say you're seeing unless something else is wrong.

Hi,

I tried your suggestion. Same problem. I also tried an outbound NAT rule "WAN, any, "WAN Interface IP/32" (as only networks can be entered) any "Public CARP VIP"  (which I think is essentially the same as what you are saying). Both where the top-most outbound NAT rule.

What I see with both NAT rules in TCPdump are icmp-echo-requests leaving as the CARP Public IP  (so that's good as it hits the new NAT rule) but nothing comes back.

To my amazement, I also still see icmp requests leaving for the same target (router at provider) with source IP 10.99.99.204 (the WAN interface of the Master). These are not mine and I guess the Gateway pinger sends them every 1 second.

So I see two source-addresses when pinging the provider's device (which is my def.gateway):

"CARP Public IP" (82.136.xx.yy)  to  "Provider's router" (82.136.xx.zz)  but only when I ping it myself directly from the Firewall's native console (i'll call these "my pings") "Master WAN IP  (the 10.99. address) to "Provider's router" every second, I guess from the gateway-check-pinger-thingy.
(tcpdump sees no return packets aka icmp-echo replies)

raw output:
IP 82.136.xx.yy > 82.136.xx.zz: ICMP echo request  (my pings"
IP 10.99.99.204 > 82.136.xx.zz: ICMP echo request  (gateway checker thingy pings, 1 every second)

The latter I don't understand. I created that outbound NAT rule and when I do pings (the "my pings"), it translates them and then sends them out.  But the gateway-check-pinger  seems to ignore NAT rules as it's icmp-echo-request packets have the physical WAN IP of the master as the source IP ????

The end-result is that pings now work. Sometimes…
So pings work for a couple of minutes, then die for a while, then pings work again, die again. The intervals are random.
I can ping the providers router now and some devices inside their network but not beyond.
When I revert to a non-CARP setup for the WAN interface, as described in my OP, all is 100% good again.

I'm totally out of ideas guys.

No you are trying to game HA.

It sounds like the port forward on the secondary is working but the target is sending its reply traffic back to its default gateway - the primary.

You can probably make this sort of work by using outbound NAT on the LAN interface so all traffic appears to come from LAN Address so the replies are same-subnet.

If you have Multi-WAN I wouldn't do HA at all.

I would use one node for both WANs and be sure to keep a regular copy of the configuration backed up and keep the other node as a warm or cold spare.

That or get the proper WAN subnets (/29 or larger) and configure HA correctly.

Depends on what you are configuring/doing.

You generally want services/VPNs to listen on CARP VIPs. You generally want outbound NAT to be a CARP VIP. You generally want inside clients to use a CARP VIP as their gateway and DNS server (if it's providing DNS).