Couple question: Force Master and Why CARP VIP on WAN?
So a couple questions I have on CARP...
First off, is there a way to force which node is primary? I don't see anything in the GUI to do this. It just seems like it's the oldest boot, or maybe the oldest link that gets to be Master. So if I boot both around the same time I end up split brained with LAN a Master on one node and WAN a Master on the other. Which I assume means traffic goes out over one node and in through the other.
Is there a way to make it so that one is always the Master as long as all links are up, or that the Inbound and outbound VIPs are always married to each other? My nodes are sandwiched between 2 stacked redundant switches inside and out, the interfaces exceed the capacity of available bandwidth by a significant margin, and fully meshed, so an active-active function buys me nothing for capacity or redundancy and actually negatively effects performance.
Also both nodes seem to serve DHCP regardless of who owns the LAN, I don't allow unknown MACs, I let new clients acquire a "dead" IP so I can find them then add them to the DHCP table. Now that I have gone CARP, I have to hunt between the two for the bogus lease. So related to the first question, how do I only have DHCP serve only on the owner of the LAN interface?
Which brings us to the 2nd question, well 3rd, or anyway, 2nd question of the subject. I followed the configuration examples, well not at first, initially I only configured a CARP VIP for the LAN, but still not sure why I need to VIP the WAN. So I did a VIP for the WAN, but why does that matter? It seens to work the same either way and I'm not sure I understand why it wouldn't. I have 2 external connections, and ultimately neither connection will hit the street as a different IP, both connections gateway through upstream, but still local, subnets. Well at the moment one does, the other is directly connected to one of the 2 nodes until my Mikrotik gets here because well Spectrum won't dole out 2 IPs on their modem, and actually seems to not dole out any if there is multiple recipients.
Also the WANs are weighed since one connection is about 5 times faster than the other, but both nodes will be connected to both via stacked switches. So the way it's intended to be setup, again awaiting the little Mikrotik router/switch, is that failure of a carrier, CPE, a stack node or that MiKrotiK should result in loss of one WAN. Maybe I could even push that upstream further so that the weighting is handled by the switch and LAGG the WAN to be truly fully meshed, but just trying to set up something that works as intended right now, though realize there is a distinct possibility that this may result in a halfway up WAN link. Anyway any other failures should just be business as usual as far as the external connection is concerned.
But back to the question(s), if one pfsense firewall owns the LAN and the other is just sitting around, ready and waiting, for that node to fail why would you need a VIP on the WAN side? Shouldn't it just NAT relative to what ever WAN IP(s) is/are there on that particular node? And how is NATing done when it get all into split brain with one node owning the LAN VIP and the other owning the WAN VIP? Is this happening via the interlink interface somehow? Can services be disabled based on who owns the LAN VIP (Specifically DHCP)?
Oh, and how do you force a Master other than rebooting the node you don't want to be Master? Is there anything in the GUI to do that? Or a specific shell command? I don't see this addressed at all in the docs, but maybe I just missed it. There is very little in the GUI concerning CARP that I can see, just config transfer parameters and the specialized VIP type, oh and some stat page that I have no idea what those hex strings indicate.
@skorpio The master should always be the master unless a connection fails. The two routers should have a different skew.
To force the master to be backup, on the master, go to Status/CARP and click "Enter Persistent CARP Maintenance Mode." This raises the skew of the master so the backup has a lower number and thus becomes master. Commonly used when upgrading.
re: VIP the WAN, that's kind of the point of HA? Configured correctly, packets go out the shared WAN IP, and when things fail over the connections are left running on the backup router but the same shared WAN IP. States sync to the backup router as long as the interfaces are the same. The server at the other end should ignore packets that suddenly start coming from a different IP, for an open connection.
It sounds like you have multiple WANs also? I haven't done that but see:
https://docs.netgate.com/pfsense/en/latest/recipes/high-availability-multi-wan.html. I might set HA up for one ISP then add the other?
I haven't set up DHCP in an HA scenario so can't help much there either.
@steveits So interestingly enough at the time I wrote the post I'd just rebooted the server that only currently has one WAN (The whole waiting on the Mikrotik thing), So right now the two firewalls are not equal, one has way more bandwidth then the other. But I also, for giggles, noticed that CARP Maintenance Mode switch, which was actually IN "Persistent Maintenance Mode" on both, and I clicked "Leave...". So maybe the whole split brain thing was because of that.
"kind of the point of HA?", well yea fair enough, though I guess I was looking at it from the standpoint of there being 1 firewall active at any one time, and that the IP of that VIP is not visible to either internal or external traffic, if it's buried in a NATed subnet. Which yea is certainly not always the case and any active session would hurl. So yea, obvious now.
Because the one firewall did not have a corresponding 2nd WAN I did not VIP that one, looks like that was the correct call as per the link.
Thanks for the Multi WAN link, let me mull that over, although it not completely relevant until I get my little stub router to deal with the cable modem issue.
So looks like there is a "Failover Peer IP" parameter in the DHCP server config that "requires CARP" so that's probably the key to it all. It also mentions "setting the skew" so where can that be done? It's described as if it can be specifically set to a particular value, is that in the GUI somewhere, or in a file?
@skorpio The CARP alias skew is set in each alias: https://docs.netgate.com/pfsense/en/latest/recipes/high-availability.html#configuring-the-carp-virtual-ips
"A primary node is typically set to 0 or 1, secondary nodes will be 100 or higher. This adjustment is handled automatically by XML-RPC synchronization."