is 3-way High Availability nodes possible?
-
We have a cluster of 3 ESXi hosts, and currently running 2 pfsense in HA mode. Is there any way to add another pfsense for 3-way HA? I'd like to account for a scenario where 2 hosts fails and we still have continued operation. We have more than enough static public IPs for this.
I always see references to "2 or more" pfsense required for HA, but I think this is the case of "2, not more". I just don't see how his can be configured from the GUI.
-
@axsd If they are virtual, can't the hosts be set to auto-start the VM on another host, if one host fails? It seems like that would be much easier to manage.
-
I can definitely run the primary pfsense as a "shadow" copy to our 3rd ESXi host (although this hasn't been tested), but doing it this way would mean if two hosts were down at the same time, there would be about 30sec to 1 minute of outage while the shadow copy boots up on the 3rd host.
pfsense's HA is better in this regards, with only about a second of outage when the primary switches to the secondary firewall. And I'd like to implement this on the 3rd host using a 3rd pfsense instance rather than depending on the ESXi cluster to handle the failover.
If there are no other options with a 3rd pfsense HA, then I will go with the route you suggested.
-
The other bandaid solution I can think of is to run the 3rd pfsense (on ESXi host #3) to sync with pfsense #2 (which syncs with primary pfsense #1).
But this would also mean that if ESXi host #2 fails, then the pfsense #3 is orphaned because it doesn't sync directly with primary pfsense #1.
-
@axsd I am not saying it won't work, but as you noted the docs say it's possible but then immediately say it's hardly ever necessary and don't explain it.
The config sync is separate from the states sync. The latter ought to be way more important for keeping connections up. That seems like the easy part: "pfsync uses multicast by default, though an IP address can be defined to force unicast updates for environments with only two firewalls where multicast traffic will not function properly."
-
You can run more than 2 HA nodes. It's not supported though.
3 is relatively easy, more requires some code changes. That's because the config sync code adds 100 to the advskew of the CARP VIPs for the target and the maximum is 255. So the 1st node is 0 on all CARP VIPs and syncs the 2nd node as 100. The 2nd node syncs (which it would not normally do) to a 3rd node at 200.
Other parts of the config sync may fail. The DHCP servers auto configure two nodes to share the load but I have no idea what it would do with 3.
The CARP VIPs can exist with many nodes as long as they advertise at slightly different rates.
You would have to use multicast state sync if you need seamless failover so all nodes you see all other nodes. That could be a significant amount or traffic.
I have occasionally seen people run more than two nodes but YMMV!
Steve
