HA on two different types of hardware
Apologies if this has been posted somewhere else, we were hoping use Pfsense on two different VMware hosts or a host and a physical desktop.
But I've noticed the comment in the documentation "Both cluster nodes are the same model with identical hardware specs."
I know it seems pretty clear by that statement that it relies on everything being identical, but has anyone tested this? Or is it just the case that the NIC's need to be the same hardware type etc.
Identical network hardware is primarily required for synchronize states to enable seemless failover.
If you want to use that feature on different hardware you can insert an abstraction layer with LAGG interfaces so that LAGG0 is WAN on both boxes, LAGG1 LAN and so on. So pfSense in HA will synchronize the states on the base of the LAGGs, which are identical at least.
per https://docs.netgate.com/pfsense/en/latest/highavailability/pfsync.html#pfsync-and-physical-interfaces "States in pfSense are bound to specific operating system Interfaces. For example, if WAN is em0, then a state on WAN would be tied to em0. If the cluster nodes have identical hardware and interface assignments then this works as expected. In cases when different hardware is used, this can be a problem. If WAN on one node is em0 but on another node it is igb0, the states will not match and they will not be treated the same."
If they are different the HA will failover but connections will drop since the state doesn't exist. LAGG will get around this as noted but as I also found out, one can't use traffic shaping on LAGG without a VLAN.
Edit: other than the interface names I don't think the rest of the hardware matters.
Hi. I recently posted a similar question on a different (old) thread, and was going to re-post here, but you saved me the trouble. In my case, I have a physical box in my data center that I want to use HA and/or test using a VM as backup.
FreeBSD does support renaming physical interfaces, so you can rename em0 to igb0, for instance. The problem seems to be that there is no supported way to make the interface renames persistent.
I'm investigating which rc* file I can hack to include the renames, which it seems to me would need to happen at the correct moment:
- Before any other pfSense services were started...
- But at the point at which the physical interfaces are available to be renamed.
And, of course, I'd have to make sure that script survives an upgrade/replacement, but that's on me.
Clearly, more than one person is interested in the ability to rename interfaces. The documented workaround, fake LAGGs, seem a bit bogus, but if all else fails, I'll try that. But, it also means you would have to implement that on both the primary and the secondary machines, making the pfSense config different on the two (vs. pushing the problem down to the OS level, leaving the primary box config pristine, at least).
Ideally, I want the config to be identical on the primary/secondary, knowing full well that there may be differences, particularly in the non-HA case, where you will need to have different interface addresses, e.g.
At a minimum, can someone document a non-LAGG workaround for this? That is, a semi-supported way to persist interface renaming. I see at least two use cases for this:
- HA on different hardware (or in a VM)
- Side-by-side testing of new releases or config changes
- (Backup support, in general, absent HA, so shared config can be used)
It just seems like interface renaming, if properly supported, would be simpler than the fake LAGG workaround.
FWIW, I looked through the startup scripts, and there is a script you can add - /etc/rc.custom_boot_early (make sure you chmod a+x on it). That seems to be a reasonable place for renaming interfaces, so I created it, and added, e.g.:
#!/bin/sh /sbin/ifconfig em0 name igb0 /sbin/ifconfig em1 name igb1
That seems to have done the trick for me. Clearly, I need to save this script and re-apply it on upgrades and such.
I have no idea what the official intent of this script is, but it would be nice if Netgate would formalize something like this - even if the onus is on us to maintain it across upgrades. To be clear, this appears to be part of pfSense, and not part of FreeBSD.