HA on two different types of hardware

matthewdaniels

Apologies if this has been posted somewhere else, we were hoping use Pfsense on two different VMware hosts or a host and a physical desktop.

But I've noticed the comment in the documentation "Both cluster nodes are the same model with identical hardware specs."

I know it seems pretty clear by that statement that it relies on everything being identical, but has anyone tested this? Or is it just the case that the NIC's need to be the same hardware type etc.

Cheers

viragomann

@matthewdaniels
Identical network hardware is primarily required for synchronize states to enable seemless failover.
If you want to use that feature on different hardware you can insert an abstraction layer with LAGG interfaces so that LAGG0 is WAN on both boxes, LAGG1 LAN and so on. So pfSense in HA will synchronize the states on the base of the LAGGs, which are identical at least.

SteveITS

per https://docs.netgate.com/pfsense/en/latest/highavailability/pfsync.html#pfsync-and-physical-interfaces "States in pfSense are bound to specific operating system Interfaces. For example, if WAN is em0, then a state on WAN would be tied to em0. If the cluster nodes have identical hardware and interface assignments then this works as expected. In cases when different hardware is used, this can be a problem. If WAN on one node is em0 but on another node it is igb0, the states will not match and they will not be treated the same."

If they are different the HA will failover but connections will drop since the state doesn't exist. LAGG will get around this as noted but as I also found out, one can't use traffic shaping on LAGG without a VLAN.

Edit: other than the interface names I don't think the rest of the hardware matters.

MushyMiddle

Hi. I recently posted a similar question on a different (old) thread, and was going to re-post here, but you saved me the trouble. In my case, I have a physical box in my data center that I want to use HA and/or test using a VM as backup.

FreeBSD does support renaming physical interfaces, so you can rename em0 to igb0, for instance. The problem seems to be that there is no supported way to make the interface renames persistent.

I'm investigating which rc* file I can hack to include the renames, which it seems to me would need to happen at the correct moment:

Before any other pfSense services were started...
But at the point at which the physical interfaces are available to be renamed.

And, of course, I'd have to make sure that script survives an upgrade/replacement, but that's on me.

Clearly, more than one person is interested in the ability to rename interfaces. The documented workaround, fake LAGGs, seem a bit bogus, but if all else fails, I'll try that. But, it also means you would have to implement that on both the primary and the secondary machines, making the pfSense config different on the two (vs. pushing the problem down to the OS level, leaving the primary box config pristine, at least).

Ideally, I want the config to be identical on the primary/secondary, knowing full well that there may be differences, particularly in the non-HA case, where you will need to have different interface addresses, e.g.

At a minimum, can someone document a non-LAGG workaround for this? That is, a semi-supported way to persist interface renaming. I see at least two use cases for this:

HA on different hardware (or in a VM)
Side-by-side testing of new releases or config changes
(Backup support, in general, absent HA, so shared config can be used)

It just seems like interface renaming, if properly supported, would be simpler than the fake LAGG workaround.

Thanks!

MushyMiddle

FWIW, I looked through the startup scripts, and there is a script you can add - /etc/rc.custom_boot_early (make sure you chmod a+x on it). That seems to be a reasonable place for renaming interfaces, so I created it, and added, e.g.:

#!/bin/sh

/sbin/ifconfig em0 name igb0
/sbin/ifconfig em1 name igb1

That seems to have done the trick for me. Clearly, I need to save this script and re-apply it on upgrades and such.

I have no idea what the official intent of this script is, but it would be nice if Netgate would formalize something like this - even if the onus is on us to maintain it across upgrades. To be clear, this appears to be part of pfSense, and not part of FreeBSD.

TechSSH

Just an update:

from the pfsense site:

https://docs.netgate.com/pfsense/en/latest/highavailability/pfsync.html#pfsync-and-physical-interfaces

"This is no longer the case on pfSense Plus software version 22.01 and later and pfSense CE software version 2.6.0 and later. On these versions, the states are no longer bound to interfaces in the way described in this section."

It still needs to keep the interface assigned identically:

https://docs.netgate.com/pfsense/en/latest/highavailability/settings.html#configuration-synchronization-settings-xmlprc-sync

"The interfaces on both nodes must be assigned identically, for example: wan=WAN, lan=LAN, opt1=Sync, opt2=DMZ. Check the config.xml contents directly to ensure a match.

If the interfaces do not match up exactly, firewall rules and other configuration items will appear to synchronize to the wrong interface on the secondary node. Additionally, this can also lead to failures in DHCP failover."

ank99

@TechSSH Could you (or someone) please clarify what is meant by "The interfaces must be assigned identically on both nodes, for example: wan=WAN, lan=LAN, opt1=Sync, opt2=DMZ. Check the config.xml contents directly to ensure a match."?
How can we rectify this condition if there is a mismatch?
Is it possible to manually edit the conifg.xml file in order to rectify this situation? ?
Is there a documented procedure that will allow me to reassign interfaces and still keep them "identically assigned" on both nodes?

In my case I opted to reassign the LAN and WAN interfaces to different hardware (a better adapter). This has created a mismatch of LAN/WAN and OPTx numbers. Obviously the HA config sync is out of whack messing up my firewall rules etc.
Thank you for your help

SteveITS

@ank99 said in HA on two different types of hardware:

what is meant by "The interfaces must be assigned identically on both nodes, for example: wan=WAN, lan=LAN

There are 3 things that show up in the config file:

the hardware interface, e.g. igb0
the logical interface, e.g. opt2
the visible/GUI name of the interface, e.g. opt2 is named GUEST_VLAN

If you are careful about what you're doing you can edit the config .xml file and change the assignments. For instance when we restored a config this summer to a Netgate 4860, it has extra interfaces but the Hurricane Electric IPv6 tunnel was assigned during the restore as opt1. Therefore the interfaces which would normally have been opt1-6 out of the box would have been opt2-7 once assigned. I edited the config to change the HE tunnel interface to opt10 and the others would be opt1-6.

<interfaces>
<lan>
<if>igb0</if>
<enable/>
<descr>LAN</descr>
<spoofmac/>
<ipaddr>YYYYY</ipaddr>
<subnet>24</subnet>
<ipaddrv6>XXXXX</ipaddrv6>
<subnetv6>64</subnetv6>
</lan>
<opt1>
<descr>OPT1_LAB</descr>
<if>igb2</if>
<spoofmac/>
<enable/>
<ipaddr>ZZZZ</ipaddr>
<subnet>24</subnet>
</opt1>
<interfaces>

ank99

@SteveITS Thank you very much. I appreciate immensely your input. It clarified my misunderstanding.