HA Setup with Multi-WAN and DHCP Guide

[vwaniel]

I'm posting this on the offhand chance that someone will get some use out of it....

I recently added a second Internet connection to my home network and configured pfSense WAN failover for redundancy. Around the same time, I started researching pfSense’s high availability features. Both of my Internet providers only offer dynamic IP addresses via DHCP, so I needed to be able to fail over without static IP addresses on any of the WAN interfaces. I found a couple of useful tutorials that I’d like to credit for helping me figure out what I’m posting:
• https://lieven.kks36.be/2022/07/03/pin-dhcp-wan-interface-to-lan-for-pfsense-carp-ha/
• https://forums.servethehome.com/index.php?threads/hacked-up-pfsense-ha-setup-for-home.30461/
• https://www.reddit.com/r/PFSENSE/comments/f2xhdi/carp_status_change_doesnt_seem_to_run_custom/
The links above were a good resource to get started, but I had a few challenges. My home pfSense setup is fairly complex – multiple WAN connections, multiple VLANs/subnets, OpenVPN (site-to-site, remote access, and client), OSPF, DHCP relay/failover, etc. Everything I could find online regarding DHCP WAN failover how-to seemed to be enough to theoretically make it work with a basic setup but not enough to make DHCP WAN failover work consistently and reliably to the point that I would be confident everything would work if I was traveling and my primary firewall failed. I have spent the last couple of months testing and tuning, and want to share what I’ve learned. I am running pfSense 2.7.2 as of this writing.

Hardware
I’m using two physical appliances based off the Supermicro A1Sri-2558F platform. Each appliance has a dual port 10 GbE PCIE NIC installed which is set up in an active/passive LAGG and connected to a pair of 10 GbE switches. The LAGG handles all internal VLANs/subnets as well as HA heartbeat. Each WAN provider’s gateway/ONT is connected to a separate unmanaged 1 GbE switch, which is also attached to one of the onboard 1 GbE ports on each node.

HA Setup
Getting basic HA up and running in pfSense is pretty straightforward and there are pretty good tutorials out there for how to do so. I won’t go into extreme detail about how to set it up, but suffice to say you’ll need a primary and a secondary node. On the primary node, under System > High Availability, configure both pfsync (which synchronizes firewall states from the primary to the secondary) and XML-RPC sync (which synchronizes configuration changes from the primary to the secondary). It is very important to configure interfaces on the secondary with the same names and in the same order as the primary node (so that OPT1 on the secondary matches OTP1 on the primary, etc). CARP VIPs will need to be configured on each local subnet and the VIP needs to be set as the gateway for clients on each subnet. For example, if I have a 192.168.1.0/24 subnet, I would make the primary node 192.168.1.2/24, the secondary node 192.168.1.3/24, and a CARP VIP of 192.168.1.1/24. I would configure DHCP to hand out 192.168.1.1 as the default gateway for the subnet. Use of CARP is essential because it both provides a monitorable system to the firewall (in order to trigger custom actions based on status) as well as providing a single IP address that clients use to access the system regardless of which HA partner is active (i.e. failover is seamless).

Adding WAN DHCP
The goal is to be able to fail the WAN connection over from one node to the other and maintain the same IP address so that firewall states will fail over transparently. In order to accomplish this, the WAN interface on both the primary node and the secondary node will need to have the same MAC address. This is easy to set up in the interface settings – just clone the MAC of the primary node’s interface to the secondary. Additionally, I would manually configure the DHCP client hostname and make sure that matches between both the primary and secondary. Once that is done, in theory, DHCP requests should look virtually identical from either system, so the ISP would have no idea (other than a gratuitous DHCP request) that the failover had occurred.

Automating Failover Steps
pfSense uses a BSD daemon called “devd” which monitors kernel output and triggers actions if certain events are logged out of the kernel. CARP events get logged to kernel output so devd is ideal for triggering actions when a CARP status change takes place. pfSense stores its devd configuration in /etc/pfSense-devd.conf. It is also supposed to read in any supplemental configuration files in /usr/local/etc/devd (more on this later). We need to set up a configuration file to trigger custom actions whenever a CARP status change takes place. This is what mine looks like:

notify 0 {
    match "system"          "CARP";
    match "subsystem"       "1@lagg0.3";
    match "type"            "MASTER";
    action "/bin/sh /root/ha_customconfig.sh master; /usr/local/sbin/pfSctl -c 'interface carpmaster '$subsystem";
};

notify 0 {
    match "system"          "CARP";
    match "subsystem"       "1@lagg0.3";
    match "type"            "BACKUP";
    action "/bin/sh /root/ha_customconfig.sh secondary; /usr/local/sbin/pfSctl -c 'interface carpbackup '$subsystem";
};

notify 0 {
    match "system"          "CARP";
    match "subsystem"       "1@lagg0.3";
    match "type"            "INIT";
    action "/bin/sh /root/ha_customconfig.sh secondary; /usr/local/sbin/pfSctl -c 'interface carpbackup '$subsystem";
};

Notice the “subsystem” line. The match is “<CARP_VHID_GROUP>@<REAL_INTERFACE>”. I have around 10 VLANs/subnets with CARP VIPs configured and if I did not define the subsystem, my script would get triggered 10 times during failover which is excessive. I only want it to be triggered once so I am filtering to only trigger on one specific CARP VIP. In each case I’m calling a script I wrote in /root/ha_customconfig.sh, and I’m feeding it parameters based on the status of the node (“master” vs. “secondary”).

Don’t forget to restart devd (service devd restart) after creating the configuration file so that devd will load it. Note that you may have to create the “devd” directory in /usr/local/etc.

The Basic Script

If the node is primary, the script needs to bring up WAN interfaces. If the node is secondary, bring down the WAN interfaces. We’ll add more things later, but here is the basic framework for the script:

#!/bin/sh

WAN_A_REALIF=”igb2”
WAN_A=”wan”
WAN_B_REALIF=”igb3”
WAN_B=”opt1”

if [ “$1” == “master” ]; then
     /sbin/ifconfig $WAN_A_REALIF up
     /sbin/ifconfig $WAN_B_REALIF up
     /usr/sbin/service netif start $WAN_A_REALIF
     /usr/sbin/service netif start $WAN_B_REALIF

     # Wait 2 seconds for interface up to stabilize
     /bin/sleep 2s

     # Start dhclient
     /sbin/dhclient -c “/var/etc/dhclient_$WAN_A.conf” -p “/var/run/dhclient.$WAN_A_REALIF.pid” $WAN_A_REALIF
     /sbin/dhclient -c “/var/etc/dhclient_$WAN_B.conf” -p “/var/run/dhclient.$WAN_B_REALIF.pid” $WAN_B_REALIF

     # Call new WAN IP notification
     /etc/rc.newwanip $WAN_A_REALIF
     /etc/rc.newwanip $WAN_B_REALIF
elif [ “$1” == “secondary” ]; then
     # Bring WAN interfaces down
     /usr/sbin/service netif stop ${WAN_A_REALIF}
     /usr/sbin/service netif stop ${WAN_B_REALIF}
else
     /bin/echo “Must specify master/secondary role.”
     exit 1
fi

One thing I still need to test is if I actually need to call “ifconfig” AND “service netif start” or if I can just call “service netif start”. At this point, there is enough configuration that testing CARP should successfully shut down the WAN interfaces on the primary node and bring them up on the secondary. However, we are far from done.

DHCP Lease Expiration

Whenever pfSense (really dhclient) requests a DHCP lease, it gets stored in /var/db/dhclient.leases.<INTERFACE_NAME>. When dhclient starts, it checks the lease database file to see if it has a valid lease for the interface. Even if it doesn’t, it will try to request the same IP address as is in the lease file (a.k.a. the “preferred” IP address). This can be problematic if one node fails over to the other, some amount of time passes in which the DHCP lease expires and a new IP address is issued, and another failover occurs. If the old IP address (left over in the lease database file) is requested and happens to be available the DHCP server may issue it to the WAN interface, which defeats our efforts to ensure that the same IP address gets leased to the active node during a failover. In order to prevent this from happening, it is best to delete the lease database files when the node is no longer primary:

[ -f /var/db/dhclient.leases.$WAN_A_REALIF ] && /bin/rm /var/db/dhclient.leases.${WAN_A_REALIF}
[ -f /var/db/dhclient.leases.$WAN_B_REALIF ] && /bin/rm /var/db/dhclient.leases.${WAN_B_REALIF}

It may be unnecessary, but for good measure I also remove the cached IP addresses for the WAN interfaces:

[ -f /var/db/$WAN_A_REALIF_ip ] && /bin/rm /var/db/${WAN_A_REALIF}_ip
[ -f /var/db/$WAN_A_REALIF_cacheip ] && /bin/rm /var/db/${WAN_A_REALIF}_cacheip
[ -f /var/db/$WAN_B_REALIF_ip ] && /bin/rm /var/db/${WAN_B_REALIF}_ip
[ -f /var/db/$WAN_B_REALIF_cacheip ] && /bin/rm /var/db/${WAN_B_REALIF}_cacheip

[ -f /tmp/$WAN_A_REALIF_defaultgw ] && /bin/rm /tmp/${WAN_A_REALIF}_defaultgw
[ -f /tmp/$WAN_A_REALIF_router ] && /bin/rm /tmp/${WAN_A_REALIF}_router
[ -f /tmp/$WAN_B_REALIF_defaultgw ] && /bin/rm /tmp/${WAN_B_REALIF}_defaultgw
[ -f /tmp/$WAN_B_REALIF_router ] && /bin/rm /tmp/${WAN_B_REALIF}_router

Rebooting

If a node is rebooted, there are a couple of problems that I have observed. Firstly, for some reason, when devd is initially started during boot it does not honor supplemental configuration files in /usr/local/etc/devd. This is problematic because a freshly rebooted system will not correctly execute the failover script. Fortunately this is easy to solve, just install the “shellcmd” package and set up shellcmd to run “service devd restart”.

Additionally, this means that our custom HA script is not going to get triggered when the system starts up. In order to make sure that it does, I placed the following script in /usr/local/etc/rc.d/:

#!/bin/csh

set carp_status=`/sbin/ifconfig lagg0.3 | /usr/bin/grep ‘carp:’ | /usr/bin/awk ‘{print $2}’`
if ( “$carp_status” != “MASTER” ) then
     /root/ha_customconfig.sh secondary
endif

Be careful with putting things in rc.d, however, because they can get triggered at times other than system startup, like on certain interface/gateway state changes. However, this works to our advantage because we want to make sure that if the node is secondary, the WAN interfaces do not inadvertently get turned on. Also note that I am once again monitoring a single CARP VIP (on lagg0.3).

OpenVPN

I’ve found that OpenVPN can have issues if it’s left running on the passive node. OpenVPN’s configuration file is supposed to include the IP address where it is supposed to listen (local x.x.x.x), and the configuration file gets generated by pfSense whenever the OpenVPN service is restarted (or certain gateway changes occur if OpenVPN is configured to use a gateway group). If the listening interface does not have an IP address when the configuration file gets generated, the “local x.x.x.x” line is not included in the configuration file and OpenVPN tries to listen on all interfaces. This could be problematic if OpenVPN is configured to listen on 443/tcp and webConfigurator is also on 443. Like I said, I find that it’s best to shut off OpenVPN services on the passive node. You’ll need to look at the OpenVPN interface name. OpenVPN servers will begin with “ovpns”, and clients will begin with “ovpnc”. For example, if I want to stop ovpns4, I would run:

/usr/local/sbin/pfSsh.php playback svc stop openvpn server 4

And to start it:

/usr/local/sbin/pfSsh.php playback svc start openvpn server 4

If you do want to have webConfigurator and OpenVPN both functionally available on 443/tcp I would suggest running OpenVPN on a different port and using destination NAT rules to match traffic coming into 443/tcp on the WAN interfaces and redirect it to the actual OpenVPN port.

Passive Node Internet Access

Another issue is that whichever node is passive is not going to be able to access the Internet. This makes package and system updates problematic, and also makes the dashboard slow to load if the system is configured to check for updates. I struggled for quite a while to solve this one – others had suggested setting one of the VIPs as a gateway in the system and building a gateway group with it available as a tier 5 gateway. That way, if both WAN connections were unavailable to a node, eventually it would use the VIP as its default gateway. However, this doesn’t work in practice because the CARP IP is always hanging off of the system’s interface as an alias, so accessing it locally from a node is essentially the same as a local loopback.

What I ended up doing is setting up an IP alias on each node that that is used as the gateway monitor IP address for the node. So, for example, if my nodes are 192.168.1.2/24 and 192.168.1.3/24 (with a CARP VIP of 192.168.1.1/24), I would set up an IP alias of 10.100.1.2 on the 192.168.1.2 node and 10.100.1.3 on the 192.168.1.3 node. Both 192.168.1.2 and 192.168.1.3 would be configured as gateways in the system, and 10.100.1.2 and 10.100.1.3 would be used as gateway monitor IPs, respectively. This way, each node would attempt to ping 10.100.1.2 via 192.168.1.2, and 10.100.1.3 via 192.168.1.3. All I need to do is have my script add the IP alias when the node is primary, and remove it when the node is secondary. Easy enough:

$GW_MON_IP=”10.100.1.2”
$LAN_REALIF=”lagg0.3”
/sbin/route delete $GW_MON_IP
/sbin/ifconfig $LAN_REALIF inet $GW_MON_IP netmask 0xffffffff alias

And to remove it:

/sbin/ifconfig ${LAN_REALIF} delete ${GW_MON_IP}

From there, I built a gateway group with my primary WAN as tier 1, secondary WAN as tier 2, and the node gateways both as tier 5, and made that the default gateway for the system. Now, whenever a node is passive it uses the active node as its gateway so updates work perfectly, and it is possible for VPN clients to access the passive node through the VPN tunnel on the active node without having to get create with NAT rules.

Dynamic DNS

The next issue I found was that sometimes during failover the wrong WAN IP address would end up getting updated with my dynamic DNS provider, but the correct IP address would remain in cache. This would affect my ability to VPN into my system, so I wanted to force a dynamic DNS update on failover. I added a 60 second wait before doing so just to make sure that WAN connections had stabilized.

/bin/sleep 60s
/bin/rm /cf/conf/dyndns*
/etc/rc.dyndns.update

Putting it Together

With all of the issues identified and fixed, /root/ha_customconfig.sh should look something like this:

#!/bin/sh

WAN_A_REALIF="igb2"
WAN_B_REALIF="igb3"
LAN_REALIF="lagg0.3"

WAN_A="wan"
WAN_B="opt1"

GW_MON_VIP="10.100.1.2"

if [ "$1" == "master" ]; then
        /bin/echo "$1"

        # Bring WAN interfaces up.
        /sbin/ifconfig $WAN_A_REALIF up
        /sbin/ifconfig $WAN_B_REALIF up
        /usr/sbin/service netif start $WAN_A_REALIF
        /usr/sbin/service netif start $WAN_B_REALIF

        # Wait 2 seconds
        /bin/sleep 2s

        # Start dhclient
        /sbin/dhclient -c "/var/etc/dhclient_$WAN_A.conf" -p "/var/run/dhclient.$WAN_A_REALIF.pid" $WAN_A_REALIF
        /sbin/dhclient -c "/var/etc/dhclient_$WAN_B.conf" -p "/var/run/dhclient.$WAN_B_REALIF.pid" $WAN_B_REALIF

        # Add HA peer monitor IP
        /sbin/route delete $GW_MON_VIP
        /sbin/ifconfig $LAN_REALIF inet $GW_MON_VIP netmask 0xffffffff alias

        # Call new WAN IP notification
        /etc/rc.newwanip $WAN_A_REALIF
        /etc/rc.newwanip $WAN_B_REALIF

	# Start OpenVPN services.
	/usr/local/sbin/pfSsh.php playback svc start openvpn client 1
	/usr/local/sbin/pfSsh.php playback svc start openvpn server 1

	# Wait 60 seconds for WAN stabilize.
	/bin/sleep 60s

	# Force Dynamic DNS update.
	/bin/rm /cf/conf/dyndns*
	/etc/rc.dyndns.update

elif [ "$1" == "secondary" ]; then
        /bin/echo "$1"

	# Stop OpenVPN services.
        /usr/local/sbin/pfSsh.php playback svc stop openvpn client 1
	/usr/local/sbin/pfSsh.php playback svc stop openvpn server 1

	# Bring WAN interfaces down
        /usr/sbin/service netif stop ${WAN_A_REALIF}
        /usr/sbin/service netif stop ${WAN_B_REALIF}

        # Remove dhclient leases
        [ -f /var/db/dhclient.leases.$WAN_A_REALIF ] && /bin/rm /var/db/dhclient.leases.${WAN_A_REALIF}
        [ -f /var/db/dhclient.leases.$WAN_B_REALIF ] && /bin/rm /var/db/dhclient.leases.${WAN_B_REALIF}

	# Remove cached IPs
	[ -f /var/db/$WAN_A_REALIF_ip ] && /bin/rm /var/db/${WAN_A_REALIF}_ip
	[ -f /var/db/$WAN_A_REALIF_cacheip ] && /bin/rm /var/db/${WAN_A_REALIF}_cacheip
	[ -f /var/db/$WAN_B_REALIF_ip ] && /bin/rm /var/db/${WAN_B_REALIF}_ip
	[ -f /var/db/$WAN_B_REALIF_cacheip ] && /bin/rm /var/db/${WAN_B_REALIF}_cacheip

	[ -f /tmp/$WAN_A_REALIF_defaultgw ] && /bin/rm /tmp/${WAN_A_REALIF}_defaultgw
	[ -f /tmp/$WAN_A_REALIF_router ] && /bin/rm /tmp/${WAN_A_REALIF}_router
	[ -f /tmp/$WAN_B_REALIF_defaultgw ] && /bin/rm /tmp/${WAN_B_REALIF}_defaultgw
	[ -f /tmp/$WAN_B_REALIF_router ] && /bin/rm /tmp/${WAN_B_REALIF}_router

        # Remove HA peer monitor IP
        /sbin/ifconfig ${LAN_REALIF} delete ${GW_MON_VIP}

else
        /bin/echo "Must specify master/secondary role."
        exit 1
fi

Bonus: DHCP Relay/Failover

This is very much a corner case due to how I use DHCP. I do not use pfSense as a DHCP server but only as a DHCP relay. I have two DHCP servers (one local and the other at a remote site) that are configured in a failover group. The local and remote sites are connected via an OpenVPN site-to-site tunnel, and I am running OSPF between the two pfSense instances on either end of the tunnel to exchange route information. Not your typical home setup.

During failover, pfSense is smart enough to automatically stop dhcrelay on the (now) passive node and start it on the (now) active node. However, this occurs before the OpenVPN tunnel re-establishes, and more importantly, before OSPF convergence is achieved. When dhcrelay starts, pfSense looks at the two DHCP server IP addresses that are configured. One of them is available locally. The other is not, and pfSense does not yet have a route to it (because OSPF hasn’t yet converged), so it assumes that the IP is accessible via the system’s default gateway (a.k.a the Internet) and starts dhcrelay with one of the WAN interfaces being used as an upstream interface. Needless to say, this is not good.

I fixed this by installing the “cron” package and running a script every 5 minutes that checks for the presence of the WAN interface names in the running dhcrelay command. If found, dhcrelay is restarted.

#!/bin/csh

set wan_if=( igb2 igb3 )

foreach i ( $wan_if )
	if ( `/bin/ps aux | /usr/bin/grep dhcrelay | /usr/bin/grep -v grep | /usr/bin/grep $i` != "" ) then
		/bin/echo "Found WAN interface $i"
		# Restart dhcrelay service
		/usr/local/sbin/pfSsh.php playback svc restart dhcrelay
	else
		/bin/echo "Did not find WAN interface $i"
	endif
end

Final Thoughts

If you’ve stuck around this long, thank you for reading. I am by no means a coder so there are probably much better and more elegant ways to write the custom scripts I’m using. And I’m sure there are probably other edge/corner cases I haven’t thought about so I’d love to hear feedback from anyone else that has tried to get HA and DHCP WAN working and keep the conversation going. Thanks!

[ulope]

@vwaniel Awesome! Thanks for taking the time to write this up.