IPv6 HE Tunelbroker Tunnels in a pfSense HA availability configuration
-
Hello,
We have a high-availability pfSense system using a Multi-WAN configuration. The system has been operating properly during the past years supporting IPv4 only.
Recently, we were presented of a requirement to IPv6-enable our network. After consulting our ISPs (different for each WAN connection), we decided to use your TunnelBroker service by registering one tunnel for each WAN (VDSL) connection we have, i.e. two tunnels. As both IPv4 addresses obtained by the ISPs are static, the creation of the tunnels was pretty straightforward by following the instructions at https://docs.netgate.com/pfsense/en/latest/interfaces/using-ipv6-with-a-tunnel-broker.html .
However, as our system is of a High Availability nature (two pfSense units), we are presented of the following issue: As per the HE tunnel specifications, one host has the ::1 link address (remote side) and the other (client) host has the ::2 address (local site). As there are two pfSense hosts, only one of them can establish HE tunnels as its client address (::2) must be declared within the GIF tunnel details and the other host must have a different IP address (e.g. ::3). It is not possible for both pfSense boxes to have the same Client IPv6 address (correct?).
The result is that pfSense host A connects to HE Tunnel #1 and pfSense host B connects to HE Tunnel #2. Within each other, the gateway that corresponds to the respective connected tunnel is online and the other is offline (see attached image). It is not possible for both pfSense hosts to connect to both tunnels at the same time. The internal LAN addresses are of fd08:: numbering and they access external IPv6 hosts by performing NPt mapping of HE Tunnel #1/#2 /48 networks as per the respective GIF interface.
The issue of only one of the two pfSense boxes connecting to each IPv6 tunnel affects IPv6 incoming traffic to hosts behind NPt (and Load Balancing on a WAN-level) as the internal hosts select as a default gateway one of the two pfSense boxes. Therefore, if someone wants to reach a host behind NPt which has a default gateway pfSense box A connected to HE Tunnel #1 (#2 is shown as offline) from the internet, this can be performed only using a remote IPv6 address belonging to (the connected) HE Tunnel #1. If we try to reach the host to its HE Tunnel #2 address, it times out EVEN THOUGH the IPv6 address space is active through pfSense box B and (theoretically, as it is properly NPt mapped) should reach the host behind the mapping. As the latter (the host) has selected pfsense box A as a default gateway and not B from the list of available gateways, it times out. Last but not least, if we were using only one HE tunnel @ one of the WAN connections, then one of the pfSense boxes (the one of the offline tunnel) would have no IPv6 connection whatsoever, inbound or outbound, as the relevant gateway would be marked as down.
Which is the proper configuration for using a HE tunnel in a pfSense High-Availability (i.e. two pfSense boxes) scenario so that both boxes have IPv6 access despite the fact that the HE Tunnel (in case of a 1-WAN scenario) is down?
-
Unfortunately there is no good way to do a proper tunnel of that nature with HA. Normally you might have an ability to shut down a tunnel on whichever system is the backup node at the time, but that can't be done for GIF tunnels like it can for OpenVPN/IPsec/etc.
Your best bet is a small VM/pfSense box/etc on the WAN side or a dedicated interface setup just to handle the HE tunnel. Use the first /64 they give you between that box and the HA cluster, and then subnets out of the /48 on the inside of the HA cluster (LAN, etc).
It does add a SPOF, but like most CPE devices, it's a SPOF no matter what you do. And arguably since you can still "fall back" to IPv4 it's not quite as critical.
-
@jimp Thank you for your reply!
I have performed several tests regarding failover and I had concluded to the same thing.. the HE tunnels should be controlled elsewhere, outside the pfSense cluster.
I have resolved the CPE SPOIF issue by using two circuits from two different ISPs using respective CPEs, hence the only SPOF would be the VM that will control the HE Tunnels.
Is this a limitation of the current version of pfSense, i.e. the inability to failover GIF tunnels, or a more general limitation of tunnels of this nature?
-
Primarily pfSense, I believe, since there is no mechanism for OS-level tunnel interfaces to have any kind of defined failover behavior. It would have to disable/deconfigure the tunnel interface on an HA backup, which could be more problematic for gif/gre than it is for some like OpenVPN.
-
@jimp If states are not to be preserved, then a disable/enable (via a heartbeat mechanism or otherwise) might do the trick.. of course with a disruption of the IPv6 connectivity while the tunnel is re-establishing itself.
