FRR OSPF with HA/CARP and multi-wan
-
My end goal is to get routable IPSEC tunnels working between multiple sites. We currently have HA and multi-WAN working with static IPSEC tunnels between all of them. However, if the tunnel between site 1 and site 2 goes down, the link doesn't reroute using both sites' connections to site 3.
In my lab I am having trouble getting FRR OSPF to work with HA and multi-WAN. I've never dealt with OSPF before, so bear with me please. My lab environment is set to have 2 routers configured with HA in site 1 with 2 WAN connections, and 1 router in site 2 with 1 WAN connection currently (I will add a 2nd for my next stage of testing). I followed the Routed IPSEC (VTI) instructions at https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/ipsec-routed.html and installed FRR. The VTI was set up to use a /30 subnet. After much fiddling I finally got OSPF to see its neighbor and update routing tables over the first WAN connection. At that point I had CARP configured but had not yet defined an IPSEC connection through WAN2.
When I configured IPSEC phase 1 connection through WAN2 (also using CARP of course) it was fine. When I configured a Phase 2 connection using another /30 subnet, the IPSEC tunnels started bouncing each other. Once phase 2 was established over WAN2, then WAN1's IPSEC connection would drop. Not long after, WAN1 would re-establish its IPSEC connection and WAN2's IPSEC connection would drop, and the cycle would repeat.
I removed WAN2's VTI and IPSEC configurations and got traffic back between the 2 "sites". After turning off the FRR option to redistribute connected networks (which I think is what caused the problems before), I recreated the WAN2 tunnel and VTI. WAN1's tunnel went down and WAN2's came up, then WAN1 came back up and WAN2 went down and stayed down. OSPF routes traffic over the functional WAN1 tunnel.
What do I need to do to get OSPF to work over both site1 WAN connections? We want relatively quick failover if WAN1 dies.