OSPF+GRE+CARP+IPSEC for a resilient multi-wan connection: IPSEC troubles

[trantorvega]

Hello to all.
Despite the title, the purpose of the post is at present geared towards the ipsec configuration that prevents me from reaching a successful completion of the configuration in the subject.
I shall begin by saying that it is actually a configuration tested several weeks ago, because different priorities have occupied me since then.

---

A brief explanation of my target configuration.
A) Two endpoints, each a clustered pfSense installation with CARP enabled between two hosts.
B) Multiple uplink connectivities at each endpoint.
C) Several L3 subnets behind each of the endpoints.
D) Seamless (or as near as I can get) transition from one connectivity to any other one on each endpoint.

I nearly reached all my objectives with a layered solution.

From here onwards I shall call the endpoints A and Z respectively, while A1 and A2, Z1 and Z2 will be the individual components of every clustered installation.

1. IPSEC in transport mode enables me to encrypt the traffic between any two public IPs (CARP IPs) of A and Z (theoretically, see below).

2. CARP allows for all of the aforementioned public IPs to be moved between A1 and A2, or between Z1 and Z2.

3. GRE tunnels, one for each public CARP IP combination between A and Z (mesh-like), give me many concurrent paths between A and Z to route traffic into. (Each external IP GRE endpoint is a CARP address. I submitted a patch to allow this from the web interface some time ago.) These tunnels must exist and be identically configured on both A1 and A2 for A and Z1 and Z2 of Z.
CARP status will determine if the CARP external IP of the GRE tunnel will be active on A1 or on A2, for example.
I tried to use GIF tunnels rather than GRE ones, but the encapsulation/decapsulation mechanisms between endpoints seem to go awry with multiple uplinks and CARP IPs.

4. OSPF router instances on A1, A2, Z1, Z2. This is a bit trickier and I won't go into details about which interface/subnet must be excluded from redistribution to the FIB etc., mostly because I haven't touched it in weeks. Basically, the OSPF daemons on A and Z select which GRE tunnel use among those available, switching to another when the path fails, for whatever reason. I managed to shorten the time needed to shift paths to a reasonable amount, making the whole circus work.

---

Now for the trouble. It's mainly a pfSense-related issue, not a FreeBSD one. It is related to the routing of ipsec traffic.
For reasons that I believe related to tunnel-mode-only IPSEC support of previous releases, whenever I configure transport-mode IPSEC between the IP address A.B.C.D of A and the address E.F.G.H of Z, I am then prevented  by the interface from creating another ««association»» (I know, wrong term) between another IP of A and the same address E.F.G.H of Z. Looking at the php code it would appear that creating a first IPSEC SA between A.B.C.D and E.F.G.H automatically creates a static route redirecting traffic for E.F.G.H through the A.B.C.D interface. That is connected to the interface lock described above, and it makes somewhat sense with a tunnel-mode-only configuration.
I am just wondering whether a redirection rule at pf level would serve the same purpose, perhaps allowing the user to choose whether to enable it or not (I think there are some lines of comment in the code on that subject).

While I realize that the configuration I described here is not something most of pfSense users would find useful, however I believe that in decentralized infrastructure the needs I have enumerated are real and in need of attention.

Thanks to everyone for any comment on the issue.  8)

[Stephane]

Hi,
It seems we try to do the same kind of design.
I'm doing my lab on GNS3 VirtualBox Edition.
The first part of my test was :
3 sites  meshed with GRE over IPSec (Transport) and OSPF as routing protocol.
Everything is ok.
Now, I'm trying to configure 2 pfsense per site but still 1 router per site.
CARP is running (LAN+WAN)and I have  temporaly disabled the IPSEC. The GRE is up and linked to the remote WAN_CARP.The second pfsense node become master if I shutdown the 1st pfsense node. The problem is when I restart the 1st pfsense node. Both PFsense stay master !!

Modification :
In fact I've found out that this problem is related to disabling and re-enabling CARP on the Master.

Stephane