FRR BGP, IPsec VTI multi site and remote gateway routing

AlexM 0

First post and first time setting up BGP and IPsec VTI, though I've been familiar with the concepts before.

The setup is 3 sites - one being a cloud instance and the two other being offices.
Both offices are connected to the cloud instance via IPsec IKEv2 VTI.
Above that I've setup BGP so I won't have to keep updating the routing table on all sites (maybe OSPF was a better choice, but I've seen reports of it misbehaving in pfSense 2.5.x).
Traffic between sites, cloud, ipsec mobile clients works fine.
FRR prefix is set to "any" and no route maps or ACLs are configured.
Each site has its own AS and each office has only the cloud instance as the neighbor.

The issue I'm facing at this point seems like a known limitation, however I've barely found any documentation about it and barely any possible workarounds - When I want to route traffic from Site A/B via the cloud WAN the packets don't ever come back unless I configure a static route pointing back to Site A/B on the cloud pfsense.
When looking at the routing table I see the BGP published route back to Site A, however without an explicit static route the setup doesn't work.
If I set BGP to redistribute kernel/pfsense routes (and not just FRR static & connected networks), then configure the static route on cloud back to Site A - I can also route traffic to Site B's WAN (from Site A) without adding another route on Site B, however any attempts at configuring this route on Site A don't seem to work.

I did find the following:
"VTI on pfSense does not support reply-to. Despite this policy routing rule on Peer1 which covers all traffic, there must also be kernel routes to remote LANs for the return traffic to find the way back." Source article
The setup in the above article is pretty similar to what I have.

The question - are there any workarounds to configuring these static routes on the central (cloud) pfSense if I want to redirect specific traffic (via FW rules or FRR routes) to remote GWs (cloud or Site B)?

I played around with trying to publish the same route from Site A via FRR routes but that didn't do the trick, neither did setting FRR routes on the cloud instance help - only pfsense static routes.
In a "perfect world" I'd like to have the ability to have each site govern its own routes (as far as the adjacent GWs) without having to explicitly configure the adjacent GWs each time (Ex. Route 8.8.8.8 from Site A to Cloud WAN without having to explicitly add a route on cloud).

So am I doing something wrong to begin with, did I miss something crucial in my googling efforts or is this just the reality I'll have to deal with?

AlexM 0

Quick update:
After writing this all down my mind drifted into another direction and I figured out that a possible workaround is manually configuring the outbound NAT - as it is set to automatic.
But this again brings me back to either configuring manual routes and having the outbound NAT set to auto or manually configuring the NAT and having BGP distribute the routes.

Any best practices out there or voodoo magic to make this work automatically?

AlexM 0

Update 2:
Added an alias for RFC1918 networks and configured an outbound NAT rule with RFC1918 as source and any destination on all pfSenses.
This solved what seemed like a routing problem but turned out to be a NATing problem.
However I'll probably have issues if/when I have multiple WAN connections.
Still would like to hear if there are any best practices.