@pete35 Update on this.
I secured this contract. We're using a pair of Juniper SRX 380s as we got 10Gbps dual DIA circuits
I am posting lessons learned for posterity's sake and a cautionary tale for others who search for something similar to this.
pfSense cannot perform advanced routing in a stable way. FRR needing to be reloaded for changes is a problem that i do not blame pfSense on. Thats just the way the package currently functions but still should be taken into account. I got over 15 sites in a hub and spoke set up. If i update frr im breaking connectivity for all sites. When won't I have to make a route map change? Add a new BGP neighbor? There is no maintenance window in the world that a company would approve a global outage. There are workarounds for this I suppose but not worth exploring.
As I outlined in my redmine, there is an issue with IPsec that impacts FRR in a negative way. The problem isnt with FRR.
If there is a need to do routing over IPsec (obviously utilizing VTIs) then pick another firewall. Imagine you have a datacenter terminating over 50 IPsec tunnels. All you do is update the IPsec configuration or even onboard another site and click apply. You just broke routing within the enterprise. Thats absolutely insane and scary. This is something that can be replicated by TAC per the redmine. I cant recommend in good conscience deploying pfSense in that situation.
I got extremely lucky in that my client paid thousands of dollars on the 6100s to make the sacrifice of getting the Juniper head-end SRXs to manage all of this.
I really do advise anyone reading this to reconsider something else if your solution requires dynamic routing with IPsec. Beware,..
Lastly, there are lots of things that pfSense gets right. I will continue to deploy it in much less advanced scenarios but cannot use it going forward on topologies that require High availability with routing. The software just cant do it. This was indeed an eye-opener for me but we all learn the hard way.