Backup route flapping problem

[mcury]

I have two sites, both sites with dual WANs, and configured two IPsec VTI tunnels between both.
Both sites running pfSense 2.7.2, all patches applied, and all packages up to date.
Configured FRR between both sites for failover and everything is working.

The problem happens when the backup path comes UP, It takes down RDP sessions, even pings stop.
This shouldn't happen, the connections are going through the primary path.

Note that I tried with BGP (neighbor with lower weight), and OSPF (neighbor with higher cost) to confirm if it was a bug with OSPF or BGP, but the problem happens with both protocols.

With BGP, I tried eBGP and iBGP (iBGP with loopbackps as source and static routes for each loopback in each side), same problem happens.

If you need more details, let me know and I'll post here, thanks.