Route Traffic from Site2 over VTI to Site1
-
I have two sites, both with pfSense, and I have a working IPSec VTI tunnel between them. I have stood up BGP and routing is working as I expect; I can ping nodes across the VTI without any issue. Now, what I am trying to figure out is how to set one VLAN up with a default route in Site1, so that all it's traffic will go across the VTI when it's up, otherwise fail back to the local gateway when the VTI is down, which is why I want to use VTI and BGP to hopefully do all the routing heavy lifting for me. If I need to set it up so that all VLANs use Site1's WAN when the VTI is up, that's ok too.
The piece I can't seem to wrap my head around is how the heck to make the VLAN route across the VTI and use Site1 WAN when it can, otherwise use it's own WAN at Site2. I do have BGP set to Originate Default with Site1, and can see in the FRR Status that is the case, but when looking at the Routing tab under Diagnostics, I don't see that the default route is going across the VTI, so I'm a little confused on to what I'm missing.
Here's the diagram (using GNS3) of the setup with relevant info. I've mocked it in GNS3 because it's a long way to Site2, which is not a manned site, so reboot/console recovery is not always an option without a long drive. Besides, this should work in GNS3 the same, it's just forming the tunnel over an "external" IP space.
-
You can route across the VTI and use Site1 WAN when it can, or use WAN at Site2 if vti fails . without BGP.
All you need, is to use Gateway Groups , put vti interface as tier 1 and its WAN as tier 2. then use Policy Routing on vlan interface.
https://docs.netgate.com/pfsense/en/latest/book/multiwan/policy-routing-configuration.html
https://docs.netgate.com/pfsense/en/latest/routing/multi-wan.html#firewall-rules
if IPsec VTi fails , the traffic will pass across its Wan.
also configure what VTI needs , like NAT and Static routes.
it works good here. -
@Zawi deceptively simple to say the least, and it took me a few times to see it in the documentation. I think I did try that before, but the key is that on my Site1 the Outbound NAT did not automatically include the subnet's from Site2, so once I put the Outbound NAT into Hybrid Mode and added the subnets, well things are now working as expected.
I am still using BGP though simply to avoid the static routes, I have a few subnets and am lazy. Couple of things I've learned also is under the Gateway entries, in Advanced you can define the thresholds for latency and packet loss for the gateway to be considered up/down, which is key here. Also, I had the VTI gateway set to disable monitoring, which in my testing also broke the failover, which was another key problem.
-
I do seem to have some odd issue's that I'm tracking down but maybe someone already has come across this. It appears most things are working, but when doing something like Speedtest, the download side works great, but upload tests fail. I will be doing some packet captures to try and figure it out, but not sure if this is a known condition of this configuration or not.
-
Well whatever was going on seems to be transient as things seem to be working now, although with the current situation extremely slow and laggy.