IPVPN and VTI 2x IPSec tunnels failover
-
Hello,
I have a bit specific situation with failover with one customer. So the customer has 2 PFS in CARP in datacenter and I've upgraded them to 2.6.0 version recently. Other side is Fortigate firewall which is ISP managed (I have no access to it). So the idea is to make 3 vpn tunnel automatic failover if IPVPN (main vpn link) goes down which is tunneled by ISP over MPLS.
So it's made as this simple diagram shows: ISP has 2 Mikrotiks that link tunnels via MPLS, and Pfsense and FG have one dedicated interface for that tunnel and when that tunnel is running it's all routing normally and perfectly. Trouble starts when we try failover then I can ping from networks on pfsense side to remote site but cannot access nor ping anything from remote side to pfsense side networks. I've made GW group with IPVPN over MPLS to be Tier1, ISP1 ipsec to be tier 2 and ISP2 to be tier3.
I've defined all policy based routes towards remote site and it works fine and failover works from pfS side. ISP technician/admin that has access to FG has configured it to work with SDWAN policy for failover. We setup one test computer on remote side to just route over IPSec so we don't interrupt customer to try out the failover.
That test comp. is pingable through ipsec tunnel from pfsense side but not opposite. When I check firewall logs and states I see that traffic is coming to IPSec interface and is outgoing through it... I'm not sure if I defined the Floating PBR properly since whatever I try configuring it doesn't return replies to ping, when I check the Wireshark packet capture I see message "no response found". Test computer on remote side is on 10.10.30.7 and one of servers in datacenter behind pfs is on 10.10.50.66.
This is firewall log:
This is the state table:
That Server is in _SERVERS network (10.10.50.64/27)
Alias RG_TESTNO_RAC is test pc on 10.10.30.7 (remote side)
Alias DBX_NETS are networks on datacenter side
Alias RUGVICA_NETS are remote networks (for actual production)
IPSEC_GW_Failover is gw group where ISP1 IPsec is tier1
VPN_FAILOVER is where IPVPN (MPLS) is Tier1
_SERVERS interface has rule for that test comp as Dst and it goes over GW of ISP1 IPSEC to utilize PBR (that one works fine).Under Advanced options I have turned on:
I'm really annoyed with this already as I'm already unsure what to do... Is there something missing to match rules or is there something that is unneccesary?
Do I maybe need to use Tags to redirect traffic differently? For example: icmp req comes over Ipsec (inbound) and to not answer outbound with Ipsec_A1 interface to instead answer through _SERVERS interface when going outbound somehow?....