multiple IPSEC tunnels via CARP vip

dakobg

Hi,

I have 2 sites with 2 ISP per site on top of that every site have 2 pfs routers with HA
For each site I have 2 carp vip for WAN (one per ISP)

site A
A_ISP1_vip
A_ISP2_vip

site B 
B_ISP1_vip
B_ISP2_vip

I try to setup IPSEC HA tunnels between those carp vip addresses

A_ISP1_vip - B_ISP1_vip
A_ISP1_vip - B_ISP2_vip
A_ISP2_vip - B_ISP1_vip
A_ISP2_vip - B_ISP2_vip

work ok:

A_ISP1_vip - B_ISP1_vip
A_ISP1_vip - B_ISP2_vip

however when I try to add

A_ISP2_vip - B_ISP1_vip
A_ISP2_vip - B_ISP2_vip

I get a error
"The remote gateway "B_ISP2_vip" is already used by phase1 "

Actually I can add

A_ISP1_vip - B_ISP1_vip
A_ISP1_vip - B_ISP2_vip
A_ISP2_vip - B_ISP1_vip

but for last one I get this error for remote GW
also
A_ISP2_vip - B_ISP1_vip is added but I get the same error If I try to edit it :)

I do a test setup before change production (pfs+) with community edition pfs 2.5.0

dakobg

I'm not sure if this setup is possible with carps vips (is strongswan check interface not ip ?)
Any Idea?

pfs GW - ISP1/2 GW are in group (tire1/tire2) we use isp1 as primary / ips2 as backup

I think I can solve this problem if I point ISEC tunnels to direct IP addresses per pfs, however this will increase number of tunnels which is not good :) from configuration point of view
it will be something like (8 tunnels but what about HA config sinc between pfs1/2 per site? The plan is phase2 VTI for OSPF)

I hope, I was able to explain my problem

Regards,

dakobg

Well I realize this will be tricky since SA cannot be duplicated.
But still I wonder what will be the best approach with such setup.

Other option can be with 3th site but in our case is not possible/reliable :(

dakobg

Is someone try

Gateway duplicates: Enable this to allow multiple phase 1 configurations with the same endpoint. When enabled, pfSense does not manage routing to the remote gateway and traffic will follow the default route without regard for the chosen interface. Static routes can override this behavior.

?

dakobg

Look like this solve my problem (gateway duplicates enabled),
All 4 tunnels are up as additional requirement for this setup is all 4 vti in OSPF with proper ACL if connected routes are redistributed.

Also proper DPD config is required, I'm still playing with that, but this is more related to OSPF

Whit this setup any site can loose ISP or PFS router with minimal downtime (around 1min related to GW groups HA and ospf) not ideal but automatic FO/HA is working.

Regards

dakobg

ok,

From HA setup prospective work ok no issues, (note add metric to OSPF interface in order to control which vti to be used :) )

However I hit other issue not related to HA setup but to VTI

For short outage is ok, however when ipsec DPD expire ipsec disconnect and must be reconnected manually

I found this: https://redmine.pfsense.org/issues/12169
but still is not implemented :(

Also I found a lot of users are using corn scripts to reconnect IPSEC VTI which is probably ok but ...

I will try to disable DPD but this probably will lead to other issues ..

will update when I do more tests ..

If someone have a solution about that let me/us know

dakobg

disable DPD was bad idea since keep SA up and when PFS1 die PFS2 is not able to connect (PFS1 from the other site sitll keep them up)

Look like for now we should use scripts or manual reconnect.

NOTE: I know this post is new but please if someone already have the same issue just to say .. no solution, you do something wrong or try this .. :( look like I'm in monologue here

I will keep do testing if someone find anything useful here however my best suggestion will be to wait for https://redmine.pfsense.org/issues/12169 to be released

dakobg

Custom script + cron do the job.

When we I have a time will summarize and will provide more info for the script