Dual Wan failover using P2P microwave connection and site to site VPN

l4tran

Hello all,

I have two sites which are 1km apart. Both sites are connected using a P2P Microwave connection. Site A is connected to Fiber internet, Site B shares internet through P2P to Site A. On a good day everything is fine. Occasionally, the Microwave P2P connection breaks down due to heavy rain, snow or humid condition, as a result Site B goes offline. Currently, both sites use Juniper SSG5 as the firewall.

The Plan;
-Site B will get its own dedicated Fiber connection with a different ISP.
-Site A and B will then be connected through OpenVPN.
-The Microwave P2P connection will be used for failover.
-Lan1 can have internet access as well as access to Lan2.
-Lan2 has no access to internet or Lan1
-Of course Junipers will be replaced with pFsense routers.

The Setup;
SiteA
-WAN Fiber ip: x.x.x.x/32
-Microwave ip: 192.168.101.1/24
-Lan1 ip: 192.168.1.1/24
-Lan2 ip: 172.16.1.1/24

SiteB
-Wan Fiber ip: y.y.y.y/32
-Microwave ip: 192.168.101.2/24
-Lan1 ip: 192.168.11.1/24
-Lan2 ip: 172.16.1.2/24

The Question:
How can all this be achieved using pFsense routers.
-Looking online, I can probably get OpenVPN to work.
-I can probably get the failover to work.
-Not sure how to get Lan1 to communicate with Lan2 on both Sites

l4tran

Here is a network diagram of what I am trying to achieve.
I created two VPN tunnel across the WAN and the P2P connection.
With both WAN connection up, everything is fine. However, when I pull the plug from one of the WAN, the default gate switches over to the OPT gateway but I can only reach the OPT interface from the other side. I cannot reach the WAN or the Lan.

This is what I am thinking. The OPT interfaces are pointing to each other as gateways, therefore they are not reaching the WAN. Would I need to create a static route on the OPT 0.0.0.0 -> WAN gateway and 192.168.0.0 -> OPT gateway?

l4tran

This post is deleted!

Derelict

Sounds like the firewall rules or outbound NAT or etc are not correct on the OPT3 interface on pfSense 2.

l4tran

When the WAN goes down on the pFsense1, GW_OPT3 becomes default. For pFsense1 to communicate with pFsense2, I would need to manually switch default gateway on pFsense2 from GW_WAN to GW_OPT3. Doing this I lose internet access to pFsense2.

Derelict

What?

chpalmer

@Derelict said in Dual Wan failover using P2P microwave connection and site to site VPN:

What?

I had to read it about ten times to understand myself..

@l4tran

When the WAN goes down on the pFsense1, GW_OPT3 becomes default. For pFsense1 ( LAN ) to communicate ( with the LAN of ) pFsense2, I would need to manually switch default gateway on pFsense2 from GW_WAN to GW_OPT3. Doing this I lose internet access to pFsense2.

(I believe I understand you by the changes I made above..)

Yes you would. How important is it that you have access to the LAN behind pfsense2 while your having a WAN outage on pfSense1? I would think that if you had resources on pfsense2 that you could NAT to them just as you would on a WAN interface.

How often do you lose connectivity on your WAN's?

l4tran

@l4tran

When the WAN goes down on the pFsense1, GW_OPT3 becomes default. For pFsense1 ( LAN ) to communicate ( with the LAN of ) pFsense2, I would need to manually switch default gateway on pFsense2 from GW_WAN to GW_OPT3. Doing this I lose internet access to pFsense2.

(I believe I understand you by the changes I made above..)

Yes you would. How important is it that you have access to the LAN behind pfsense2 while your having a WAN outage on pfSense1? I would think that if you had resources on pfsense2 that you could NAT to them just as you would on a WAN interface.

How often do you lose connectivity on your WAN's?

Sorry for the confusion and thanks for rephrasing my sentence.
I unplugged both WAN interface from both routers and test only the OpenVPN connection on the OPT3 interface. It doesn't seem to be working even though the status says it's up.
What should I select as the Upstream gateway on the P2P OPT3 interface? Should I select "none" or what is shown in the picture? Either way, the status on the OpenVPN shows that the connection is UP, but neither work.

The WAN connection on pFsense2 is a brand new fiber connection, just installed less than a week ago. We were relying on the P2P Microwave connection on a Juniper ssg20 for our connection to both sites. But it wasn't as stable, any moderate weather condition ie wind, hail, snow, rain could drop the connection.

l4tran

I got it to work. Partially.

I modified my diagram and relabel all the interfaces to easily reference what is going on.
I also removed the OpenVPN tunnel on the P2P connection.

After more testing this is what I found.

Case1:
When both WANs are up, both LANs can ping each other and both can access internet.

Case2:
When both WANs are down, the default gateways both points to GW_OPT1&2 and both LANs can ping each other. Obviously no internet on both.

Case3:
When WAN1 goes down, the default gateway on pFsense1 points to GW_OPT1, LAN1 can ping LAN2, not the other way around.
To get LAN2 to ping LAN1, I created a static route on pFsense2 192.168.1.0 to gateway GW_OPT2.
To get internet to LAN1, I created a nat rule source 192.168.101.0/24 on pFsense2.

Here is the catch. This is not a true fail over because I have to manually manipulate the Static Route. Is it possible to write a script on pFsense2 to detect if the traffic over the VPN tunnel has high latency or packet loss? If yes then enable the Static route. If no then disable the static route. Then set a Cron job to run every 5 mins?

Derelict

It looks to me like you need to:

Change your default gateway from Automatic to the gateway group.
Policy route your LAN traffic to the gateway group

https://docs.netgate.com/pfsense/en/latest/book/multiwan/index.html#multiple-wan-connections