Route between two locations with separate ISPs and LANs
I have two physical locations, each with their own internet connection and each running their own PFSense box.
Until now we've had a T1 connection at each end handling phone and data traffic between the two locations, each location with it's own Cisco DSU and router. We just had a new fiber connection installed at each location giving us a 10mbps ethernet connection between the two buildings.
While it's possible to just plug the smaller office directly into our main LAN through this 10mbps connection, I'd rather keep them on separate subnets with separate internet connections. Rather than replace the cisco routers, we'd like to use our existing PFSense boxes.
Is it possible to add an optional interface to both pfsense boxes to route traffic between both lans while keeping internet traffic routed to the respective WAN connections?
ISP A (WAN Interface) ISP B (WAN Interface)
PFS A –(OPT1) ------- LINK ------ (OPT1)--- PFS B
LAN A (LAN Interface) LAN B (LAN Interface)
I thought I could just assign a subnet for the two optional interfaces to talk to each other (Say 10.10.0.1 and 10.10.0.2) and set up a static route for the other site's subnet to go through the 10.10.x.x network. Unfortunately, possibly due to firewall rules or NAT, traffic is not routing.
The two subnets share full access to servers and printers on both sides, so I want no filtering of traffic between them.
It should be possible. On PFS-A, add a route that states that all traffic bound for 10.2.0.0/16 goes to 10.10.0.2. Create a rule in LAN that states that anything going to 10.2.0.0/16 goes through that new gateway. You will need to put that rule above all rules. The in the OPT1 rules tab, set a rule that allows everything to anywhere.
You will want to do the same thing on PFS-B but using the PFS-A LAN subnet. You might need to restart to make sure there is not anything left in memory that would not allow this to work. On the OPT interface rules, you will want to change the default behavior to keep states. You will not want to keep states on these since it should just be routing. On the LAN rules that use the new gateways, you will also want not to keep states.
Keeping states will use unnecessary memory space.