Configuring pfSense for a point to point link between two buildings
-
Our company is looking to build an additional facility within 1/2 mile of our current HQ. Both facilities will remain in use. We are also going to be able to arrange to install a point to point single mode fiber link between the two buildings. The idea is for the Expansion to use the existing internet service in the HQ.
My network is kind of complex in my HQ right now in that we have an HA/CARP cluster running with two ISPs serving that location. For the sake of discussion and helping me understand how I should proceed, let's assume I've got a single ISP providing a /29 block of public IP addresses and a single router in the HQ and a single router in the Expansion.
What is the best way to go about this? I've had three ideas so far:
A. The first obvious thought is to just connect a switch in Expansion to a switch in HQ so they are part of the same L2 network, but I have two problems with this. 1) I have read that this can cause issues in L2 networks with certain protocols like spanning tree (though I have no direct experience with this) due to slightly increased latency and 2) traffic crossing the fiber between hosts at each location could easily contain unencrypted sensitive packets. The traffic on that fiber link is "inside the network traffic" but it's crossing a wire that's outdoors, so that seems like a real security concern to me.
B. I also considered using a perimeter switch that is attached to the ISP demarc and the WAN interface of my HQ router behind it (in point of fact, I actually have this now, because while I'm discussing this as if I've got one router and one ISP, in reality I've got an HA cluster and two ISPs, and a perimeter switch makes it possible for the HA/CARP cluster to work correctly) and then have the perimeter switch attached via the point to point fiber to the WAN interface of a router in the expansion. This solves the security issue in that traffic heading to the internet from the expansion will be no different than traffic in the HQ heading to the internet; you expect normal TLS enabled protocols to protect things here. In this case we would establish an IPSEC tunnel between the Expansion and HQ just like all our other sites already have IPSEC tunnels to HQ. Internal network traffic between the locations would then be encrypted as they traversed the point to point link (the physical path would be Expansion LAN network <-> Expansion router WAN interface/point to point fiber <-> Perimeter Switch in HQ <-> HQ Router WAN interface <-> HQ Router LAN network).
This might still incur L2 issues over a point to point link though. It's not extending my L2 internal LAN over the point to point, but it is extending the L2 "lan" of our public ip block over the point to point fiber.
C. My final and third idea is to contact the ISP and see if they will provision another static block of IP addresses and route them to one of the unused SFP interfaces on the demarc router they have in our network closet, and then connect the demarc to our Expansion's router via point to point fiber.
What is the best way to do this? Leaning towards Option C if I can get the ISP to cooperate, then Option B if I can't, but I'm not sure if it's the best practice, or if there is an alternative I'm not thinking of that's better than any of these.