Site-to-Site TAP Bridge (Layer 2) VPN with identical VLANs / subnets on both ends?
-
(trying to understand how this works)
Site1 (ISP1) - pfSense1 - Trunk1A - Switch1 - (VLAN10, VLAN20...) - Subnets(10.10.x.x/16, 10.20.x.x/16 - matching the VLAN numbers)
Site2 (ISP2) - pfSense2 - Trunk2A - Switch2 - (VLAN10, VLAN20...) - Subnets(10.10.x.x/16, 10.20.x.x/16 - matching the VLAN numbers)
I want to setup a VPN connections between Site1 and Site2 such that it appears as if it is just one site with two switches trunked together, two pfSense-es, one on each switch, connected to two different ISPs. Hosts on the same VLAN/subnet wouldn't know that they are far apart.
The major difference would be that both pfSense1 and pfSense2 will have to run DHCP (different ranges) and that for some (magical) reason when one of the ISP connections goes down the trunk between the switches goes down as well.In my search I have seen several questions asked online about such a Site-to-Site setup and in most cases there are comments strongly discouraging it. Here is an official Netgate video
https://youtu.be/ku-fNfJJV7w?t=4531
saying that it would create "a mess of conflicts and multicast traffic."What problems should I expect when, in my case, both sites are the typical household jumble of computers, smartphones, IOT devices? I also plan to setup a two-way TrueNAS replication across the VPN.
My understanding of the problem is that pfSense1 receives packets over Trunk1A destined to what is normally a local ip address / MAC address and not internet bound traffic to go out of the WAN port. Before it wraps the packets to be sent through the VPN (again WAN) it has to make sure that the destination host is not truly local.
I want to ask if the following is a viable solution that will eliminate all conflicts etc.:
What if I make two trunk connections between pfSense and the switch on both Sites.Trunk1A, Trunk1B on Site1
Trunk2A, Trunk2B on Site2On Site1, Trunk1A is used for the normal internet traffic (default gateway, DHCP, DNS...)
Trunk1B is dedicated to the VPN traffic. Anything pfSense receives on Trunk1B it blindly forwards to Site2 - no need to make sure the hosts are local etc., just blindly forward anything it receives on that dedicated Ethernet port.
When pfSense2 receives traffic on the specific VPN port number, it blindly (securely, of course) forwards whatever it receives out Trunk2B.
Trunk1B and Trunk2B establish a normal trunk port between Switch1 and Switch2. What traffic goes out Trunk1A or Trunk1B is determined by the ARP table on Switch1. pfSense1 does not have to disambiguate normal internet traffic from VPN traffic, by virtue of having separate physical Ethernet interfaces for each. Broadcasts should not be a problem either. pfSense1 knows not to forward broadcasts received on Trunk1A out of the WAN port, and it should not have to worry if they go out of Trunk1B as the possibility of a loop is eliminated. Multicast, that should be handled at the switch level too, right?I am sure it is more complicated than that. Please let me know what I am missing.
Thanks!