Different multi-wan scenario: 2 sites with own internet connection, interconnected

  • Hi!

    I work at a school and we've migrated our 2 sites from custom-built Linux firewalls to pfSense firewalls. We have a multi-wan environment which is quite different from the multi-wan environment that is described a lot in the documentation and, even though it's been working great, we're not sure this is the correct way of implementing it using pfSense (we come from a Linux networking background, but we don't have a lot of experience with pfSense).

    In most of the multi-wan scenarios, people have one pfSense instance with two internet connections. Our scenario is quite different: we have 2 sites, each one with its own internet connection (one of them has an 80mbit connection and the other has a 100mbit connection) and the sites are linked together by a gigabit fibre connection and a wireless backup link. Internal traffic flows freely between site A and site B (both are on different subnets) and OSPF handles the routes between both sites and the switchover from the [faster] fibre connection to the [slower] wireless connection if the former ever breaks. This layout is shown in the following diagram:


    OSPF provides ensures we never lose connectivity between the sites. In order to ensure we never lose internet connectivity too, we did the following:

    1 - We've created a GRE tunnel between the sites. On each site, the tunnel's local endpoint is pfSense's IP on the internal network and the remote endpoint is pfSense's IP on the internal network on the remote site. By using the tunnel's endpoint on each sites' internal networks we can be sure the traffic on the tunnel will switch from the fibre connection to the wireless connection if OSPF decided it has to. The tunnel interface has addresses and;

    2 - On each site, there's a rule which states that traffic whose destination is the internet and which arrived by the GRE tunnel interface MUST use that site's WAN link. This ensures that if one pfSense box sent traffic through the GRE tunnel, the receiving end will handle it by always sending it through its own internet connection (or failing to deliver it if the link is down). This prevents loops, as this traffic will never be sent back to the sending pfSense box if it targets the internet;

    3 - Hosts on each site's internal network use a gateway group for internet traffic. This gateway group uses the site's internet connection as Tier 1 gateway and the GRE tunnel as a Tier 2 gateway. As such, they only use the GRE tunnel (and the other site's internet connection) if there's a problem with the site's own internet connection.

    The GRE tunnel is only used for internet traffic; traffic whose destination is the other site's internal network use the fibre interface (or the radio interface) directly.

    We've used this tunnel setup in order to simplify the gateway group rules, as it seemed wrong to have 3 gateways on the gateway group (WAN, remote fibre endpoint and remote radio endpoint) when, in fact, we had only 2 internet connections. But, to me, even though this setup has been working great for almost one year, it still seems kind of hacky.

    Is this the correct way of implementing it or is there some other feature (which we might have overlooked) which would fit this scenario better?

Log in to reply