Suggestions for linking two pfsense setups

[MakOwner]

Not really sure if this is even the right place to ask, or if I can adequately explain this without images.
I have two buildings, about ~100 feet or so apart, each with their own network configuration.
Site A is served by a static public IP address and is used mostly for incoming traffic, VPN connections, etc -- mostly office stuff.
Site B is served by a /29 public network and has (web, mail, media) servers, (backup, media) storage - most traffic is outgoing traffic.
Some things like the media and backup we never want traversing the internet.

Each site sits behind pfsense and the LAN at each site is a 192.168.n.0/24 network.
(Internet connectivity at each building is abysmal, thus a connection at each one. It's Frontier DSL.)

I recently installed a set of switches and cabling between the buildings to create a bonded 10GB trunk for backups and access to media, and to keep me from trekking between buildings in the middle of the night.

I'm looking for suggestions on how to reorganize the network connectivity and maintain the DHCP/DNS on both ends without exposing more than I have to or doing more work than I have to.
I rely heavily on both sides on DHCP and assigned IPs designated by mac address.
Is it just a matter of opening up the netmask range on selected clients in order to expose the other subnets?

I'm still in the stage of configuring the LAGs for the switches between the buildings right now.
Two 10GB links aggregated between buildings and up to 4 1GB links aggregated at each end into the main switches from the 10GB switches.

[stephenw10]

Setup a small transport subnet for the link.
Add static routes to each side for subnets you need to access.
Add firewall rules on each side to allow only the connections you require.

Should be fairly straight-forward.

For bonus points add each side as a fail-over gateway for the other to use in the event of a WAN outage.

Steve

[MakOwner]

Granted, I'm no mental giant with networking, but as near as I can tell I have followed the netgate docs on setting up static routes on the LAN side of pfsense.

When pinging from one side you get an immediate "no route to host response" and from the other you get no response at all.

I have configured similarly on both sides -- a static route to a Class C /24 pointing to the other pfsense with some additional advanced options selected as advised in the documentation: Enabling System -> Advanced -> Firewall & NAT -> Bypass firewall rules for traffic on the same interface

What am I missing here?

Edit: spelling.

[bingo600]

@MakOwner
In order to get the two pfSenses to speak to each other, you would need to have an interface on each device in the same subnet the conect(ed) net.

Ie. make (configure) an interface on each device , that is member of the connect net. Ie. connect net = 192.168.255.0/24 or /30 or ?

pf1 : 192.168.255.1 /30
pf2 : 192.168.255.2 /30

Now you should (must) be able to ping 1 from 2 , and 2 from 1.
Remember to allow icmp on the opposite inbound if.

A static route tells the pfSense that you can reach network xxxx/yy via zzzz.

So if pf1 has Lan net 192.168.1.0/24 , and pf2 has Lan net 192.168.2.0/24

You have to tell pf1 that pf2's lan can be reached via the pf2 connect net IP (gateway).
Static route 192.168.2.0/24 - Gateway 192.168.255.2

And pf2 that pf1's lan can be reached via the pf1 connect net IP (gateway).
Static route 192.168.1.0/24 - Gateway 192.168.255.1

Add some traffic rules to the connect net interfaces in both ends, and voila .. You're done

/bingo

[MakOwner]

So I need to add an interface on each end to provide the third common subnet between the two pfsense systems then?

[bingo600]

@MakOwner
You would need a "common" net , that has an interface from both pf1 & pf2 in it.

It could be to put an interface of ie. pf1 , into an existing net on pf2 (lan).
But the common way to do it , is to create a separate net for "interconnect".

That way you have full firewall control over the data that flows between pf1 & pf2 , as the connect net is only used for transport , not clients.

The choice is yours.

/Bingo

[MakOwner]

I have a bonded 10GB pair between buildings, with a 4x1GB bond between the pair-link and each end's local switch.

Forgive the crude drawing, but this is the physical setup I have:

This is all close enough - and small enough - it can be handled easily in one Class B /24 for physical/virtual hosts. LAN2 ISP account has a public /29 and LAN1 has a /22.

I am trying to keep DNS/NTP/DHCP service local to each end.
I'm not really sure that even makes much sense at this point.

LAN2 will serve public facing mail, web and cloud services on the public IP addresses, and the primary intent for the link between the buildings is for local access to those resources so the traffic doesn't go out over the much slower internet connection. I have done public facing services to an ISP before with a single pfsense with no issue.

Just not sure the easiest approach in this case.
I just want to keep management as simple as possible and not have to add/adjust routes every time I add or remove a new system in LAN2. It happens frequently.

[bingo600]

@MakOwner

What is the purpose of the upcomming pfSense connection between sites ?
Is it just to be able to "configure the other pfsense" wo. leaving the building ?

Do you now have access to building1 lan in building2 , and reverse ?

What is the IP or Vlan properties of the bonded "inter building" connection ?
Can you create vlans on the switches making the building interconnect ?

Do you have any free interfaces on the building pfSenses ?

As the old IBM Mainframe APAR team would have said : "More DOC Required"

/Bingo

[MakOwner]

Ah yes. "This page intentionally left blank."

The site on LAN 1 is existing office and general use.
The site on LAN 2 is a new climate controlled structure built to house rack mount virtualized equipment -- and some physical systems that provide raw storage. It has one desktop seat now, and that will be used more in the future - especially when the main office space is remodeled in the spring.

The 10GB links between the sites is primarily to provide administrative access to everything in the other site (vCenter, Remote desktops, etc.) and on occasion will serve media off NFS storage from LAN2 to LAN1.
Backups and various large file saves from the desktops in LAN1 would traverse the link to LAN2 storage.

The switches on both ends are old Dell switches.
LAN1 is a PowerConnect 5224 and LAN2 is a PowerConnect 6248.
The switches with the 10GB links are Netgear GS110EMX.

All are VLAN capable

LAN1 pfsense is a physical pfsense and has a free physical port,
LAN2 is virtual and there is an available port on the ESXi host I could use.

I'm still running 2.4.4 on LAN1 and 2.4.5 on LAN2.
(Separate issue -- I haven't upgraded LAN1 to 2.4.5 because even after the last upgrade DNS Resolver daemon dies on occasion, whereas on 2.4.4 it will run for as long as pfSense does. Last update made it run longer, but it still dies. Thinking of just reverting back to the latest 2.4.4 on LAN2.)

It may just make sense to bite the bullet and try to do load balance across the pfSense boxes, but I'm not sure how that would work considering I have 5 public IPs on one side and 1 public IP on the other. And I have never messed with any of the load balancing / failover stuff in pfSense...

Do you now have access to building1 lan in building2 , and reverse ?
No network connectivity at present, just the physical layer.

What is the IP or Vlan properties of the bonded "inter building" connection ?
Everything is using the default-vlan (1) on all switches.

Can you create vlans on the switches making the building interconnect ?
I could if needed.

Do you have any free interfaces on the building pfSenses ?
Yes. The hope is, however that traffic (especially backup traffic) pass directly across the switches.

[bingo600]

@MakOwner

OT: Unbound Resolver dying.

Install the package service watchdog , and add unbound there (under service)

[MakOwner]

Thanks for the watchdog tip -- I'll give that a shot!

Even if I used a VLAN for the cross-site link, if I don't do one big flat network, traffic is going to go through the pfsense interfaces before going across the link, isn't it?

[bingo600]

@MakOwner
If everything is running in Vlan1 , then if you connect the building switches (10Gb interlink) you will have "Short circuted" your two building lans , into the same broadcast domain Vlan1.
That would/Could give you DHCP issues , as the devices in both buildings will now see answers from both DHCP servers , and will prob. take the offer that arrives first.

So we have 3 uses for the interconnect

1: Management/Admin (Low traffic volume)
2: NFS (??? Traffic Volume)
3: Backup (High traffic volume)

Re1:
Management
Quite easy to solve , described here
https://forum.netgate.com/post/944384

Re2:
NFS:
If the pfsense can handle the traffic , solved by Re1:

Re3:
Backup
If the pfsense can handle the traffic , solved by Re1:

If Not
Can your pfSense handle the traffic volume if running backups through it ?
Do you / Can you use dedicated backup interfaces on the servers ?
Then you might be able to make a separate Backup Vlan , and send the backup directly via the switches , would require the backup server also got an interface in the Backup Vlan.

/Bingo

[bingo600]

@MakOwner said in Suggestions for linking two pfsense setups:

Thanks for the watchdog tip -- I'll give that a shot!

Even if I used a VLAN for the cross-site link, if I don't do one big flat network, traffic is going to go through the pfsense interfaces before going across the link, isn't it?

Yes ..

Well Depends
If you created a specific Backup Vlan , and gave all your servers + Backup unit an interface in that Vlan , then you could transport Backup data on that Vlan without passing the pfSense.

[bingo600]

@MakOwner

I would create several Vlans

Building1 Lan - Vlan 10
Building2 Lan - Vlan 20
Backup Vlan - Vlan 30

Transport (tag) all vlans between all switches

pfSense1 gets an interface in Vlan 10+20 -Maybe 30
pfSense2 gets an interface in Vlan 10+20 -Maybe 30
No static routes needed , as the lans are known via the interfaces.

The Advantage of doing that would be that you could actually make a building1 machine appear as it is in building2 , just by putting the machine interface in vlan20. Same goes the other way around.

IMHO less flexible
You could keep your Vlan1's on both sides , and just create
Backup Vlan - Vlan 30
Building connect Vlan - Vlan 40
pfSense1 gets an interface in Vlan40 -Maybe 30
pfSense2 gets an interface in Vlan40 -Maybe 30
You add the static routes as described above (GW via vlan 40 ip's).

Then you only tag Vlan40 + 30 on the Netgears , now vlan1 can not traverse between buildings. Watch out for native vlan (PVID or whatever dell/netgear calls it).
Tag Vlan 30 on the DELL's to Netgear , so the servers can see the backup vlan cross site.

Servers still get an interface in the Backup Vlan , so pfsense is not needed to transport those data , they only flow within the vlan.

/Bingo

[MakOwner]

@bingo600 said in Suggestions for linking two pfsense setups:

If everything is running in Vlan1 , then if you connect the building switches (10Gb interlink) you will have "Short circuted" your two building lans , into the same broadcast domain Vlan1.

That's how I have it connected now, and traffic doesn't cross the two LANs because LAN1 is a different /24 than LAN2 and the default gateway is the only route - that works.
I tried setting up a route to the other gateway in pfsense through the existing LAN interface as you described in the link below. That's not working.

I did temporarily add an idle interface from one of the media servers to the switch with the 10GB ports in LAN2 (one of it's 1GB ports) and it picked up an address from the DHCP server in LAN1.
I can ssh to that server and from there ssh into any other server that is on LAN2.

I'm thinking maybe just plumb the spare interface on LAN1 pfsense and give it a static address on LAN2, and configure rules similiar to the interface I use for a wireless access point, but slightly less restrictive?

The physical pfsense I'm not sure can handle the backups and the in/outbound VPN.
It's rather old hardware - it works for the light duty it's doing now but I suspect pushing backups through it will be a challenge, thus the hope to pass traffic directly over the switch.

The backup targets will have dedicated interfaces -- the sources of backups in LAN1 won't.
Not really sure how stressfull they will be just yet.

That would/Could give you DHCP issues , as the devices in both buildings will now see answers from both DHCP servers , and will prob. take the offer that arrives first.
So we have 3 uses for the interconnect
1: Management/Admin (Low traffic volume)
2: NFS (??? Traffic Volume)
3: Backup (High traffic volume)
Re1:
Management
Quite easy to solve , described here
https://forum.netgate.com/post/944384
Re2:
NFS:
If the pfsense can handle the traffic , solved by Re1:
Re3:
Backup
If the pfsense can handle the traffic , solved by Re1:
If Not
Can your pfSense handle the traffic volume if running backups through it ?
Do you / Can you use dedicated backup interfaces on the servers ?
Then you might be able to make a separate Backup Vlan , and send the backup directly via the switches , would require the backup server also got an interface in the Backup Vlan.

[stephenw10]

You have 2 subnets on one network segment and that's always going to give problems.

If you want to keep the sites separate then use a separate transport subnet between the two sites on it's own interface in the two firewalls. Yes, that means all traffic will go through the firewall and, yes, that probably means you probably won't see 10Gbps across it. However it will give you far better control over what can see what. You will be correctly routing between the two subnets. You can keep the local DHCP/DNS etc as you specified.

If you want traffic between the two sites not to go through the firewall you should use layer 3 switches there to route it correctly. But otherwise just use one large subnet so hosts on eash side can 'see' each other. You will have to carefully setup a single dhcp server and configure it carefully so traffic uses the local gateway. You will still have two routers/gateways on the same subnet which is bad.

I would definitely go with option 1 there unless you have layer 3 switches.

Steve

[bingo600]

@MakOwner said in Suggestions for linking two pfsense setups:

@bingo600 said in Suggestions for linking two pfsense setups:

If everything is running in Vlan1 , then if you connect the building switches (10Gb interlink) you will have "Short circuted" your two building lans , into the same broadcast domain Vlan1.

That's how I have it connected now, and traffic doesn't cross the two LANs because LAN1 is a different /24 than LAN2 and the default gateway is the only route - that works.

If you have done as you said , all switches are having their ports in VLAN1.
Then traffic do cross the buildings , you are just not "listening for it".
Your DHCP setup would be a "ticking bomb" , as PC's in both buildings will get responses from both DHCP servers , and prob. take the offer that arrives first.

I tried setting up a route to the other gateway in pfsense through the existing LAN interface as you described in the link below. That's not working.

You would need the "Connect net" and pfSense interfaces from both FW's in that net , for doing that correctly.

I did temporarily add an idle interface from one of the media servers to the switch with the 10GB ports in LAN2 (one of it's 1GB ports) and it picked up an address from the DHCP server in LAN1.

LAN1 DHCP server answered fastest (Proof of the DHCP BOMB)

I can ssh to that server and from there ssh into any other server that is on LAN2.

Hmmm ... Does pfSense run "Proxy arp" ?

I'm thinking maybe just plumb the spare interface on LAN1 pfsense and give it a static address on LAN2, and configure rules similiar to the interface I use for a wireless access point, but slightly less restrictive?

You could , but you still have the DHCP BOMB
Without 2 DHCP servers active , you might just pull it off.

The physical pfsense I'm not sure can handle the backups and the in/outbound VPN.
It's rather old hardware - it works for the light duty it's doing now but I suspect pushing backups through it will be a challenge, thus the hope to pass traffic directly over the switch.

The correct way would be a separate Vlan for backups , but also separate interfaces in the backup Vlan from each server.

The backup targets will have dedicated interfaces -- the sources of backups in LAN1 won't.
Not really sure how stressfull they will be just yet.

You could give your Backup Destination Server , an interface (ip address) in both LAN1 & LAN2
That way you should be able to do backup , without passing the firewall.

If you want to do it "Wrong" .... And only have VLAN1.
You will have to solve the DHCP issue of having two active servers (DHCP Scopes) in the same Vlan. That will "Bite your behind" at some point, when a machine in Building2 (LAN2) gets a DHCP address from the server in Building1 (LAN1).

But why not do it right the first time

You have been told twice by stephenw and I , that you are going down a dangerous path.
And DHCP will be your "Murphy" ...

/Bingo
/

[bingo600]

@MakOwner

You could pull it off by running both LAN's in VLAN1 , if you solve the DHCP issue.
And give your Backup Server an ip address in both lans.

Backup:
All devices in LAN1 would then backup to the BACKUP servers LAN1 address
All devices in LAN2 would then backup to the BACKUP servers LAN2 address

DHCP:
Both DHCP servers will respond to the DHCP Broadcast , and will offer an ip address.
Your device will get an offer for both LAN1 & LAN2 , it usually picks the one that arrives first.

What do you use the DHCP for ?
Servers ?? .. They could be set to use static ip info (Lan1 or Lan2).

Clients
You could disable LAN2 DHCP , and let all DHCP Clients get DHCP from the LAN1 server.
Then Clients would only use internet of pfSense1 (LAN1).

Technically there's nothing that prevents you from having two subnets active in one Vlan.
It's just not "Best Practice" , and a bit of a hack.

But get rid of one of the dhcp scopes (servers) , or accept random Lan1/Lan2 assignments.

pfSense:
Give pfsense1 , an interface in Lan2
Give pfsense2 , an interface in Lan1

@stephenw10 - Would VIP Alias work as the 2'nd interface ?

I have only used VIP Aliases once , but that might work for you.
Instead of making real interfaces.

Select your "native" Lan interface , and make the "other lan" as an ip alias network

/Bingo

[MakOwner]

I am listening to everything you both said -- I'm trying to re-configure these without getting so many changes in at once I can't get myself back to an operational state - especially on LAN1 where I have a day job that I have to use that link.

I really would just collapse this down to one account, except that these are two DSL accounts - the download speed is barely livable, and the upload speed is... well, it's Frontier.
At least it's not AT&T, but that's all that can be said for it.
LAN1 has the single static IP and LAN2 has 5 static IPs.
(Originally, LAN2 was in a different city on FIOS... And that's why we have two accounts. Stuck with this for at least another year. Nothing creates a legacy configuration like doing something on a temporary basis, amiright?)

I'll get a couple of weeks around Christmas of downtime and might look into trying to HA/load balance the two DSL accounts and just use weighting to favor gateways for each building.
I have enough lines/ports between the buildings for the direct heartbeat link between the two pfsense. That's a lot of homework.

In the meantime --- I'm hoping to poke at this until I get something working.
It's not that I'm not trying to do what you both have suggested -- I am working my way through each suggestion, I just haven't hit on one that works yet.
Most of the routing information in the Netgate documentation appears to be aimed at routing in a single pfsense, so it's not always obvious that the examples are correct for my scenario.
I have yet to see any examples of dual WAN + dual pfsense setups.
(probably because it's a really bad idea...)

(I thought I hit submit on this the other day... :/ )

[MakOwner]

To simplify this and create a functional, even if suboptimal - because traffic will go through the low end physical pfsense -- setup -- would this work:

Adding a third interface on each pfsense, creating VLAN5 across the link.
Question I have, will using (for example) 192.168.30.1 on one end and 192.168.30.2 on the other work? (apologize for the crude graphic.)