Suggestions for linking two pfsense setups
-
@MakOwner
If everything is running in Vlan1 , then if you connect the building switches (10Gb interlink) you will have "Short circuted" your two building lans , into the same broadcast domain Vlan1.
That would/Could give you DHCP issues , as the devices in both buildings will now see answers from both DHCP servers , and will prob. take the offer that arrives first.So we have 3 uses for the interconnect
1: Management/Admin (Low traffic volume)
2: NFS (??? Traffic Volume)
3: Backup (High traffic volume)Re1:
Management
Quite easy to solve , described here
https://forum.netgate.com/post/944384Re2:
NFS:
If the pfsense can handle the traffic , solved by Re1:Re3:
Backup
If the pfsense can handle the traffic , solved by Re1:If Not
Can your pfSense handle the traffic volume if running backups through it ?
Do you / Can you use dedicated backup interfaces on the servers ?
Then you might be able to make a separate Backup Vlan , and send the backup directly via the switches , would require the backup server also got an interface in the Backup Vlan./Bingo
-
@MakOwner said in Suggestions for linking two pfsense setups:
Thanks for the watchdog tip -- I'll give that a shot!
Even if I used a VLAN for the cross-site link, if I don't do one big flat network, traffic is going to go through the pfsense interfaces before going across the link, isn't it?
Yes ..
Well Depends
If you created a specific Backup Vlan , and gave all your servers + Backup unit an interface in that Vlan , then you could transport Backup data on that Vlan without passing the pfSense. -
I would create several Vlans
Building1 Lan - Vlan 10
Building2 Lan - Vlan 20
Backup Vlan - Vlan 30Transport (tag) all vlans between all switches
pfSense1 gets an interface in Vlan 10+20 -Maybe 30
pfSense2 gets an interface in Vlan 10+20 -Maybe 30
No static routes needed , as the lans are known via the interfaces.The Advantage of doing that would be that you could actually make a building1 machine appear as it is in building2 , just by putting the machine interface in vlan20. Same goes the other way around.
IMHO less flexible
You could keep your Vlan1's on both sides , and just create
Backup Vlan - Vlan 30
Building connect Vlan - Vlan 40
pfSense1 gets an interface in Vlan40 -Maybe 30
pfSense2 gets an interface in Vlan40 -Maybe 30
You add the static routes as described above (GW via vlan 40 ip's).Then you only tag Vlan40 + 30 on the Netgears , now vlan1 can not traverse between buildings. Watch out for native vlan (PVID or whatever dell/netgear calls it).
Tag Vlan 30 on the DELL's to Netgear , so the servers can see the backup vlan cross site.Servers still get an interface in the Backup Vlan , so pfsense is not needed to transport those data , they only flow within the vlan.
/Bingo
-
@bingo600 said in Suggestions for linking two pfsense setups:
If everything is running in Vlan1 , then if you connect the building switches (10Gb interlink) you will have "Short circuted" your two building lans , into the same broadcast domain Vlan1.
That's how I have it connected now, and traffic doesn't cross the two LANs because LAN1 is a different /24 than LAN2 and the default gateway is the only route - that works.
I tried setting up a route to the other gateway in pfsense through the existing LAN interface as you described in the link below. That's not working.I did temporarily add an idle interface from one of the media servers to the switch with the 10GB ports in LAN2 (one of it's 1GB ports) and it picked up an address from the DHCP server in LAN1.
I can ssh to that server and from there ssh into any other server that is on LAN2.I'm thinking maybe just plumb the spare interface on LAN1 pfsense and give it a static address on LAN2, and configure rules similiar to the interface I use for a wireless access point, but slightly less restrictive?
The physical pfsense I'm not sure can handle the backups and the in/outbound VPN.
It's rather old hardware - it works for the light duty it's doing now but I suspect pushing backups through it will be a challenge, thus the hope to pass traffic directly over the switch.The backup targets will have dedicated interfaces -- the sources of backups in LAN1 won't.
Not really sure how stressfull they will be just yet.That would/Could give you DHCP issues , as the devices in both buildings will now see answers from both DHCP servers , and will prob. take the offer that arrives first.
So we have 3 uses for the interconnect
1: Management/Admin (Low traffic volume)
2: NFS (??? Traffic Volume)
3: Backup (High traffic volume)
Re1:
Management
Quite easy to solve , described here
https://forum.netgate.com/post/944384
Re2:
NFS:
If the pfsense can handle the traffic , solved by Re1:
Re3:
Backup
If the pfsense can handle the traffic , solved by Re1:
If Not
Can your pfSense handle the traffic volume if running backups through it ?
Do you / Can you use dedicated backup interfaces on the servers ?
Then you might be able to make a separate Backup Vlan , and send the backup directly via the switches , would require the backup server also got an interface in the Backup Vlan. -
You have 2 subnets on one network segment and that's always going to give problems.
If you want to keep the sites separate then use a separate transport subnet between the two sites on it's own interface in the two firewalls. Yes, that means all traffic will go through the firewall and, yes, that probably means you probably won't see 10Gbps across it. However it will give you far better control over what can see what. You will be correctly routing between the two subnets. You can keep the local DHCP/DNS etc as you specified.
If you want traffic between the two sites not to go through the firewall you should use layer 3 switches there to route it correctly. But otherwise just use one large subnet so hosts on eash side can 'see' each other. You will have to carefully setup a single dhcp server and configure it carefully so traffic uses the local gateway. You will still have two routers/gateways on the same subnet which is bad.
I would definitely go with option 1 there unless you have layer 3 switches.
Steve
-
@MakOwner said in Suggestions for linking two pfsense setups:
@bingo600 said in Suggestions for linking two pfsense setups:
If everything is running in Vlan1 , then if you connect the building switches (10Gb interlink) you will have "Short circuted" your two building lans , into the same broadcast domain Vlan1.
That's how I have it connected now, and traffic doesn't cross the two LANs because LAN1 is a different /24 than LAN2 and the default gateway is the only route - that works.
If you have done as you said , all switches are having their ports in VLAN1.
Then traffic do cross the buildings , you are just not "listening for it".
Your DHCP setup would be a "ticking bomb" , as PC's in both buildings will get responses from both DHCP servers , and prob. take the offer that arrives first.I tried setting up a route to the other gateway in pfsense through the existing LAN interface as you described in the link below. That's not working.
You would need the "Connect net" and pfSense interfaces from both FW's in that net , for doing that correctly.
I did temporarily add an idle interface from one of the media servers to the switch with the 10GB ports in LAN2 (one of it's 1GB ports) and it picked up an address from the DHCP server in LAN1.
LAN1 DHCP server answered fastest (Proof of the DHCP BOMB)
I can ssh to that server and from there ssh into any other server that is on LAN2.
Hmmm ... Does pfSense run "Proxy arp" ?
I'm thinking maybe just plumb the spare interface on LAN1 pfsense and give it a static address on LAN2, and configure rules similiar to the interface I use for a wireless access point, but slightly less restrictive?
You could , but you still have the DHCP BOMB
Without 2 DHCP servers active , you might just pull it off.The physical pfsense I'm not sure can handle the backups and the in/outbound VPN.
It's rather old hardware - it works for the light duty it's doing now but I suspect pushing backups through it will be a challenge, thus the hope to pass traffic directly over the switch.The correct way would be a separate Vlan for backups , but also separate interfaces in the backup Vlan from each server.
The backup targets will have dedicated interfaces -- the sources of backups in LAN1 won't.
Not really sure how stressfull they will be just yet.You could give your Backup Destination Server , an interface (ip address) in both LAN1 & LAN2
That way you should be able to do backup , without passing the firewall.If you want to do it "Wrong"
.... And only have VLAN1.
You will have to solve the DHCP issue of having two active servers (DHCP Scopes) in the same Vlan. That will "Bite your behind" at some point, when a machine in Building2 (LAN2) gets a DHCP address from the server in Building1 (LAN1).But why not do it right the first time
You have been told twice by stephenw and I , that you are going down a dangerous path.
And DHCP will be your "Murphy" .../Bingo
/ -
You could pull it off by running both LAN's in VLAN1 , if you solve the DHCP issue.
And give your Backup Server an ip address in both lans.Backup:
All devices in LAN1 would then backup to the BACKUP servers LAN1 address
All devices in LAN2 would then backup to the BACKUP servers LAN2 addressDHCP:
Both DHCP servers will respond to the DHCP Broadcast , and will offer an ip address.
Your device will get an offer for both LAN1 & LAN2 , it usually picks the one that arrives first.What do you use the DHCP for ?
Servers ?? .. They could be set to use static ip info (Lan1 or Lan2).Clients
You could disable LAN2 DHCP , and let all DHCP Clients get DHCP from the LAN1 server.
Then Clients would only use internet of pfSense1 (LAN1).Technically there's nothing that prevents you from having two subnets active in one Vlan.
It's just not "Best Practice" , and a bit of a hack.But get rid of one of the dhcp scopes (servers) , or accept random Lan1/Lan2 assignments.
pfSense:
Give pfsense1 , an interface in Lan2
Give pfsense2 , an interface in Lan1@stephenw10 - Would VIP Alias work as the 2'nd interface ?
I have only used VIP Aliases once , but that might work for you.
Instead of making real interfaces.Select your "native" Lan interface , and make the "other lan" as an ip alias network
/Bingo
-
I am listening to everything you both said -- I'm trying to re-configure these without getting so many changes in at once I can't get myself back to an operational state - especially on LAN1 where I have a day job that I have to use that link.
I really would just collapse this down to one account, except that these are two DSL accounts - the download speed is barely livable, and the upload speed is... well, it's Frontier.
At least it's not AT&T, but that's all that can be said for it.
LAN1 has the single static IP and LAN2 has 5 static IPs.
(Originally, LAN2 was in a different city on FIOS... And that's why we have two accounts. Stuck with this for at least another year. Nothing creates a legacy configuration like doing something on a temporary basis, amiright?)I'll get a couple of weeks around Christmas of downtime and might look into trying to HA/load balance the two DSL accounts and just use weighting to favor gateways for each building.
I have enough lines/ports between the buildings for the direct heartbeat link between the two pfsense. That's a lot of homework.In the meantime --- I'm hoping to poke at this until I get something working.
It's not that I'm not trying to do what you both have suggested -- I am working my way through each suggestion, I just haven't hit on one that works yet.
Most of the routing information in the Netgate documentation appears to be aimed at routing in a single pfsense, so it's not always obvious that the examples are correct for my scenario.
I have yet to see any examples of dual WAN + dual pfsense setups.
(probably because it's a really bad idea...)(I thought I hit submit on this the other day... :/ )
-
To simplify this and create a functional, even if suboptimal - because traffic will go through the low end physical pfsense -- setup -- would this work:
Adding a third interface on each pfsense, creating VLAN5 across the link.
Question I have, will using (for example) 192.168.30.1 on one end and 192.168.30.2 on the other work? (apologize for the crude graphic.)
-
Yes it will work
You'd need to do a static route on each pfsense pointing to the opposite lan
via the "opposite" gateway , create gateways if needed.pfSense 1 (Vlan5) interface 192.168.30.1 /24
Create gateway 192.168.30.2
Static route 192.168.20.0/24 via GW 192.168.30.2pfSense 2 (Vlan5) interface 192.168.30.2 /24
Create gateway 192.168.30.1
Static route 192.168.10.0/24 via GW 192.168.30.1pfSense1 = the Lan 192.168.10.0/24 box
-
I'll set this up and piddle with it on weekends from there.
I'm still having cable termination issues on one of the CAT7 runs, so this gives me something to work on while I wait for parts.Hopefully soon we will get decent broadband (I'm sure you can see the sarcasm dripping).
Soon I hope to get both DSL accounts terminated in the same building and use load balancing from a single pfsense. At least it's a more widely used configuration and easier to manage. -
Loadbalancing
withon 1:1 Nat mapped to servers ...
That's another can of worms
Especially if done correct w. dual ISP's.Get your new design up first ....
And you'd already be able to "share" the lines after that.Do you really have a /22 on @ ?
/Bingo
I'm still wondering why no thumbs up on any answers in this thread. -
Do you really have a /22 on @ ?
/22 on the account for LAN1 and a /29 for account on LAN2.
Anyone that has ever dealt with Frontier DSL in areas where they took over from Verizon will know the special hell that is.The modem itself is DHCP mode, and they have managed to lock out ever modem-only DSL device so that it won't work on their service.
So you get stuck with this modem/router/firewall that is impossible to do real bridging - that you have to pay rent.
And since Frontier has gone into bankruptcy it is next to impossible to get the actual network support teams on a support call. -
So ... I removed all the VLAN configurations from everythin, all switches and ports are using the defaut-vlan and no parts are set to accept any traffic.
I configure an interface on each pfsense connected directly to the netgear switches for the link between buildings (removing any possible issues of the switches being the issue -- the other switches used in this setup are different generations of Dell Powerconnect switches).
192.168.30.1 on the LAN1 interface, 192.168.30.2 on the LAN2 interface with routes created
I can ping the local 192.168.30/ interface from anywhere on either end, but I can't ping the gateway (the interface on the opposite pfsense).
Can't get any traffic through the link at all.I can see port stats increase on the net gear management interface from each end though ...
-
EDIT: You did make "Allow rules" on the new pfSense VLAN2 interfaces (both ends) , else everything would be blocked.
If yes , read on , else make them , and retest.Did you connect the pfSense Lan ports directly to the Netgear's ?
Did you do anything with Vlans on the pfSense (you should'nt) , right now. You should just create normal interfaces.How are the pfSense VLAN2 switchports on the Netgear's defined ?
They should be Untagged members of Vlan2I have no experience w. Netgear
If going "simple" , the Netgear Site to Site interfaces should be untagged members of VLAN2 , too.Later :
I would prob. make them tagged members of VLAN2 , as that would open up for defining more vlans to be transported over the site-to-site interlink.Make sure the above is done first (before the debugging , below)
Debugging:
If you make an extra Untagged Vlan2 port on both of the Netgears , you should be able to connect a PC to that port (set it to Ie. 192.168.30.10/24) , then test if you can ping the "Local pfSense interface" , If yes , then VLAN2 is working (on the local side).
You could do the same on the other site , if working , then VLAN2 is working (on the other side).Then we know the error is on the site-to-site interlink (Maybe those ports are not member of VLAN2)
How did you define the interface on your Virtual pfSense, did you have a spare IF there too.
It is important for me to now if you are using (defining) vlans on the pfSense , or you only use "Plain ethernet interfaces"
/Bingo
Tip: You should give your Netgears , management ip addreses in VLAN2 , just pick a free ip in the /24 range.
-
I think the issue may be in the rules for the interface on LAN2.
I reused what was DMZ interface and manually created rules.This is what LAN1 side rules look like:
(I think this should really be restricted to LAB to OFFICE and OFFICE to LAN to prevent unexpected routes out to the internet.)
Yes the spare interface is connected directly to the Netgear switches.
Everything is physically separated now - so long as traffic will pass over the pfsense from one interface to another, traffic should flow, and DHCP on both ends is restricted to it's own subnet.No VLANs are configured (Let me rephrase that -- nothing other than default) anywhere on the Netgear or pfsense, and none of the switches that had ports in other than default VLAN are in use.
-
You should not indicate VLAN2 on your drawing if the netgears are using default (witch is prob Vlan1) , but it should work fine , using Vlan1 on the netgears too , as the pfSense (Layer3) is preventing the Vlans (Layer2) , to be propagated to the other intefaces.
So it seems like it is firewall rules that are blocking your pings.
/Bingo
-
If you want to do ping tests from the pfsenses on the Netgear link
Remember to allow the 192.168.30.0/24 net too , on both sides.If the pfSense do the ping (to the other gw) , it will (default) use the source address of the "local gw interface".
Edit. Since i have no clue what ip ranges LAB,LAB & Office are
The rules doesn't say much to me./Bingo
-
ah, yeah, missed that VLAN edit.
I just double checked and neither Netgear has anything but default VLAN.LAN1 - 192.168.10.0/24
LAN2 - 192.168.20/24
OFFICE interface is 192.168.30.1 on pfsense in LAN1
LAB interface is 192.168.30.2 on pfsense in LAN2the ruleset on LAN1
This is the ruleset on LAN2
Looking at this I see that on some of the rules only TCP is allowed, so I went back and changed all of the any protocol - "IPV4 *".
No difference :/
I can see the routing table on the pfsense and I can't see why LAN1 can't ping 192.168.30.2 -- unless I need to add the gateway directly to that interface rather than just depend on the static route?
-
You need to allow ICMP (ping) on the new pfsense intefaces (Netgear) : Source 192.168.30.0/24 to (i'd prob do) ANY
On both sidesRemember now the trafic from lanx also passes the site-to-site interfaces on the pfsenses , and must be allowed accordingly.
Do you see any deny's in the pfsenses ??
What does a Diagnostics -> ARP Table show (entries beginning with 192.168.30.x)
