Suggestions for linking two pfsense setups
-
I am listening to everything you both said -- I'm trying to re-configure these without getting so many changes in at once I can't get myself back to an operational state - especially on LAN1 where I have a day job that I have to use that link.
I really would just collapse this down to one account, except that these are two DSL accounts - the download speed is barely livable, and the upload speed is... well, it's Frontier.
At least it's not AT&T, but that's all that can be said for it.
LAN1 has the single static IP and LAN2 has 5 static IPs.
(Originally, LAN2 was in a different city on FIOS... And that's why we have two accounts. Stuck with this for at least another year. Nothing creates a legacy configuration like doing something on a temporary basis, amiright?)I'll get a couple of weeks around Christmas of downtime and might look into trying to HA/load balance the two DSL accounts and just use weighting to favor gateways for each building.
I have enough lines/ports between the buildings for the direct heartbeat link between the two pfsense. That's a lot of homework.In the meantime --- I'm hoping to poke at this until I get something working.
It's not that I'm not trying to do what you both have suggested -- I am working my way through each suggestion, I just haven't hit on one that works yet.
Most of the routing information in the Netgate documentation appears to be aimed at routing in a single pfsense, so it's not always obvious that the examples are correct for my scenario.
I have yet to see any examples of dual WAN + dual pfsense setups.
(probably because it's a really bad idea...)(I thought I hit submit on this the other day... :/ )
-
To simplify this and create a functional, even if suboptimal - because traffic will go through the low end physical pfsense -- setup -- would this work:
Adding a third interface on each pfsense, creating VLAN5 across the link.
Question I have, will using (for example) 192.168.30.1 on one end and 192.168.30.2 on the other work? (apologize for the crude graphic.)
-
Yes it will work
You'd need to do a static route on each pfsense pointing to the opposite lan
via the "opposite" gateway , create gateways if needed.pfSense 1 (Vlan5) interface 192.168.30.1 /24
Create gateway 192.168.30.2
Static route 192.168.20.0/24 via GW 192.168.30.2pfSense 2 (Vlan5) interface 192.168.30.2 /24
Create gateway 192.168.30.1
Static route 192.168.10.0/24 via GW 192.168.30.1pfSense1 = the Lan 192.168.10.0/24 box
-
I'll set this up and piddle with it on weekends from there.
I'm still having cable termination issues on one of the CAT7 runs, so this gives me something to work on while I wait for parts.Hopefully soon we will get decent broadband (I'm sure you can see the sarcasm dripping).
Soon I hope to get both DSL accounts terminated in the same building and use load balancing from a single pfsense. At least it's a more widely used configuration and easier to manage. -
Loadbalancing
withon 1:1 Nat mapped to servers ...
That's another can of worms
Especially if done correct w. dual ISP's.Get your new design up first ....
And you'd already be able to "share" the lines after that.Do you really have a /22 on @ ?
/Bingo
I'm still wondering why no thumbs up on any answers in this thread. -
Do you really have a /22 on @ ?
/22 on the account for LAN1 and a /29 for account on LAN2.
Anyone that has ever dealt with Frontier DSL in areas where they took over from Verizon will know the special hell that is.The modem itself is DHCP mode, and they have managed to lock out ever modem-only DSL device so that it won't work on their service.
So you get stuck with this modem/router/firewall that is impossible to do real bridging - that you have to pay rent.
And since Frontier has gone into bankruptcy it is next to impossible to get the actual network support teams on a support call. -
So ... I removed all the VLAN configurations from everythin, all switches and ports are using the defaut-vlan and no parts are set to accept any traffic.
I configure an interface on each pfsense connected directly to the netgear switches for the link between buildings (removing any possible issues of the switches being the issue -- the other switches used in this setup are different generations of Dell Powerconnect switches).
192.168.30.1 on the LAN1 interface, 192.168.30.2 on the LAN2 interface with routes created
I can ping the local 192.168.30/ interface from anywhere on either end, but I can't ping the gateway (the interface on the opposite pfsense).
Can't get any traffic through the link at all.I can see port stats increase on the net gear management interface from each end though ...
-
EDIT: You did make "Allow rules" on the new pfSense VLAN2 interfaces (both ends) , else everything would be blocked.
If yes , read on , else make them , and retest.Did you connect the pfSense Lan ports directly to the Netgear's ?
Did you do anything with Vlans on the pfSense (you should'nt) , right now. You should just create normal interfaces.How are the pfSense VLAN2 switchports on the Netgear's defined ?
They should be Untagged members of Vlan2I have no experience w. Netgear
If going "simple" , the Netgear Site to Site interfaces should be untagged members of VLAN2 , too.Later :
I would prob. make them tagged members of VLAN2 , as that would open up for defining more vlans to be transported over the site-to-site interlink.Make sure the above is done first (before the debugging , below)
Debugging:
If you make an extra Untagged Vlan2 port on both of the Netgears , you should be able to connect a PC to that port (set it to Ie. 192.168.30.10/24) , then test if you can ping the "Local pfSense interface" , If yes , then VLAN2 is working (on the local side).
You could do the same on the other site , if working , then VLAN2 is working (on the other side).Then we know the error is on the site-to-site interlink (Maybe those ports are not member of VLAN2)
How did you define the interface on your Virtual pfSense, did you have a spare IF there too.
It is important for me to now if you are using (defining) vlans on the pfSense , or you only use "Plain ethernet interfaces"
/Bingo
Tip: You should give your Netgears , management ip addreses in VLAN2 , just pick a free ip in the /24 range.
-
I think the issue may be in the rules for the interface on LAN2.
I reused what was DMZ interface and manually created rules.This is what LAN1 side rules look like:
(I think this should really be restricted to LAB to OFFICE and OFFICE to LAN to prevent unexpected routes out to the internet.)
Yes the spare interface is connected directly to the Netgear switches.
Everything is physically separated now - so long as traffic will pass over the pfsense from one interface to another, traffic should flow, and DHCP on both ends is restricted to it's own subnet.No VLANs are configured (Let me rephrase that -- nothing other than default) anywhere on the Netgear or pfsense, and none of the switches that had ports in other than default VLAN are in use.
-
You should not indicate VLAN2 on your drawing if the netgears are using default (witch is prob Vlan1) , but it should work fine , using Vlan1 on the netgears too , as the pfSense (Layer3) is preventing the Vlans (Layer2) , to be propagated to the other intefaces.
So it seems like it is firewall rules that are blocking your pings.
/Bingo
-
If you want to do ping tests from the pfsenses on the Netgear link
Remember to allow the 192.168.30.0/24 net too , on both sides.If the pfSense do the ping (to the other gw) , it will (default) use the source address of the "local gw interface".
Edit. Since i have no clue what ip ranges LAB,LAB & Office are
The rules doesn't say much to me./Bingo
-
ah, yeah, missed that VLAN edit.
I just double checked and neither Netgear has anything but default VLAN.LAN1 - 192.168.10.0/24
LAN2 - 192.168.20/24
OFFICE interface is 192.168.30.1 on pfsense in LAN1
LAB interface is 192.168.30.2 on pfsense in LAN2the ruleset on LAN1
This is the ruleset on LAN2
Looking at this I see that on some of the rules only TCP is allowed, so I went back and changed all of the any protocol - "IPV4 *".
No difference :/
I can see the routing table on the pfsense and I can't see why LAN1 can't ping 192.168.30.2 -- unless I need to add the gateway directly to that interface rather than just depend on the static route?
-
You need to allow ICMP (ping) on the new pfsense intefaces (Netgear) : Source 192.168.30.0/24 to (i'd prob do) ANY
On both sidesRemember now the trafic from lanx also passes the site-to-site interfaces on the pfsenses , and must be allowed accordingly.
Do you see any deny's in the pfsenses ??
What does a Diagnostics -> ARP Table show (entries beginning with 192.168.30.x)
-
the LAN1 pfsense ARP table:
Both Netgear switches are visible here too, and accessible on LAN1,
So on LAN1 traffic is crossing the .30.1 interface and coming back.Something on the pfsense on LAN2 is blocking I suppose.
I just filtered the firewall log for any activity with a destination address of 192.168.30.2 (the interface on LAN2 which I have been pinging quite frequently to test access) and there are no entries.
Won't the default LAN allow to any rule cover traffic from 192.168.10 or 192.168.20 to any other subnet or network? Seems to as the management interfaces for the netgear switches are working from LAN1.
And I just checked, I can ping both from LAN1.Perhaps I should just delete the interface on LAN2 and start over ...
-
If you can ping both netgears from the "Lan1" pfSense , that's good news.
Then the site-to-site link is working.
As the arp says ... something is fishy with 192.168.30.2
Maybe delete and redo it , would be a good start.
Is the 192.168.30.2 end the VM pfSense ?
/Bingo
-
IPsec and failover GW's??
-
@cool_corona -- it's a really squirrely dual ISP setup with endpoints in two buildings. One /22 and one /29 public network.
@bingo600 - this is getting a little nuts.
LAN2 is the virtual pfsense.
I removed the interlink interface from LAN2, all of the routes and gateways, etc..
Did a physical check on the ESXi box to validate that the NIC I have assigned IS in fact the interface connected to the NetGear switch (validated the port link status, noted the MAC address and MAC address assignment to the VM.)
Restarted pfsense.
Reassigned the interface, rebuilt the gateways and routes.
Still nothing gets past the interface on the pfsense in LAN2.I even opened up the rules on the interface in LAN2 to allow traffic form that interface to any any in.
I added lan to link and link to lan rules on the LAN interface (although I don't think that's relavant for the arp table -- the arp table shows up the same on both ends -- the other end is (incomplete). -
Do you have a PC you can connect to the Netgear on Site 2 ?
And give it ie. 192.168.30.10
Then you could ping a little around with that , if the pc can ping pfsense1 Site1 (192.168.30.1) , and not pfsense site2 (192.168.30.2). Then you have some kind of layer2 "challenge", between the netgear on site2 , and the pfsense interface on site2.Even if you block everything on the pfSense , the interface MAC should still show up in the ARP Table (with a valid mac)
Edit: You should be able to see the pfSense MAC in the Netgear's MAC table , on Site1.
That should be the same for Site2 (the VM/pfSense mac) (but i expect it not to be there)
Sometimes you have to do a little traffic (ie pings) , before the mac address appears in the "tables" on switch or (arp table on pfsense)
/Bingo
-
@bingo600
I have a system with multiple interfaces and I set up a second, static interface on 192.168.30.20 and plugged it into the Netgear switch on LAN2.pfsense on LAN1 can see it properly in the ARP table but there is no response from the host on LAN1 -- ping or ssh, even a ping directly from the 192.168.30.1 interface.
(Keeping in mind the primary interface on this system and it's default routes all come from LAN2 so ...)
I have spare laptop I'll plug it in and see what happens.
-
I see a successful ping from pfSense Lan1 , to the ".20 - device" connected to Netgear (Site2).
That verifies that your site-to-site link works (netgears are transporting data).
The issue must be with the interface/connection from pfSense2 to Netgear2.
