Suggestions for linking two pfsense setups
-
Need clarification here.
Have you actually added the VLAN2 interface to each pfSense to use as the transport link?
What have you named that interface? Is that LAB?
You can't put hosts in a transport subnet without creating an asymmetric route which is why your test with a laptop earlier failed.
Can we see a diagram showing exactly how this is connected? With interface names and IPs/subnet?
Steve
-
AFAIK
This ts the current setup
https://forum.netgate.com/post/945569BUT every switch has default settings (All in VLAN1) the site-to-site L2 is also VLAN1 on the 10G-Netgear "building transport switches"
All is transported (connected) via L3 (pfSense interfaces)
No tagging on the pfSense , all is "Pure ethernet" going to the switches
We got it "all up & running" , now only pfSense rules need to be made (adjusted)
WiFi nets should not be static (site2site) routed , as they are "only allowed to go to the internet" on their respective site ... I hope i got that right
The OP mentioned one ISSUE - If he adds a 4'th interface on the VM based pfSense , it will not forward any traffic. Only the first 3 IF's works.
I think we fought that issue earlier , and yep ... That (4'th) IF would not respond to ARP, or show up in the "MAC table" on the switches.
I still think the OP runs w. just 3 IF's on the VM pfSense (No WiFi IF)
/Bingo
-
This is what it looks like:
The interface on pfsense on the left for the link is "Lab - 192.168.30.1"
The interface on pfsense on the right for the link is "Office - 192.168.30.2"In LAN1 I'm using 4 interfaces and the wireless works -- it's running 2.4.4-p3.
In LAN2 it -was running latest 2.4.5 but that's where the 4th interface wouldn't work, so I punted and reinstalled again this time to 2.4.4 and updated to -p3.
I swapped around interfaces in the several re-installs and the NIC appears to be working fine...It's a VM so I'll snapshot it and try the upgrade to 2.4.5, but really as badly as unbound crashes on 2.4.5. I think I want to go see what security fixes are in there beforehand.
Everything right now is in the default VLAN.
I'm using two identical NetGear GS110EMX switches for the 10GB link, and (really old) Dell Powerconnect switches on each end. 5224 on the office end and a 2748 on the Lab end.I had suspected at first possibly the NetGears and Dells weren't really exchanging VLAN/LAG info properly, but based on their interfaces it looks like they are picking up the port configuration on the fly if I configure the Netgear..
But as I said, for right now, it's all open and default.
The wireless networks can't access but the internet -- I have confirmed that on the office side, later today I have to go check the access point for the Lab and ensure everything is right there.
-
@MakOwner said in Suggestions for linking two pfsense setups:
It's a VM so I'll snapshot it and try the upgrade to 2.4.5, but really as badly as unbound crashes on 2.4.5. I think I want to go see what security fixes are in there beforehand.
You could use the Service watchdog for that
I see (saw) the occationally unbound crash , on previous 2.4.4's , and had service watchdog restart it automatically.I had suspected at first possibly the NetGears and Dells weren't really exchanging VLAN/LAG info properly, but based on their interfaces it looks like they are picking up the port configuration on the fly if I configure the Netgear..
I do not hope the NetGears respond to anything (VLAN/LAG) on the DELL's right now , as they are L3 separated.
/Bingo
-
@bingo600 said in Suggestions for linking two pfsense setups:
I do not hope the NetGears respond to anything (VLAN/LAG) on the DELL's right now , as they are L3 separated.
/Bingo
Yea, me too!
They are both doing what they should be doing at this point -- just being dumb switches.I do have a Dell 6248 that I can't get to reset so I can't get into the CLI or the web interface.
Whole other issue. -
Ah I see, no VLANs just direct intrefaces. Well that removes that potential issue.
So "LAN1" is the Office side pfSense and "LAN2" is the Lab side pfSense?
And link interfaces are named according to where the connect to, so the opposite of where they actually are?
I think the problem here is that you are using system aliases like
LAN netand they are invalid on the side you're applying them.You have rules using
LAN netas a source and destination but that can only be right on one of those rules.On LAN1 you need rules on the link interface for:
192.168.20.0/24 toLAN netany protocol
LAB nettoLAB addressicmp (to allow gateway monitoring)On LAN2 you need rules on the link interface for:
192.168.10.0/24 toLAN netany protocol
OFFICE nettoOFFICE addressicmpSteve
-
I did exactly that -- but it didn't work so I added the second rule that reverses that logic -- and it still didn't work.
Adding the any to any rule made it work.
I didn't try reversing rules before going to the any to any rule though .... -
Do you have the link interfaces assigned with the gateways actually on them?
If so auto outbound NAT rules will be applied as though they are a WAN and the other side will see all traffic coming from the Link IP.
Steve
-
@stephenw10
No, the link interfaces have "None" for gateways.
(Although I did try that at point before I re installed pfsense on LAN2 ...) -
Ok, then I would start a continuous ping on one side and check the state tables and firewall logs on both firewalls to see where that traffic going.
Ultimately run a pcap on each interface involved to check the packets are arrving/leaving at each point.
Steve
-
@MakOwner
Here is a "Crash course" in Vlan tagging on pfSense , if ever needed.
https://forum.netgate.com/post/944381I hope you find it usable, even though i would not recommend you begin on that stuff right now. Tame your current beast, before releasing a new.
/Bingo
-
So, I thought since I have sort of got this working well enough thanks to all your help, I'd share what I have done -- so far anyway. Thanks for all the help!
I have had to revert to 2.4.4-p3 on both ends.
2.4.5 doesn't seem to play well at this point.
Gateway monitoring seem to be the downfall for some reason.
I'll wait for another update before trying again.I currently have a physical layout like this:
DHCP servers are active with very limited ranges on each interface.
The office interface handles the DHCP interface handles the .30/24 on the link between buildings and provides the addresses for the management interfaces on the two switches.
Each DHCP server is configured to serve addresses within a specified range and each /24 has a different range just for conveniance. .10/24 uses .110-.120, .20/24 uses .120-.130, .30/24 uses .130-.140 for example.
Not a single DHCP collision or bomb - yet, anyway.
All active physical systems use static DHCP assignment outside the assignment range so that it becomes very obvious if anything goes south.Al internet bound traffic in the Lab goes out the .20/24 network, all internet bound traffic from the Office goes out the .10/24 network.
Wireless is isolated on both sides to just internet access.
I added the .30/24 interface of the opposite system as a gateway on each pfsense.
I added static routes to .10/24 and .20/24 networks as appropriate.
On the .30/24 interface on each pfsense I added an any-to-any firewall rule.That let traffic pass both directions.
From there I began working on DNS.
The DNS resolver in 2.4.4 doesn't quite behave as I expected.Still piddling with that.
After DNS is working like I expect I'll move on to configuring the direct link.
-
@MakOwner said in Suggestions for linking two pfsense setups:
So, I thought since I have sort of got this working well enough thanks to all your help, I'd share what I have done -- so far anyway. Thanks for all the help!
I have had to revert to 2.4.4-p3 on both ends.
2.4.5 doesn't seem to play well at this point.
Gateway monitoring seem to be the downfall for some reason.
I'll wait for another update before trying again.Why does GW monitoring pose a problem ?
It's usually just a "Ping of the GW".DHCP servers are active with very limited ranges on each interface.
The office interface handles the DHCP interface handles the .30/24 on the link between buildings and provides the addresses for the management interfaces on the two switches.
Each DHCP server is configured to serve addresses within a specified range and each /24 has a different range just for conveniance. .10/24 uses .110-.120, .20/24 uses .120-.130, .30/24 uses .130-.140 for example.
Not a single DHCP collision or bomb - yet, anyway.
All active physical systems use static DHCP assignment outside the assignment range so that it becomes very obvious if anything goes south.Special way to "reckognize" which DHCP server hands out the ip.
But if it works for you .....
Once you trust DHCP , you could move to something more "default".I usually let my dhcp start at .129 .. 250
And uses 25..125 for static assignments (1..24) are for infrastructure like routers , switches etc.I added the .30/24 interface of the opposite system as a gateway on each pfsense.
I added static routes to .10/24 and .20/24 networks as appropriate.Great ... Do you understand why you have to do that now ?
Tell one pfSense how to "reach" a remote lan on the "other".On the .30/24 interface on each pfsense I added an any-to-any firewall rule.
That is "fine" , basically making the "Connect net" transparent.
You could fine tune that , later if ever needed.The .30/24 rules , are like all other "normal" pfSense interface rules "Enter" rules :
See it as: "This permitted traffic" is allowed to "enter" the pfSense-box via "this interface" , everything not permitted is blocked , and will NOT enter the receiving pfSense box.From there I began working on DNS.
The DNS resolver in 2.4.4 doesn't quite behave as I expected.What is your issue ?
Still Unbound crashes ?
Did you try the service watchdog ?After DNS is working like I expect I'll move on to configuring the direct link.
???
Remember when making changes to :
Backup pfSense configs often.
Don't make 10 changes at the same time.All in all a nice excersise , you certainly got your hands dirty.
Well done , especially for not giving up./Bingo
Ps:
If you ever want to"do it right"more flexible ,youwe should enable multi-vlan on ALL your switches. That would give you the flexibility to make ANY switchport in any building , a member of "ANY vlan".
Ie. a server in building-1 , would be seen as a member of the building-2 Lan.But that would require that you are reading up on "tagging" on all your switches , including the old "access" switches you use.
And start playing a little with vlans & tagging , before starting that up.
