Suggestions for linking two pfsense setups
-
If the ARP (MAC) is good , and you can ping 192.168.30.1 and .2 from both sides , the Layer2 issue seems to be solved.
What does the gateway monitoring say (remember you prob have to allow ICMP to make the monitoring happy)
What does Diagnostic -> Routes show (both ends)
Do you see any blocks in the pfSense Log ?
Routing was described here
https://forum.netgate.com/post/945282/Bingo
-
I'm seeing ICMP being dropped in the firewall logs for 192.168.30.1 on the .1 side:
Nov 15 14:43:35 LAB Default deny rule IPv4 (1000000103) 192.168.30.2 192.168.30.1 ICMPI don't see any rules that would trigger this, and I have added these rules on both sides:
This is the rule set from LAN1 for .30.1 -- I can't access LAN2 from here to capture the rules but they are identical on the reverse end -- using the other interface of course.
-
Hmm.. I have traffic crossing the link now, but I had to create a set of far too lax rules on both sides for the .30 interfaces.
LAN1 side:
LAN2 side:
No collisions so far on DHCP.
Seems everyone is getting IPs assigned via the closest interface, which works well.
Not sure how stable this is and it leaves both ends wide open.Not sure I undertsand why the two rules have traffic on one side but not the other -- see states column?
-
1 : Static Routing
You will have to tell a pfSense about every net (on the other side) you want reachable , including the wifi net.- Firewall
If/when you "just" ping from one pfSense to the other pfSense , the pfSense is creating the traffic on the "nearest interface". This means a packet from ie. 192.168.30.1 would flow to 192.168.30.2 (or any other Lan).
Don't forget to permit 192.168.30.0/24 to some extent , at least ICMP (pings).
Or your gateway monitoring will "barf" at you.ICMP is a protocol like TCP or UDP (select it there) , in a "friendly" environment i usually allow ICMP any any
You have to consider you now have 5 nets.
pfSense1 : Lan1 + Wifi1 + Connect
pfSense2: Lan2 + WiFi2 + ConnectPackets that crosses buildings will have to be permittet on the Connect interface , in the opposite building end , in order to be allowed (received).
Re: States & traffic
That is not unusual , think about how traffic flows , and how much of it./Bingo
- Firewall
-
The wifi nets are normally locked out of the LAN.
They get firewalled access t the internet but no traffic from wireless enters the LANs.
I had LAN1 open temporarily to use a wireless laptop for access to pfsense in LAN1, but that needs to be locked back down.I expected the explicit "LAN to alternate network" and "alternate network to LAN" set of rules to work, whereas I had to just create a single rule that essentially allowed all traffic across the link, not just the traffic from the defined network from the opposite pfsense.
LAN1 rules on the interlink interface:
LAN2 rules on the interlink interface:
Rules that didn't work -- I used net rules and specified the appropriate /24 of the LAN on the opposite pfsense -- one inbound, one outbound.
Adding the third rule (top on each LAN) opened things up.
I feel like there is probably a better way to do this?
I'll work on getting the LAGs and VLANs functioning across the switches first I suppose. -
Rerember pfSense default filters "Inbound" , meaning :
You can filter traffic from entering via an interface.
pfSense default permits NOTHING from entering via an interface (besides mgt on/via the LAN interface) so you WILL have to do permits on the interfaces to allow traffic , to enter the firewall.That also goes for the "site to site" interlink interfaces (192.168.30.1 & 2).
The top rule on the Lan1 / Lan2 : ipv4 any any
Effectively sets the other two rules out of "play" , as the top rule would match anything , and no traffic would hit the below rules.First match : Proccesses the packet.
If i were you i'd start with a :
Permit ICMP any any and a Permit IPV4 any any on both "Site to site" interfaces (effectively permit anything to pass between buildings) (if it has been allowed into the firewall. by the LAN/WAN rules)
That way the rules on the LAN rules could be made "as you are used to".
Then after you get the felling for the Lan rules , you could narrow down the site-to-site rules.
/Bingo
-
Need clarification here.
Have you actually added the VLAN2 interface to each pfSense to use as the transport link?
What have you named that interface? Is that LAB?
You can't put hosts in a transport subnet without creating an asymmetric route which is why your test with a laptop earlier failed.
Can we see a diagram showing exactly how this is connected? With interface names and IPs/subnet?
Steve
-
AFAIK
This ts the current setup
https://forum.netgate.com/post/945569BUT every switch has default settings (All in VLAN1) the site-to-site L2 is also VLAN1 on the 10G-Netgear "building transport switches"
All is transported (connected) via L3 (pfSense interfaces)
No tagging on the pfSense , all is "Pure ethernet" going to the switches
We got it "all up & running" , now only pfSense rules need to be made (adjusted)
WiFi nets should not be static (site2site) routed , as they are "only allowed to go to the internet" on their respective site ... I hope i got that right
The OP mentioned one ISSUE - If he adds a 4'th interface on the VM based pfSense , it will not forward any traffic. Only the first 3 IF's works.
I think we fought that issue earlier , and yep ... That (4'th) IF would not respond to ARP, or show up in the "MAC table" on the switches.
I still think the OP runs w. just 3 IF's on the VM pfSense (No WiFi IF)
/Bingo
-
This is what it looks like:
The interface on pfsense on the left for the link is "Lab - 192.168.30.1"
The interface on pfsense on the right for the link is "Office - 192.168.30.2"In LAN1 I'm using 4 interfaces and the wireless works -- it's running 2.4.4-p3.
In LAN2 it -was running latest 2.4.5 but that's where the 4th interface wouldn't work, so I punted and reinstalled again this time to 2.4.4 and updated to -p3.
I swapped around interfaces in the several re-installs and the NIC appears to be working fine...It's a VM so I'll snapshot it and try the upgrade to 2.4.5, but really as badly as unbound crashes on 2.4.5. I think I want to go see what security fixes are in there beforehand.
Everything right now is in the default VLAN.
I'm using two identical NetGear GS110EMX switches for the 10GB link, and (really old) Dell Powerconnect switches on each end. 5224 on the office end and a 2748 on the Lab end.I had suspected at first possibly the NetGears and Dells weren't really exchanging VLAN/LAG info properly, but based on their interfaces it looks like they are picking up the port configuration on the fly if I configure the Netgear..
But as I said, for right now, it's all open and default.
The wireless networks can't access but the internet -- I have confirmed that on the office side, later today I have to go check the access point for the Lab and ensure everything is right there.
-
@MakOwner said in Suggestions for linking two pfsense setups:
It's a VM so I'll snapshot it and try the upgrade to 2.4.5, but really as badly as unbound crashes on 2.4.5. I think I want to go see what security fixes are in there beforehand.
You could use the Service watchdog for that
I see (saw) the occationally unbound crash , on previous 2.4.4's , and had service watchdog restart it automatically.I had suspected at first possibly the NetGears and Dells weren't really exchanging VLAN/LAG info properly, but based on their interfaces it looks like they are picking up the port configuration on the fly if I configure the Netgear..
I do not hope the NetGears respond to anything (VLAN/LAG) on the DELL's right now , as they are L3 separated.
/Bingo
-
@bingo600 said in Suggestions for linking two pfsense setups:
I do not hope the NetGears respond to anything (VLAN/LAG) on the DELL's right now , as they are L3 separated.
/Bingo
Yea, me too!
They are both doing what they should be doing at this point -- just being dumb switches.I do have a Dell 6248 that I can't get to reset so I can't get into the CLI or the web interface.
Whole other issue. -
Ah I see, no VLANs just direct intrefaces. Well that removes that potential issue.
So "LAN1" is the Office side pfSense and "LAN2" is the Lab side pfSense?
And link interfaces are named according to where the connect to, so the opposite of where they actually are?
I think the problem here is that you are using system aliases like
LAN netand they are invalid on the side you're applying them.You have rules using
LAN netas a source and destination but that can only be right on one of those rules.On LAN1 you need rules on the link interface for:
192.168.20.0/24 toLAN netany protocol
LAB nettoLAB addressicmp (to allow gateway monitoring)On LAN2 you need rules on the link interface for:
192.168.10.0/24 toLAN netany protocol
OFFICE nettoOFFICE addressicmpSteve
-
I did exactly that -- but it didn't work so I added the second rule that reverses that logic -- and it still didn't work.
Adding the any to any rule made it work.
I didn't try reversing rules before going to the any to any rule though .... -
Do you have the link interfaces assigned with the gateways actually on them?
If so auto outbound NAT rules will be applied as though they are a WAN and the other side will see all traffic coming from the Link IP.
Steve
-
@stephenw10
No, the link interfaces have "None" for gateways.
(Although I did try that at point before I re installed pfsense on LAN2 ...) -
Ok, then I would start a continuous ping on one side and check the state tables and firewall logs on both firewalls to see where that traffic going.
Ultimately run a pcap on each interface involved to check the packets are arrving/leaving at each point.
Steve
-
@MakOwner
Here is a "Crash course" in Vlan tagging on pfSense , if ever needed.
https://forum.netgate.com/post/944381I hope you find it usable, even though i would not recommend you begin on that stuff right now. Tame your current beast, before releasing a new.
/Bingo
-
So, I thought since I have sort of got this working well enough thanks to all your help, I'd share what I have done -- so far anyway. Thanks for all the help!
I have had to revert to 2.4.4-p3 on both ends.
2.4.5 doesn't seem to play well at this point.
Gateway monitoring seem to be the downfall for some reason.
I'll wait for another update before trying again.I currently have a physical layout like this:
DHCP servers are active with very limited ranges on each interface.
The office interface handles the DHCP interface handles the .30/24 on the link between buildings and provides the addresses for the management interfaces on the two switches.
Each DHCP server is configured to serve addresses within a specified range and each /24 has a different range just for conveniance. .10/24 uses .110-.120, .20/24 uses .120-.130, .30/24 uses .130-.140 for example.
Not a single DHCP collision or bomb - yet, anyway.
All active physical systems use static DHCP assignment outside the assignment range so that it becomes very obvious if anything goes south.Al internet bound traffic in the Lab goes out the .20/24 network, all internet bound traffic from the Office goes out the .10/24 network.
Wireless is isolated on both sides to just internet access.
I added the .30/24 interface of the opposite system as a gateway on each pfsense.
I added static routes to .10/24 and .20/24 networks as appropriate.
On the .30/24 interface on each pfsense I added an any-to-any firewall rule.That let traffic pass both directions.
From there I began working on DNS.
The DNS resolver in 2.4.4 doesn't quite behave as I expected.Still piddling with that.
After DNS is working like I expect I'll move on to configuring the direct link.
-
@MakOwner said in Suggestions for linking two pfsense setups:
So, I thought since I have sort of got this working well enough thanks to all your help, I'd share what I have done -- so far anyway. Thanks for all the help!
I have had to revert to 2.4.4-p3 on both ends.
2.4.5 doesn't seem to play well at this point.
Gateway monitoring seem to be the downfall for some reason.
I'll wait for another update before trying again.Why does GW monitoring pose a problem ?
It's usually just a "Ping of the GW".DHCP servers are active with very limited ranges on each interface.
The office interface handles the DHCP interface handles the .30/24 on the link between buildings and provides the addresses for the management interfaces on the two switches.
Each DHCP server is configured to serve addresses within a specified range and each /24 has a different range just for conveniance. .10/24 uses .110-.120, .20/24 uses .120-.130, .30/24 uses .130-.140 for example.
Not a single DHCP collision or bomb - yet, anyway.
All active physical systems use static DHCP assignment outside the assignment range so that it becomes very obvious if anything goes south.Special way to "reckognize" which DHCP server hands out the ip.
But if it works for you .....
Once you trust DHCP , you could move to something more "default".I usually let my dhcp start at .129 .. 250
And uses 25..125 for static assignments (1..24) are for infrastructure like routers , switches etc.I added the .30/24 interface of the opposite system as a gateway on each pfsense.
I added static routes to .10/24 and .20/24 networks as appropriate.Great ... Do you understand why you have to do that now ?
Tell one pfSense how to "reach" a remote lan on the "other".On the .30/24 interface on each pfsense I added an any-to-any firewall rule.
That is "fine" , basically making the "Connect net" transparent.
You could fine tune that , later if ever needed.The .30/24 rules , are like all other "normal" pfSense interface rules "Enter" rules :
See it as: "This permitted traffic" is allowed to "enter" the pfSense-box via "this interface" , everything not permitted is blocked , and will NOT enter the receiving pfSense box.From there I began working on DNS.
The DNS resolver in 2.4.4 doesn't quite behave as I expected.What is your issue ?
Still Unbound crashes ?
Did you try the service watchdog ?After DNS is working like I expect I'll move on to configuring the direct link.
???
Remember when making changes to :
Backup pfSense configs often.
Don't make 10 changes at the same time.All in all a nice excersise , you certainly got your hands dirty.
Well done , especially for not giving up./Bingo
Ps:
If you ever want to"do it right"more flexible ,youwe should enable multi-vlan on ALL your switches. That would give you the flexibility to make ANY switchport in any building , a member of "ANY vlan".
Ie. a server in building-1 , would be seen as a member of the building-2 Lan.But that would require that you are reading up on "tagging" on all your switches , including the old "access" switches you use.
And start playing a little with vlans & tagging , before starting that up.
