Suggestions for linking two pfsense setups
-
I see a successful ping from pfSense Lan1 , to the ".20 - device" connected to Netgear (Site2).
That verifies that your site-to-site link works (netgears are transporting data).
The issue must be with the interface/connection from pfSense2 to Netgear2. -
I enabled the DHCP server on the 192.68.30.1 interface in LAN1.
Connected a laptop to the netgear switch in LAN2.
It grabbed the first available address.
It can get to the pfsense in LAN1 but not LAN2 (which is directly connected to the same switch ...)
From a desktop in LAN1 I can ping the laptop, ssh to it, but not the interface on the pfsense in LAN2. -
@bingo600
I have no idea what else to do.
I have changed everything in this equation except pfsense -- however there a lot of moving parts. :/ -
I really can't believe this is not possible.
pfsense aside, this should work, being a fairly straight forward static route to a different network.
Does it being a set of 192.168 Class Bs cause pfsense heartache in particular?I unchecked these options on both .30 interfaces:
Maybe there are better options than .1 and .2 on those interfaces that would make routing easier?
-
@MakOwner said in Suggestions for linking two pfsense setups:
I really can't believe this is not possible.
pfsense aside, this should work, being a fairly straight forward static route to a different network.
Does it being a set of 192.168 Class Bs cause pfsense heartache in particular?I unchecked these options on both .30 interfaces:
That was a nice find , and i overlooked those
Maybe there are better options than .1 and .2 on those interfaces that would make routing easier?
.1 and .2 should function as well as any other ip in the /24.
It seems like the .2 interface has some issue , in the definition or connection to the Netgear switch.
Both Netgears are working and are connected , since you can transport data (ping/ssh etc) from the PC in Netgear2 to the pfSense connected to Netgear1.
Did you mention something about CAT7 , cable issues ?
But the Site-to-Site cable must be working since the Netgears can transport data across sites.
The issue must be between the (VM) pfSense , and the "local" Netgear.
You say the PC works on one of the Netgear2 ports.
Long shot ... swap the pfSense & PC ports on the Netgear.I would expect you can see MAC addresses on the Netgear switches. Can you see the pfSense1 MAC on both switches (i'd expect) , can you see the PC mac on both ? ....
And now Can you see pfSense2 MAC on any ports , if not : There must be some cable or interface issue.
@stephenw10
I'm out of ideas here.
Do you have any hints for debugging a L2 problem on a (VM based pfSense) - Seems like the issue is with the VM interface , and the switch it connects to. ... Pending the outcome of @MakOwner seeing the pfSense VM interface MAC in the switch it connects to.I have never touched a VM based pfSense
@MakOwner , did you put "permit ipv4 any any" rules on both Netgear interfaces ?
Could we be fighting Gateway monitor , in conjunction w. L2 issues ?
/Bingo
-
**@bingo600 said in Suggestions for linking two pfsense setups:
@MakOwner said in Suggestions for linking two pfsense setups:
Did you mention something about CAT7 , cable issues ?
I have more CAT7 runs between the buildings than I needed just in case I had issues, and sure enough.. I did initially have some cable termination issues.
I do have a solid 10GB link between the two netgear switches now.But the Site-to-Site cable must be working since the Netgears can transport data across sites.
The issue must be between the (VM) pfSense , and the "local" Netgear.
You say the PC works on one of the Netgear2 ports.
Long shot ... swap the pfSense & PC ports on the Netgear.I have already swapped patch cables and ports on the netgear. I haven't swapped ports on the VM yet. I did do a complete power cycle on the ESXi server and a walk through of the networking configuration there just to ensure nothing could blocking. This hardware won't do passthrough of the Intel NICs though. And the VM is using e1000 emulation.
I would expect you can see MAC addresses on the Netgear switches. Can you see the pfSense1 MAC on both switches (i'd expect) , can you see the PC mac on both ? ....
Can't see anything beyond the interface on LAN2.
And now Can you see pfSense2 MAC on any ports , if not : There must be some cable or interface issue.
@stephenw10
I'm out of ideas here.
Do you have any hints for debugging a L2 problem on a (VM based pfSense) - Seems like the issue is with the VM interface , and the switch it connects to. ... Pending the outcome of @MakOwner seeing the pfSense VM interface MAC in the switch it connects to.I have never touched a VM based pfSense
@MakOwner , did you put "permit ipv4 any any" rules on both Netgear interfaces ?
The default for Netgear is wide-open, any traffic. you have to work at it to make it more secure.
Could we be fighting Gateway monitor , in conjunction w. L2 issues ?
When defining the routes and gateways on both sides, the gateways are immediately marked offline. I tried setting the monitor off in LAN2 but that didn't seem to help.
/Bingo
I'll clone the VM and blow away all the interfaces except the LAN and WAN use a different physical interface today.
-
I can't be of much help here , Layer2 issues have to be solved "onsite".
We're so "close" ... But stil no Cigar
Don't give up
/Bingo
-
I cleared all the switches of any confifguration.
everything is essentially acting like an unmanaged switch.
I shut down the pfsense in LAN2.
Installed an all new VM, only adding the WAN and LAN interface on installation.
I added the third interface after initial setup for the building interlink.
(Side note: adding the 4th interface is where things seem to break -- the wireless (last added) interface is no longer passing traffic. )
Before adding routes I checked the arp table on both ends.
Good entries for 30.1 and 30.2!I can open pfsense on LAN1 from LAN2 (using the .1 interface).
I added the gateway and routes on LAN2 to match LAN1.I can get to the pfsense on LAN1 from LAN2.
I can get to both Netgear switch mgmy interface from both ends.
Arp tables on both ends look good.Can't get beyond the .30 interface to access anything on the LAN from the building in which I'm not directly sitting in.
-
If the ARP (MAC) is good , and you can ping 192.168.30.1 and .2 from both sides , the Layer2 issue seems to be solved.
What does the gateway monitoring say (remember you prob have to allow ICMP to make the monitoring happy)
What does Diagnostic -> Routes show (both ends)
Do you see any blocks in the pfSense Log ?
Routing was described here
https://forum.netgate.com/post/945282/Bingo
-
I'm seeing ICMP being dropped in the firewall logs for 192.168.30.1 on the .1 side:
Nov 15 14:43:35 LAB Default deny rule IPv4 (1000000103) 192.168.30.2 192.168.30.1 ICMPI don't see any rules that would trigger this, and I have added these rules on both sides:
This is the rule set from LAN1 for .30.1 -- I can't access LAN2 from here to capture the rules but they are identical on the reverse end -- using the other interface of course.
-
Hmm.. I have traffic crossing the link now, but I had to create a set of far too lax rules on both sides for the .30 interfaces.
LAN1 side:
LAN2 side:
No collisions so far on DHCP.
Seems everyone is getting IPs assigned via the closest interface, which works well.
Not sure how stable this is and it leaves both ends wide open.Not sure I undertsand why the two rules have traffic on one side but not the other -- see states column?
-
1 : Static Routing
You will have to tell a pfSense about every net (on the other side) you want reachable , including the wifi net.- Firewall
If/when you "just" ping from one pfSense to the other pfSense , the pfSense is creating the traffic on the "nearest interface". This means a packet from ie. 192.168.30.1 would flow to 192.168.30.2 (or any other Lan).
Don't forget to permit 192.168.30.0/24 to some extent , at least ICMP (pings).
Or your gateway monitoring will "barf" at you.ICMP is a protocol like TCP or UDP (select it there) , in a "friendly" environment i usually allow ICMP any any
You have to consider you now have 5 nets.
pfSense1 : Lan1 + Wifi1 + Connect
pfSense2: Lan2 + WiFi2 + ConnectPackets that crosses buildings will have to be permittet on the Connect interface , in the opposite building end , in order to be allowed (received).
Re: States & traffic
That is not unusual , think about how traffic flows , and how much of it./Bingo
- Firewall
-
The wifi nets are normally locked out of the LAN.
They get firewalled access t the internet but no traffic from wireless enters the LANs.
I had LAN1 open temporarily to use a wireless laptop for access to pfsense in LAN1, but that needs to be locked back down.I expected the explicit "LAN to alternate network" and "alternate network to LAN" set of rules to work, whereas I had to just create a single rule that essentially allowed all traffic across the link, not just the traffic from the defined network from the opposite pfsense.
LAN1 rules on the interlink interface:
LAN2 rules on the interlink interface:
Rules that didn't work -- I used net rules and specified the appropriate /24 of the LAN on the opposite pfsense -- one inbound, one outbound.
Adding the third rule (top on each LAN) opened things up.
I feel like there is probably a better way to do this?
I'll work on getting the LAGs and VLANs functioning across the switches first I suppose. -
Rerember pfSense default filters "Inbound" , meaning :
You can filter traffic from entering via an interface.
pfSense default permits NOTHING from entering via an interface (besides mgt on/via the LAN interface) so you WILL have to do permits on the interfaces to allow traffic , to enter the firewall.That also goes for the "site to site" interlink interfaces (192.168.30.1 & 2).
The top rule on the Lan1 / Lan2 : ipv4 any any
Effectively sets the other two rules out of "play" , as the top rule would match anything , and no traffic would hit the below rules.First match : Proccesses the packet.
If i were you i'd start with a :
Permit ICMP any any and a Permit IPV4 any any on both "Site to site" interfaces (effectively permit anything to pass between buildings) (if it has been allowed into the firewall. by the LAN/WAN rules)
That way the rules on the LAN rules could be made "as you are used to".
Then after you get the felling for the Lan rules , you could narrow down the site-to-site rules.
/Bingo
-
Need clarification here.
Have you actually added the VLAN2 interface to each pfSense to use as the transport link?
What have you named that interface? Is that LAB?
You can't put hosts in a transport subnet without creating an asymmetric route which is why your test with a laptop earlier failed.
Can we see a diagram showing exactly how this is connected? With interface names and IPs/subnet?
Steve
-
AFAIK
This ts the current setup
https://forum.netgate.com/post/945569BUT every switch has default settings (All in VLAN1) the site-to-site L2 is also VLAN1 on the 10G-Netgear "building transport switches"
All is transported (connected) via L3 (pfSense interfaces)
No tagging on the pfSense , all is "Pure ethernet" going to the switches
We got it "all up & running" , now only pfSense rules need to be made (adjusted)
WiFi nets should not be static (site2site) routed , as they are "only allowed to go to the internet" on their respective site ... I hope i got that right
The OP mentioned one ISSUE - If he adds a 4'th interface on the VM based pfSense , it will not forward any traffic. Only the first 3 IF's works.
I think we fought that issue earlier , and yep ... That (4'th) IF would not respond to ARP, or show up in the "MAC table" on the switches.
I still think the OP runs w. just 3 IF's on the VM pfSense (No WiFi IF)
/Bingo
-
This is what it looks like:
The interface on pfsense on the left for the link is "Lab - 192.168.30.1"
The interface on pfsense on the right for the link is "Office - 192.168.30.2"In LAN1 I'm using 4 interfaces and the wireless works -- it's running 2.4.4-p3.
In LAN2 it -was running latest 2.4.5 but that's where the 4th interface wouldn't work, so I punted and reinstalled again this time to 2.4.4 and updated to -p3.
I swapped around interfaces in the several re-installs and the NIC appears to be working fine...It's a VM so I'll snapshot it and try the upgrade to 2.4.5, but really as badly as unbound crashes on 2.4.5. I think I want to go see what security fixes are in there beforehand.
Everything right now is in the default VLAN.
I'm using two identical NetGear GS110EMX switches for the 10GB link, and (really old) Dell Powerconnect switches on each end. 5224 on the office end and a 2748 on the Lab end.I had suspected at first possibly the NetGears and Dells weren't really exchanging VLAN/LAG info properly, but based on their interfaces it looks like they are picking up the port configuration on the fly if I configure the Netgear..
But as I said, for right now, it's all open and default.
The wireless networks can't access but the internet -- I have confirmed that on the office side, later today I have to go check the access point for the Lab and ensure everything is right there.
-
@MakOwner said in Suggestions for linking two pfsense setups:
It's a VM so I'll snapshot it and try the upgrade to 2.4.5, but really as badly as unbound crashes on 2.4.5. I think I want to go see what security fixes are in there beforehand.
You could use the Service watchdog for that
I see (saw) the occationally unbound crash , on previous 2.4.4's , and had service watchdog restart it automatically.I had suspected at first possibly the NetGears and Dells weren't really exchanging VLAN/LAG info properly, but based on their interfaces it looks like they are picking up the port configuration on the fly if I configure the Netgear..
I do not hope the NetGears respond to anything (VLAN/LAG) on the DELL's right now , as they are L3 separated.
/Bingo
-
@bingo600 said in Suggestions for linking two pfsense setups:
I do not hope the NetGears respond to anything (VLAN/LAG) on the DELL's right now , as they are L3 separated.
/Bingo
Yea, me too!
They are both doing what they should be doing at this point -- just being dumb switches.I do have a Dell 6248 that I can't get to reset so I can't get into the CLI or the web interface.
Whole other issue. -
Ah I see, no VLANs just direct intrefaces. Well that removes that potential issue.
So "LAN1" is the Office side pfSense and "LAN2" is the Lab side pfSense?
And link interfaces are named according to where the connect to, so the opposite of where they actually are?
I think the problem here is that you are using system aliases like
LAN netand they are invalid on the side you're applying them.You have rules using
LAN netas a source and destination but that can only be right on one of those rules.On LAN1 you need rules on the link interface for:
192.168.20.0/24 toLAN netany protocol
LAB nettoLAB addressicmp (to allow gateway monitoring)On LAN2 you need rules on the link interface for:
192.168.10.0/24 toLAN netany protocol
OFFICE nettoOFFICE addressicmpSteve
