Suggestions for linking two pfsense setups
-
If you want to do ping tests from the pfsenses on the Netgear link
Remember to allow the 192.168.30.0/24 net too , on both sides.If the pfSense do the ping (to the other gw) , it will (default) use the source address of the "local gw interface".
Edit. Since i have no clue what ip ranges LAB,LAB & Office are
The rules doesn't say much to me./Bingo
-
ah, yeah, missed that VLAN edit.
I just double checked and neither Netgear has anything but default VLAN.LAN1 - 192.168.10.0/24
LAN2 - 192.168.20/24
OFFICE interface is 192.168.30.1 on pfsense in LAN1
LAB interface is 192.168.30.2 on pfsense in LAN2the ruleset on LAN1
This is the ruleset on LAN2
Looking at this I see that on some of the rules only TCP is allowed, so I went back and changed all of the any protocol - "IPV4 *".
No difference :/
I can see the routing table on the pfsense and I can't see why LAN1 can't ping 192.168.30.2 -- unless I need to add the gateway directly to that interface rather than just depend on the static route?
-
You need to allow ICMP (ping) on the new pfsense intefaces (Netgear) : Source 192.168.30.0/24 to (i'd prob do) ANY
On both sidesRemember now the trafic from lanx also passes the site-to-site interfaces on the pfsenses , and must be allowed accordingly.
Do you see any deny's in the pfsenses ??
What does a Diagnostics -> ARP Table show (entries beginning with 192.168.30.x)
-
the LAN1 pfsense ARP table:
Both Netgear switches are visible here too, and accessible on LAN1,
So on LAN1 traffic is crossing the .30.1 interface and coming back.Something on the pfsense on LAN2 is blocking I suppose.
I just filtered the firewall log for any activity with a destination address of 192.168.30.2 (the interface on LAN2 which I have been pinging quite frequently to test access) and there are no entries.
Won't the default LAN allow to any rule cover traffic from 192.168.10 or 192.168.20 to any other subnet or network? Seems to as the management interfaces for the netgear switches are working from LAN1.
And I just checked, I can ping both from LAN1.Perhaps I should just delete the interface on LAN2 and start over ...
-
If you can ping both netgears from the "Lan1" pfSense , that's good news.
Then the site-to-site link is working.
As the arp says ... something is fishy with 192.168.30.2
Maybe delete and redo it , would be a good start.
Is the 192.168.30.2 end the VM pfSense ?
/Bingo
-
IPsec and failover GW's??
-
@cool_corona -- it's a really squirrely dual ISP setup with endpoints in two buildings. One /22 and one /29 public network.
@bingo600 - this is getting a little nuts.
LAN2 is the virtual pfsense.
I removed the interlink interface from LAN2, all of the routes and gateways, etc..
Did a physical check on the ESXi box to validate that the NIC I have assigned IS in fact the interface connected to the NetGear switch (validated the port link status, noted the MAC address and MAC address assignment to the VM.)
Restarted pfsense.
Reassigned the interface, rebuilt the gateways and routes.
Still nothing gets past the interface on the pfsense in LAN2.I even opened up the rules on the interface in LAN2 to allow traffic form that interface to any any in.
I added lan to link and link to lan rules on the LAN interface (although I don't think that's relavant for the arp table -- the arp table shows up the same on both ends -- the other end is (incomplete). -
Do you have a PC you can connect to the Netgear on Site 2 ?
And give it ie. 192.168.30.10
Then you could ping a little around with that , if the pc can ping pfsense1 Site1 (192.168.30.1) , and not pfsense site2 (192.168.30.2). Then you have some kind of layer2 "challenge", between the netgear on site2 , and the pfsense interface on site2.Even if you block everything on the pfSense , the interface MAC should still show up in the ARP Table (with a valid mac)
Edit: You should be able to see the pfSense MAC in the Netgear's MAC table , on Site1.
That should be the same for Site2 (the VM/pfSense mac) (but i expect it not to be there)
Sometimes you have to do a little traffic (ie pings) , before the mac address appears in the "tables" on switch or (arp table on pfsense)
/Bingo
-
@bingo600
I have a system with multiple interfaces and I set up a second, static interface on 192.168.30.20 and plugged it into the Netgear switch on LAN2.pfsense on LAN1 can see it properly in the ARP table but there is no response from the host on LAN1 -- ping or ssh, even a ping directly from the 192.168.30.1 interface.
(Keeping in mind the primary interface on this system and it's default routes all come from LAN2 so ...)
I have spare laptop I'll plug it in and see what happens.
-
I see a successful ping from pfSense Lan1 , to the ".20 - device" connected to Netgear (Site2).
That verifies that your site-to-site link works (netgears are transporting data).
The issue must be with the interface/connection from pfSense2 to Netgear2. -
I enabled the DHCP server on the 192.68.30.1 interface in LAN1.
Connected a laptop to the netgear switch in LAN2.
It grabbed the first available address.
It can get to the pfsense in LAN1 but not LAN2 (which is directly connected to the same switch ...)
From a desktop in LAN1 I can ping the laptop, ssh to it, but not the interface on the pfsense in LAN2. -
@bingo600
I have no idea what else to do.
I have changed everything in this equation except pfsense -- however there a lot of moving parts. :/ -
I really can't believe this is not possible.
pfsense aside, this should work, being a fairly straight forward static route to a different network.
Does it being a set of 192.168 Class Bs cause pfsense heartache in particular?I unchecked these options on both .30 interfaces:
Maybe there are better options than .1 and .2 on those interfaces that would make routing easier?
-
@MakOwner said in Suggestions for linking two pfsense setups:
I really can't believe this is not possible.
pfsense aside, this should work, being a fairly straight forward static route to a different network.
Does it being a set of 192.168 Class Bs cause pfsense heartache in particular?I unchecked these options on both .30 interfaces:
That was a nice find , and i overlooked those
Maybe there are better options than .1 and .2 on those interfaces that would make routing easier?
.1 and .2 should function as well as any other ip in the /24.
It seems like the .2 interface has some issue , in the definition or connection to the Netgear switch.
Both Netgears are working and are connected , since you can transport data (ping/ssh etc) from the PC in Netgear2 to the pfSense connected to Netgear1.
Did you mention something about CAT7 , cable issues ?
But the Site-to-Site cable must be working since the Netgears can transport data across sites.
The issue must be between the (VM) pfSense , and the "local" Netgear.
You say the PC works on one of the Netgear2 ports.
Long shot ... swap the pfSense & PC ports on the Netgear.I would expect you can see MAC addresses on the Netgear switches. Can you see the pfSense1 MAC on both switches (i'd expect) , can you see the PC mac on both ? ....
And now Can you see pfSense2 MAC on any ports , if not : There must be some cable or interface issue.
@stephenw10
I'm out of ideas here.
Do you have any hints for debugging a L2 problem on a (VM based pfSense) - Seems like the issue is with the VM interface , and the switch it connects to. ... Pending the outcome of @MakOwner seeing the pfSense VM interface MAC in the switch it connects to.I have never touched a VM based pfSense
@MakOwner , did you put "permit ipv4 any any" rules on both Netgear interfaces ?
Could we be fighting Gateway monitor , in conjunction w. L2 issues ?
/Bingo
-
**@bingo600 said in Suggestions for linking two pfsense setups:
@MakOwner said in Suggestions for linking two pfsense setups:
Did you mention something about CAT7 , cable issues ?
I have more CAT7 runs between the buildings than I needed just in case I had issues, and sure enough.. I did initially have some cable termination issues.
I do have a solid 10GB link between the two netgear switches now.But the Site-to-Site cable must be working since the Netgears can transport data across sites.
The issue must be between the (VM) pfSense , and the "local" Netgear.
You say the PC works on one of the Netgear2 ports.
Long shot ... swap the pfSense & PC ports on the Netgear.I have already swapped patch cables and ports on the netgear. I haven't swapped ports on the VM yet. I did do a complete power cycle on the ESXi server and a walk through of the networking configuration there just to ensure nothing could blocking. This hardware won't do passthrough of the Intel NICs though. And the VM is using e1000 emulation.
I would expect you can see MAC addresses on the Netgear switches. Can you see the pfSense1 MAC on both switches (i'd expect) , can you see the PC mac on both ? ....
Can't see anything beyond the interface on LAN2.
And now Can you see pfSense2 MAC on any ports , if not : There must be some cable or interface issue.
@stephenw10
I'm out of ideas here.
Do you have any hints for debugging a L2 problem on a (VM based pfSense) - Seems like the issue is with the VM interface , and the switch it connects to. ... Pending the outcome of @MakOwner seeing the pfSense VM interface MAC in the switch it connects to.I have never touched a VM based pfSense
@MakOwner , did you put "permit ipv4 any any" rules on both Netgear interfaces ?
The default for Netgear is wide-open, any traffic. you have to work at it to make it more secure.
Could we be fighting Gateway monitor , in conjunction w. L2 issues ?
When defining the routes and gateways on both sides, the gateways are immediately marked offline. I tried setting the monitor off in LAN2 but that didn't seem to help.
/Bingo
I'll clone the VM and blow away all the interfaces except the LAN and WAN use a different physical interface today.
-
I can't be of much help here , Layer2 issues have to be solved "onsite".
We're so "close" ... But stil no Cigar
Don't give up
/Bingo
-
I cleared all the switches of any confifguration.
everything is essentially acting like an unmanaged switch.
I shut down the pfsense in LAN2.
Installed an all new VM, only adding the WAN and LAN interface on installation.
I added the third interface after initial setup for the building interlink.
(Side note: adding the 4th interface is where things seem to break -- the wireless (last added) interface is no longer passing traffic. )
Before adding routes I checked the arp table on both ends.
Good entries for 30.1 and 30.2!I can open pfsense on LAN1 from LAN2 (using the .1 interface).
I added the gateway and routes on LAN2 to match LAN1.I can get to the pfsense on LAN1 from LAN2.
I can get to both Netgear switch mgmy interface from both ends.
Arp tables on both ends look good.Can't get beyond the .30 interface to access anything on the LAN from the building in which I'm not directly sitting in.
-
If the ARP (MAC) is good , and you can ping 192.168.30.1 and .2 from both sides , the Layer2 issue seems to be solved.
What does the gateway monitoring say (remember you prob have to allow ICMP to make the monitoring happy)
What does Diagnostic -> Routes show (both ends)
Do you see any blocks in the pfSense Log ?
Routing was described here
https://forum.netgate.com/post/945282/Bingo
-
I'm seeing ICMP being dropped in the firewall logs for 192.168.30.1 on the .1 side:
Nov 15 14:43:35 LAB Default deny rule IPv4 (1000000103) 192.168.30.2 192.168.30.1 ICMPI don't see any rules that would trigger this, and I have added these rules on both sides:
This is the rule set from LAN1 for .30.1 -- I can't access LAN2 from here to capture the rules but they are identical on the reverse end -- using the other interface of course.
-
Hmm.. I have traffic crossing the link now, but I had to create a set of far too lax rules on both sides for the .30 interfaces.
LAN1 side:
LAN2 side:
No collisions so far on DHCP.
Seems everyone is getting IPs assigned via the closest interface, which works well.
Not sure how stable this is and it leaves both ends wide open.Not sure I undertsand why the two rules have traffic on one side but not the other -- see states column?
