Suggestions for linking two pfsense setups
-
Do you really have a /22 on @ ?
/22 on the account for LAN1 and a /29 for account on LAN2.
Anyone that has ever dealt with Frontier DSL in areas where they took over from Verizon will know the special hell that is.The modem itself is DHCP mode, and they have managed to lock out ever modem-only DSL device so that it won't work on their service.
So you get stuck with this modem/router/firewall that is impossible to do real bridging - that you have to pay rent.
And since Frontier has gone into bankruptcy it is next to impossible to get the actual network support teams on a support call. -
So ... I removed all the VLAN configurations from everythin, all switches and ports are using the defaut-vlan and no parts are set to accept any traffic.
I configure an interface on each pfsense connected directly to the netgear switches for the link between buildings (removing any possible issues of the switches being the issue -- the other switches used in this setup are different generations of Dell Powerconnect switches).
192.168.30.1 on the LAN1 interface, 192.168.30.2 on the LAN2 interface with routes created
I can ping the local 192.168.30/ interface from anywhere on either end, but I can't ping the gateway (the interface on the opposite pfsense).
Can't get any traffic through the link at all.I can see port stats increase on the net gear management interface from each end though ...
-
EDIT: You did make "Allow rules" on the new pfSense VLAN2 interfaces (both ends) , else everything would be blocked.
If yes , read on , else make them , and retest.Did you connect the pfSense Lan ports directly to the Netgear's ?
Did you do anything with Vlans on the pfSense (you should'nt) , right now. You should just create normal interfaces.How are the pfSense VLAN2 switchports on the Netgear's defined ?
They should be Untagged members of Vlan2I have no experience w. Netgear
If going "simple" , the Netgear Site to Site interfaces should be untagged members of VLAN2 , too.Later :
I would prob. make them tagged members of VLAN2 , as that would open up for defining more vlans to be transported over the site-to-site interlink.Make sure the above is done first (before the debugging , below)
Debugging:
If you make an extra Untagged Vlan2 port on both of the Netgears , you should be able to connect a PC to that port (set it to Ie. 192.168.30.10/24) , then test if you can ping the "Local pfSense interface" , If yes , then VLAN2 is working (on the local side).
You could do the same on the other site , if working , then VLAN2 is working (on the other side).Then we know the error is on the site-to-site interlink (Maybe those ports are not member of VLAN2)
How did you define the interface on your Virtual pfSense, did you have a spare IF there too.
It is important for me to now if you are using (defining) vlans on the pfSense , or you only use "Plain ethernet interfaces"
/Bingo
Tip: You should give your Netgears , management ip addreses in VLAN2 , just pick a free ip in the /24 range.
-
I think the issue may be in the rules for the interface on LAN2.
I reused what was DMZ interface and manually created rules.This is what LAN1 side rules look like:
(I think this should really be restricted to LAB to OFFICE and OFFICE to LAN to prevent unexpected routes out to the internet.)
Yes the spare interface is connected directly to the Netgear switches.
Everything is physically separated now - so long as traffic will pass over the pfsense from one interface to another, traffic should flow, and DHCP on both ends is restricted to it's own subnet.No VLANs are configured (Let me rephrase that -- nothing other than default) anywhere on the Netgear or pfsense, and none of the switches that had ports in other than default VLAN are in use.
-
You should not indicate VLAN2 on your drawing if the netgears are using default (witch is prob Vlan1) , but it should work fine , using Vlan1 on the netgears too , as the pfSense (Layer3) is preventing the Vlans (Layer2) , to be propagated to the other intefaces.
So it seems like it is firewall rules that are blocking your pings.
/Bingo
-
If you want to do ping tests from the pfsenses on the Netgear link
Remember to allow the 192.168.30.0/24 net too , on both sides.If the pfSense do the ping (to the other gw) , it will (default) use the source address of the "local gw interface".
Edit. Since i have no clue what ip ranges LAB,LAB & Office are
The rules doesn't say much to me./Bingo
-
ah, yeah, missed that VLAN edit.
I just double checked and neither Netgear has anything but default VLAN.LAN1 - 192.168.10.0/24
LAN2 - 192.168.20/24
OFFICE interface is 192.168.30.1 on pfsense in LAN1
LAB interface is 192.168.30.2 on pfsense in LAN2the ruleset on LAN1
This is the ruleset on LAN2
Looking at this I see that on some of the rules only TCP is allowed, so I went back and changed all of the any protocol - "IPV4 *".
No difference :/
I can see the routing table on the pfsense and I can't see why LAN1 can't ping 192.168.30.2 -- unless I need to add the gateway directly to that interface rather than just depend on the static route?
-
You need to allow ICMP (ping) on the new pfsense intefaces (Netgear) : Source 192.168.30.0/24 to (i'd prob do) ANY
On both sidesRemember now the trafic from lanx also passes the site-to-site interfaces on the pfsenses , and must be allowed accordingly.
Do you see any deny's in the pfsenses ??
What does a Diagnostics -> ARP Table show (entries beginning with 192.168.30.x)
-
the LAN1 pfsense ARP table:
Both Netgear switches are visible here too, and accessible on LAN1,
So on LAN1 traffic is crossing the .30.1 interface and coming back.Something on the pfsense on LAN2 is blocking I suppose.
I just filtered the firewall log for any activity with a destination address of 192.168.30.2 (the interface on LAN2 which I have been pinging quite frequently to test access) and there are no entries.
Won't the default LAN allow to any rule cover traffic from 192.168.10 or 192.168.20 to any other subnet or network? Seems to as the management interfaces for the netgear switches are working from LAN1.
And I just checked, I can ping both from LAN1.Perhaps I should just delete the interface on LAN2 and start over ...
-
If you can ping both netgears from the "Lan1" pfSense , that's good news.
Then the site-to-site link is working.
As the arp says ... something is fishy with 192.168.30.2
Maybe delete and redo it , would be a good start.
Is the 192.168.30.2 end the VM pfSense ?
/Bingo
-
IPsec and failover GW's??
-
@cool_corona -- it's a really squirrely dual ISP setup with endpoints in two buildings. One /22 and one /29 public network.
@bingo600 - this is getting a little nuts.
LAN2 is the virtual pfsense.
I removed the interlink interface from LAN2, all of the routes and gateways, etc..
Did a physical check on the ESXi box to validate that the NIC I have assigned IS in fact the interface connected to the NetGear switch (validated the port link status, noted the MAC address and MAC address assignment to the VM.)
Restarted pfsense.
Reassigned the interface, rebuilt the gateways and routes.
Still nothing gets past the interface on the pfsense in LAN2.I even opened up the rules on the interface in LAN2 to allow traffic form that interface to any any in.
I added lan to link and link to lan rules on the LAN interface (although I don't think that's relavant for the arp table -- the arp table shows up the same on both ends -- the other end is (incomplete). -
Do you have a PC you can connect to the Netgear on Site 2 ?
And give it ie. 192.168.30.10
Then you could ping a little around with that , if the pc can ping pfsense1 Site1 (192.168.30.1) , and not pfsense site2 (192.168.30.2). Then you have some kind of layer2 "challenge", between the netgear on site2 , and the pfsense interface on site2.Even if you block everything on the pfSense , the interface MAC should still show up in the ARP Table (with a valid mac)
Edit: You should be able to see the pfSense MAC in the Netgear's MAC table , on Site1.
That should be the same for Site2 (the VM/pfSense mac) (but i expect it not to be there)
Sometimes you have to do a little traffic (ie pings) , before the mac address appears in the "tables" on switch or (arp table on pfsense)
/Bingo
-
@bingo600
I have a system with multiple interfaces and I set up a second, static interface on 192.168.30.20 and plugged it into the Netgear switch on LAN2.pfsense on LAN1 can see it properly in the ARP table but there is no response from the host on LAN1 -- ping or ssh, even a ping directly from the 192.168.30.1 interface.
(Keeping in mind the primary interface on this system and it's default routes all come from LAN2 so ...)
I have spare laptop I'll plug it in and see what happens.
-
I see a successful ping from pfSense Lan1 , to the ".20 - device" connected to Netgear (Site2).
That verifies that your site-to-site link works (netgears are transporting data).
The issue must be with the interface/connection from pfSense2 to Netgear2. -
I enabled the DHCP server on the 192.68.30.1 interface in LAN1.
Connected a laptop to the netgear switch in LAN2.
It grabbed the first available address.
It can get to the pfsense in LAN1 but not LAN2 (which is directly connected to the same switch ...)
From a desktop in LAN1 I can ping the laptop, ssh to it, but not the interface on the pfsense in LAN2. -
@bingo600
I have no idea what else to do.
I have changed everything in this equation except pfsense -- however there a lot of moving parts. :/ -
I really can't believe this is not possible.
pfsense aside, this should work, being a fairly straight forward static route to a different network.
Does it being a set of 192.168 Class Bs cause pfsense heartache in particular?I unchecked these options on both .30 interfaces:
Maybe there are better options than .1 and .2 on those interfaces that would make routing easier?
-
@MakOwner said in Suggestions for linking two pfsense setups:
I really can't believe this is not possible.
pfsense aside, this should work, being a fairly straight forward static route to a different network.
Does it being a set of 192.168 Class Bs cause pfsense heartache in particular?I unchecked these options on both .30 interfaces:
That was a nice find , and i overlooked those
Maybe there are better options than .1 and .2 on those interfaces that would make routing easier?
.1 and .2 should function as well as any other ip in the /24.
It seems like the .2 interface has some issue , in the definition or connection to the Netgear switch.
Both Netgears are working and are connected , since you can transport data (ping/ssh etc) from the PC in Netgear2 to the pfSense connected to Netgear1.
Did you mention something about CAT7 , cable issues ?
But the Site-to-Site cable must be working since the Netgears can transport data across sites.
The issue must be between the (VM) pfSense , and the "local" Netgear.
You say the PC works on one of the Netgear2 ports.
Long shot ... swap the pfSense & PC ports on the Netgear.I would expect you can see MAC addresses on the Netgear switches. Can you see the pfSense1 MAC on both switches (i'd expect) , can you see the PC mac on both ? ....
And now Can you see pfSense2 MAC on any ports , if not : There must be some cable or interface issue.
@stephenw10
I'm out of ideas here.
Do you have any hints for debugging a L2 problem on a (VM based pfSense) - Seems like the issue is with the VM interface , and the switch it connects to. ... Pending the outcome of @MakOwner seeing the pfSense VM interface MAC in the switch it connects to.I have never touched a VM based pfSense
@MakOwner , did you put "permit ipv4 any any" rules on both Netgear interfaces ?
Could we be fighting Gateway monitor , in conjunction w. L2 issues ?
/Bingo
-
**@bingo600 said in Suggestions for linking two pfsense setups:
@MakOwner said in Suggestions for linking two pfsense setups:
Did you mention something about CAT7 , cable issues ?
I have more CAT7 runs between the buildings than I needed just in case I had issues, and sure enough.. I did initially have some cable termination issues.
I do have a solid 10GB link between the two netgear switches now.But the Site-to-Site cable must be working since the Netgears can transport data across sites.
The issue must be between the (VM) pfSense , and the "local" Netgear.
You say the PC works on one of the Netgear2 ports.
Long shot ... swap the pfSense & PC ports on the Netgear.I have already swapped patch cables and ports on the netgear. I haven't swapped ports on the VM yet. I did do a complete power cycle on the ESXi server and a walk through of the networking configuration there just to ensure nothing could blocking. This hardware won't do passthrough of the Intel NICs though. And the VM is using e1000 emulation.
I would expect you can see MAC addresses on the Netgear switches. Can you see the pfSense1 MAC on both switches (i'd expect) , can you see the PC mac on both ? ....
Can't see anything beyond the interface on LAN2.
And now Can you see pfSense2 MAC on any ports , if not : There must be some cable or interface issue.
@stephenw10
I'm out of ideas here.
Do you have any hints for debugging a L2 problem on a (VM based pfSense) - Seems like the issue is with the VM interface , and the switch it connects to. ... Pending the outcome of @MakOwner seeing the pfSense VM interface MAC in the switch it connects to.I have never touched a VM based pfSense
@MakOwner , did you put "permit ipv4 any any" rules on both Netgear interfaces ?
The default for Netgear is wide-open, any traffic. you have to work at it to make it more secure.
Could we be fighting Gateway monitor , in conjunction w. L2 issues ?
When defining the routes and gateways on both sides, the gateways are immediately marked offline. I tried setting the monitor off in LAN2 but that didn't seem to help.
/Bingo
I'll clone the VM and blow away all the interfaces except the LAN and WAN use a different physical interface today.
