Suggestions for linking two pfsense setups

MakOwner

To simplify this and create a functional, even if suboptimal - because traffic will go through the low end physical pfsense -- setup -- would this work:

Adding a third interface on each pfsense, creating VLAN5 across the link.
Question I have, will using (for example) 192.168.30.1 on one end and 192.168.30.2 on the other work? (apologize for the crude graphic.)

bingo600

Yes it will work

You'd need to do a static route on each pfsense pointing to the opposite lan
via the "opposite" gateway , create gateways if needed.

pfSense 1 (Vlan5) interface 192.168.30.1 /24
Create gateway 192.168.30.2
Static route 192.168.20.0/24 via GW 192.168.30.2

pfSense 2 (Vlan5) interface 192.168.30.2 /24
Create gateway 192.168.30.1
Static route 192.168.10.0/24 via GW 192.168.30.1

pfSense1 = the Lan 192.168.10.0/24 box

MakOwner

I'll set this up and piddle with it on weekends from there.
I'm still having cable termination issues on one of the CAT7 runs, so this gives me something to work on while I wait for parts.

Hopefully soon we will get decent broadband (I'm sure you can see the sarcasm dripping).
Soon I hope to get both DSL accounts terminated in the same building and use load balancing from a single pfsense. At least it's a more widely used configuration and easier to manage.

bingo600

Loadbalancing ~~with~~ on 1:1 Nat mapped to servers ...
That's another can of worms
Especially if done correct w. dual ISP's.

Get your new design up first ....
And you'd already be able to "share" the lines after that.

Do you really have a /22 on @ ?

/Bingo
I'm still wondering why no thumbs up on any answers in this thread.

MakOwner

Do you really have a /22 on @ ?

/22 on the account for LAN1 and a /29 for account on LAN2.
Anyone that has ever dealt with Frontier DSL in areas where they took over from Verizon will know the special hell that is.

The modem itself is DHCP mode, and they have managed to lock out ever modem-only DSL device so that it won't work on their service.
So you get stuck with this modem/router/firewall that is impossible to do real bridging - that you have to pay rent.
And since Frontier has gone into bankruptcy it is next to impossible to get the actual network support teams on a support call.

MakOwner

@bingo600

So ... I removed all the VLAN configurations from everythin, all switches and ports are using the defaut-vlan and no parts are set to accept any traffic.

I configure an interface on each pfsense connected directly to the netgear switches for the link between buildings (removing any possible issues of the switches being the issue -- the other switches used in this setup are different generations of Dell Powerconnect switches).

192.168.30.1 on the LAN1 interface, 192.168.30.2 on the LAN2 interface with routes created

I can ping the local 192.168.30/ interface from anywhere on either end, but I can't ping the gateway (the interface on the opposite pfsense).
Can't get any traffic through the link at all.

I can see port stats increase on the net gear management interface from each end though ...

bingo600

@MakOwner

EDIT: You did make "Allow rules" on the new pfSense VLAN2 interfaces (both ends) , else everything would be blocked.
If yes , read on , else make them , and retest.

Did you connect the pfSense Lan ports directly to the Netgear's ?
Did you do anything with Vlans on the pfSense (you should'nt) , right now. You should just create normal interfaces.

How are the pfSense VLAN2 switchports on the Netgear's defined ?
They should be Untagged members of Vlan2

I have no experience w. Netgear
If going "simple" , the Netgear Site to Site interfaces should be untagged members of VLAN2 , too.

Later :
I would prob. make them tagged members of VLAN2 , as that would open up for defining more vlans to be transported over the site-to-site interlink.

Make sure the above is done first (before the debugging , below)

Debugging:

If you make an extra Untagged Vlan2 port on both of the Netgears , you should be able to connect a PC to that port (set it to Ie. 192.168.30.10/24) , then test if you can ping the "Local pfSense interface" , If yes , then VLAN2 is working (on the local side).
You could do the same on the other site , if working , then VLAN2 is working (on the other side).

Then we know the error is on the site-to-site interlink (Maybe those ports are not member of VLAN2)

How did you define the interface on your Virtual pfSense, did you have a spare IF there too.

It is important for me to now if you are using (defining) vlans on the pfSense , or you only use "Plain ethernet interfaces"

/Bingo

Tip: You should give your Netgears , management ip addreses in VLAN2 , just pick a free ip in the /24 range.

MakOwner

I think the issue may be in the rules for the interface on LAN2.
I reused what was DMZ interface and manually created rules.

This is what LAN1 side rules look like:

(I think this should really be restricted to LAB to OFFICE and OFFICE to LAN to prevent unexpected routes out to the internet.)

Yes the spare interface is connected directly to the Netgear switches.
Everything is physically separated now - so long as traffic will pass over the pfsense from one interface to another, traffic should flow, and DHCP on both ends is restricted to it's own subnet.

No VLANs are configured (Let me rephrase that -- nothing other than default) anywhere on the Netgear or pfsense, and none of the switches that had ports in other than default VLAN are in use.

bingo600

@MakOwner

You should not indicate VLAN2 on your drawing if the netgears are using default (witch is prob Vlan1) , but it should work fine , using Vlan1 on the netgears too , as the pfSense (Layer3) is preventing the Vlans (Layer2) , to be propagated to the other intefaces.

So it seems like it is firewall rules that are blocking your pings.

/Bingo

bingo600

@MakOwner

If you want to do ping tests from the pfsenses on the Netgear link
Remember to allow the 192.168.30.0/24 net too , on both sides.

If the pfSense do the ping (to the other gw) , it will (default) use the source address of the "local gw interface".

Edit. Since i have no clue what ip ranges LAB,LAB & Office are
The rules doesn't say much to me.

/Bingo

MakOwner

ah, yeah, missed that VLAN edit.
I just double checked and neither Netgear has anything but default VLAN.

LAN1 - 192.168.10.0/24
LAN2 - 192.168.20/24
OFFICE interface is 192.168.30.1 on pfsense in LAN1
LAB interface is 192.168.30.2 on pfsense in LAN2

the ruleset on LAN1

This is the ruleset on LAN2

Looking at this I see that on some of the rules only TCP is allowed, so I went back and changed all of the any protocol - "IPV4 *".

No difference :/

I can see the routing table on the pfsense and I can't see why LAN1 can't ping 192.168.30.2 -- unless I need to add the gateway directly to that interface rather than just depend on the static route?

bingo600

@MakOwner

You need to allow ICMP (ping) on the new pfsense intefaces (Netgear) : Source 192.168.30.0/24 to (i'd prob do) ANY
On both sides

Remember now the trafic from lanx also passes the site-to-site interfaces on the pfsenses , and must be allowed accordingly.

Do you see any deny's in the pfsenses ??

What does a Diagnostics -> ARP Table show (entries beginning with 192.168.30.x)

MakOwner

the LAN1 pfsense ARP table:

Both Netgear switches are visible here too, and accessible on LAN1,
So on LAN1 traffic is crossing the .30.1 interface and coming back.

Something on the pfsense on LAN2 is blocking I suppose.

I just filtered the firewall log for any activity with a destination address of 192.168.30.2 (the interface on LAN2 which I have been pinging quite frequently to test access) and there are no entries.

Won't the default LAN allow to any rule cover traffic from 192.168.10 or 192.168.20 to any other subnet or network? Seems to as the management interfaces for the netgear switches are working from LAN1.
And I just checked, I can ping both from LAN1.

Perhaps I should just delete the interface on LAN2 and start over ...

bingo600

If you can ping both netgears from the "Lan1" pfSense , that's good news.

Then the site-to-site link is working.

As the arp says ... something is fishy with 192.168.30.2

Maybe delete and redo it , would be a good start.

Is the 192.168.30.2 end the VM pfSense ?

/Bingo

Cool_Corona

IPsec and failover GW's??

MakOwner

@cool_corona -- it's a really squirrely dual ISP setup with endpoints in two buildings. One /22 and one /29 public network.

@bingo600 - this is getting a little nuts.
LAN2 is the virtual pfsense.
I removed the interlink interface from LAN2, all of the routes and gateways, etc..
Did a physical check on the ESXi box to validate that the NIC I have assigned IS in fact the interface connected to the NetGear switch (validated the port link status, noted the MAC address and MAC address assignment to the VM.)
Restarted pfsense.
Reassigned the interface, rebuilt the gateways and routes.
Still nothing gets past the interface on the pfsense in LAN2.

I even opened up the rules on the interface in LAN2 to allow traffic form that interface to any any in.
I added lan to link and link to lan rules on the LAN interface (although I don't think that's relavant for the arp table -- the arp table shows up the same on both ends -- the other end is (incomplete).

bingo600

Do you have a PC you can connect to the Netgear on Site 2 ?
And give it ie. 192.168.30.10
Then you could ping a little around with that , if the pc can ping pfsense1 Site1 (192.168.30.1) , and not pfsense site2 (192.168.30.2). Then you have some kind of layer2 "challenge", between the netgear on site2 , and the pfsense interface on site2.

Even if you block everything on the pfSense , the interface MAC should still show up in the ARP Table (with a valid mac)

Edit: You should be able to see the pfSense MAC in the Netgear's MAC table , on Site1.

That should be the same for Site2 (the VM/pfSense mac) (but i expect it not to be there)

Sometimes you have to do a little traffic (ie pings) , before the mac address appears in the "tables" on switch or (arp table on pfsense)

/Bingo

MakOwner

@bingo600
I have a system with multiple interfaces and I set up a second, static interface on 192.168.30.20 and plugged it into the Netgear switch on LAN2.

pfsense on LAN1 can see it properly in the ARP table but there is no response from the host on LAN1 -- ping or ssh, even a ping directly from the 192.168.30.1 interface.
(Keeping in mind the primary interface on this system and it's default routes all come from LAN2 so ...)

I have spare laptop I'll plug it in and see what happens.

bingo600

@MakOwner

I see a successful ping from pfSense Lan1 , to the ".20 - device" connected to Netgear (Site2).
That verifies that your site-to-site link works (netgears are transporting data).
The issue must be with the interface/connection from pfSense2 to Netgear2.

MakOwner

@bingo600

I enabled the DHCP server on the 192.68.30.1 interface in LAN1.
Connected a laptop to the netgear switch in LAN2.
It grabbed the first available address.
It can get to the pfsense in LAN1 but not LAN2 (which is directly connected to the same switch ...)
From a desktop in LAN1 I can ping the laptop, ssh to it, but not the interface on the pfsense in LAN2.