Suggestions for linking two pfsense setups

bingo600

@MakOwner said in Suggestions for linking two pfsense setups:

It's a VM so I'll snapshot it and try the upgrade to 2.4.5, but really as badly as unbound crashes on 2.4.5. I think I want to go see what security fixes are in there beforehand.

You could use the Service watchdog for that
I see (saw) the occationally unbound crash , on previous 2.4.4's , and had service watchdog restart it automatically.

I had suspected at first possibly the NetGears and Dells weren't really exchanging VLAN/LAG info properly, but based on their interfaces it looks like they are picking up the port configuration on the fly if I configure the Netgear..

I do not hope the NetGears respond to anything (VLAN/LAG) on the DELL's right now , as they are L3 separated.

/Bingo

MakOwner

@bingo600 said in Suggestions for linking two pfsense setups:

I do not hope the NetGears respond to anything (VLAN/LAG) on the DELL's right now , as they are L3 separated.

/Bingo

Yea, me too!
They are both doing what they should be doing at this point -- just being dumb switches.

I do have a Dell 6248 that I can't get to reset so I can't get into the CLI or the web interface.
Whole other issue.

stephenw10

Ah I see, no VLANs just direct intrefaces. Well that removes that potential issue.

So "LAN1" is the Office side pfSense and "LAN2" is the Lab side pfSense?

And link interfaces are named according to where the connect to, so the opposite of where they actually are?

I think the problem here is that you are using system aliases like LAN net and they are invalid on the side you're applying them.

You have rules using LAN net as a source and destination but that can only be right on one of those rules.

On LAN1 you need rules on the link interface for:
192.168.20.0/24 to LAN net any protocol
LAB net to LAB address icmp (to allow gateway monitoring)

On LAN2 you need rules on the link interface for:
192.168.10.0/24 to LAN net any protocol
OFFICE net to OFFICE address icmp

Steve

MakOwner

I did exactly that -- but it didn't work so I added the second rule that reverses that logic -- and it still didn't work.
Adding the any to any rule made it work.
I didn't try reversing rules before going to the any to any rule though ....

stephenw10

Do you have the link interfaces assigned with the gateways actually on them?

If so auto outbound NAT rules will be applied as though they are a WAN and the other side will see all traffic coming from the Link IP.

Steve

MakOwner

@stephenw10
No, the link interfaces have "None" for gateways.
(Although I did try that at point before I re installed pfsense on LAN2 ...)

stephenw10

Ok, then I would start a continuous ping on one side and check the state tables and firewall logs on both firewalls to see where that traffic going.

Ultimately run a pcap on each interface involved to check the packets are arrving/leaving at each point.

Steve

bingo600

@MakOwner
Here is a "Crash course" in Vlan tagging on pfSense , if ever needed.
https://forum.netgate.com/post/944381

I hope you find it usable, even though i would not recommend you begin on that stuff right now. Tame your current beast, before releasing a new.

/Bingo

MakOwner

So, I thought since I have sort of got this working well enough thanks to all your help, I'd share what I have done -- so far anyway. Thanks for all the help!

I have had to revert to 2.4.4-p3 on both ends.
2.4.5 doesn't seem to play well at this point.
Gateway monitoring seem to be the downfall for some reason.
I'll wait for another update before trying again.

I currently have a physical layout like this:

DHCP servers are active with very limited ranges on each interface.
The office interface handles the DHCP interface handles the .30/24 on the link between buildings and provides the addresses for the management interfaces on the two switches.
Each DHCP server is configured to serve addresses within a specified range and each /24 has a different range just for conveniance. .10/24 uses .110-.120, .20/24 uses .120-.130, .30/24 uses .130-.140 for example.
Not a single DHCP collision or bomb - yet, anyway.
All active physical systems use static DHCP assignment outside the assignment range so that it becomes very obvious if anything goes south.

Al internet bound traffic in the Lab goes out the .20/24 network, all internet bound traffic from the Office goes out the .10/24 network.

Wireless is isolated on both sides to just internet access.

I added the .30/24 interface of the opposite system as a gateway on each pfsense.
I added static routes to .10/24 and .20/24 networks as appropriate.
On the .30/24 interface on each pfsense I added an any-to-any firewall rule.

That let traffic pass both directions.

From there I began working on DNS.
The DNS resolver in 2.4.4 doesn't quite behave as I expected.

Still piddling with that.

After DNS is working like I expect I'll move on to configuring the direct link.

bingo600

@MakOwner said in Suggestions for linking two pfsense setups:

So, I thought since I have sort of got this working well enough thanks to all your help, I'd share what I have done -- so far anyway. Thanks for all the help!

I have had to revert to 2.4.4-p3 on both ends.
2.4.5 doesn't seem to play well at this point.
Gateway monitoring seem to be the downfall for some reason.
I'll wait for another update before trying again.

Why does GW monitoring pose a problem ?
It's usually just a "Ping of the GW".

DHCP servers are active with very limited ranges on each interface.
The office interface handles the DHCP interface handles the .30/24 on the link between buildings and provides the addresses for the management interfaces on the two switches.
Each DHCP server is configured to serve addresses within a specified range and each /24 has a different range just for conveniance. .10/24 uses .110-.120, .20/24 uses .120-.130, .30/24 uses .130-.140 for example.
Not a single DHCP collision or bomb - yet, anyway.
All active physical systems use static DHCP assignment outside the assignment range so that it becomes very obvious if anything goes south.

Special way to "reckognize" which DHCP server hands out the ip.
But if it works for you .....
Once you trust DHCP , you could move to something more "default".

I usually let my dhcp start at .129 .. 250
And uses 25..125 for static assignments (1..24) are for infrastructure like routers , switches etc.

I added the .30/24 interface of the opposite system as a gateway on each pfsense.
I added static routes to .10/24 and .20/24 networks as appropriate.

Great ... Do you understand why you have to do that now ?
Tell one pfSense how to "reach" a remote lan on the "other".

On the .30/24 interface on each pfsense I added an any-to-any firewall rule.

That is "fine" , basically making the "Connect net" transparent.
You could fine tune that , later if ever needed.

The .30/24 rules , are like all other "normal" pfSense interface rules "Enter" rules :
See it as: "This permitted traffic" is allowed to "enter" the pfSense-box via "this interface" , everything not permitted is blocked , and will NOT enter the receiving pfSense box.

From there I began working on DNS.
The DNS resolver in 2.4.4 doesn't quite behave as I expected.

What is your issue ?
Still Unbound crashes ?
Did you try the service watchdog ?

After DNS is working like I expect I'll move on to configuring the direct link.

???

Remember when making changes to :
Backup pfSense configs often.
Don't make 10 changes at the same time.

All in all a nice excersise , you certainly got your hands dirty.
Well done , especially for not giving up.

/Bingo

Ps:
If you ever want to ~~"do it right"~~ more flexible , ~~you~~ we should enable multi-vlan on ALL your switches. That would give you the flexibility to make ANY switchport in any building , a member of "ANY vlan".
Ie. a server in building-1 , would be seen as a member of the building-2 Lan.

But that would require that you are reading up on "tagging" on all your switches , including the old "access" switches you use.
And start playing a little with vlans & tagging , before starting that up.