routing for dual PFSense routers

marmar_75

Hi All,
I have need for two PFSense routers in my house; i have two separate internet connections, one comes in upstairs and the other downstairs. I need to be able to set up a route between the two, so that the upstairs network can communicate to the downstairs so that it can access the domain controller and other resources set up downstairs.

Based on my (limited) understanding from working with Cisco, I have set up 3 networks - 192.168.0.0 /24 for upstairs, 192.168.1.0 /24 for downstairs, and 192.168.2.0 / 30 for the connection between the nodes. Each node has 4 network ports; one is being used for WAN, one for LAN, one for OPT1 (which is the connector network), and then an unused one.

My understanding is that i should route with no issue between each LAN network and the connector network. When trying to implement a static route, i stumbled as to what I should set the gateway to, i.e. the local IP or the remote. I tried it with both, with very limited success.

Most recently in trying this I set up RIP on both nodes, and I can now ping at least to the other router, but I"m having trouble confirming that I'm going beyond it.

Can anyone give some guidance as to the best way to get this to work?

Thanks!!

akuma1x

Do you HAVE to run 2 separate pfsense firewalls, one on each internet connection? I ask this, because 1 pfsense box can easily handle multiple internet/WAN/ISP connections, and can route between them. You could simplify your network and just run 1 box.

Jeff

marmar_75

As noted, the physical locations of the internet connections drives this. One modem is upstairs, one is downstairs. Putting both connections in a location that could physically access a single device is not an option.

marmar_75

Making progress. I can get pings to desktops going both ways; added gateways and static routes (gateways pointing to the remote host in each case) as well as rules to allow traffic from each network. Down to fine tuning now, it seems.

netblues

@marmar_75 Well, apart from the fact that it sounds strange that you can't have two modems/lines on the same floor, you haven't mentioned how the two pf boxes communicate between floors, physically.

marmar_75

Ok, I figured this out. Here's what I have, which is working. (Disclaimer: I stopped when this started working; I have done very little testing to see if any of this is redundant or unnecessary, that will be my next step. If you feel any of this is redundant, feel free to let me know which parts to test, and I will test and update as necessary.)

First, recap the configuration, and operating theory.

Router 1 - LAN - 192.168.1.0
OPT1 - 192.168.2.1

Router 2 - LAN - 192.168.0.0
OPT1 -192.168.2.2

We are using 3 separate networks - each router's LAN (2) and then the network formed by the OPT1 connections between the hosts (in this case, formed by an ethernet-over-power adapter, which due to internal wiring only gives 80 megabit at highest speed; upstairs connection is 200 megabit, don't want to limit the upstairs users to less than full speed, only give them access to the downstairs nework).

All of these steps will be carried out twice, once on each router.

Step 1 - go to System, Routing, Gateways, and add a gateway for the other node's OPT port. Choose the interface of the OPT port, the appropriate address family(s), and enter the gateway IP address.
example - router 1 has a gateway pointed to 192.168.2.2, router 2 has a gateway pointing to 192.168.2.1

Step 2 - Create a static route, using the destination network of the other router's LAN Subnet, and the gateway you just created.
example - router 1 is pointed to 192.168.0.0 /24, using the gateway with IP 192.168.2.2

Step 3 - Click Firewall, Rules, LAN, and create the following rules (I don't recall which are default, so if you already have it,
leave it).

(these will show on LAN tab)

A) Action Pass, Interface LAN, address family as appropriate, protocol Any, source Network subnet of the other
router's LAN, destination any, give the description a friendly name you'll recognize.
this route is to allow traffic from the other router's LAN into your local LAN.
B) Action Pass, interface LAN, address family as appropriate, protocol Any, source network LAN net, destination
any, friendly name Default allow LAN to any rule
I think this is a default rule; i'm including it here for the sake of completeness
C) Action Pass, interface LAN, address family IPv4, protcol any, source OPT1 net, destination any, description default allow opt to any rule
this will allow OPT traffic to pass to your LAN

(these will show on interface OPT1)

D) Action Pass, Interface OPT1, Address family as appropriate, protocol any, source network address of other
node's LAN, destination LAN net
E) Action Pass, Interface OPT1, Address family as appropriate, Protocol Any, Source Network remote LAN
network, destination LAN net
allow traffic from remote LAN to local LAN
F) Action pass, interface OPT1, address family as appropriate, source opt1 net, destination Any, description
Default allow OPT to any rule
allow traffic from OPT to any address
G) Interface OPT1, address family as appropriate, protocol any, source LAN net, destination any, description
Default allow LAN traffic to OPT
allow traffic OPT to any address
(the following is now redundant, but allowed me to access the remote router via the OPT port before the rest
of the routing was working)
H) Create this rule twice, once for 443 and once for 80; Action Pass, interface OPt1, address family as
appropriate, protcol TCP/UDP, source any, destination <HTTPS or HTTP, remember, one rule for each>,
description "Allow web admin access on port <insert appropriate here>

This allows routing to pass properly, and was tested with ping, remote desktop protocol connections and SMB connections (the SMB connections were successful but showed no data in the folder because they weren't logged into the domain).

After this, I configured the DHCP to include the DNS server of the Domain controller, by adding the IP of the DNS server to the DHCP configuration on both routers. I then had the upstairs computers log out and back in, and tested successful logins by checking rights when accessing SMB shares.

netblues

You have made things complicated.
Even with two pf boxes, I would just plug them together on the same lan and use the powerline to connect the two lans.

I would use one dhcp server on the lan and assign custom default gateways as needed
There are even allow/deny capabilities on dhcp pools that make management a bit easier.
This also takes care of master browsers and such, needed for windows networking.

Then access rules needed are trivial and easy to follow.

If you need more access control between two floors you could consider bridging interfaces, but this complicates things again.

marmar_75

@netblues Complicated perhaps, but you responded to a detailed outline with 10K' overview steps; no indication as to how DHCP on a single lan would differentiate between two different sets of users and route them to two separate gateways. Your answer is not helpful. If you're not going to be helpful, why waste your time?

netblues

No , I responded with an alternative approach ASSUMING that is also fit to your purpose.
I have no idea what security considerations exist between floors, how many stations are on the lans, etc.
You are at the drivers seat.
For example, if one link is used for servers with static ip's and the other for dhcp users, the scenario is straight forward.
On the other hand if you really need two dhcp ranges serving the same lan, with access lists, based on mac addresses, it would be more beneficial than static dhcp reservations if they could be grouped.
And perhaps you know your way around this approach too.

There is no point discussing implementation details if there are reasons which make the approach absurd, don't you think?

If however you find this approach fit to purpose and you are willing to provide necessary input, we could come up with ideas on the best approach to identify groups of users.

chpalmer

I use an interface on each router on a single subnet 10.50.90.0/30 in my case and run the routed package on each of the routers. 10.50.91.1 on box one and 10.50.90.2 on box two.

Firewall rules on each box to control access.

marmar_75

@chpalmer this sounds like what I did, but I was lacking the rules after implementation of routed to allow the pass through.

marmar_75

@chpalmer I think I may go back and disable the static routes and leave the rules, and see if routed did its thing or not...

marmar_75

OK, I refined this. Here's what seems to be absolutely necessary. This is about 20K' overview so if anyone wants details I'll provide happily.

PFsense setup for linked routers

Routed RIP on both
WAN and LAN set up and working
3rd network connection physically linking two routers, same subnet, separate IP addresses
gateway configured for the physical link between the routers, pointing at the remote IP address for the link
static route to remote router's LAN, pointing at gateway from #3
on physical link network, configure these rules:
A) remote network IP to LAN network
B) Routerlink network to any
c) LAN network to any
d) IPv4 ICMP protocol enabled for all (allows ping testing)

Configure this way on both sides, and you should work. :)