Two pfSense Stacks - Management Network - Routing

inzanez

Hello

I've got two pfsense firewalls running, setup looks like this basically:

DMZ < –- |pfSense1| < --- LAN --- > |pfSense 2|< --- N1, N2, Management --- >

So, pfSense1 is in the LAN and the DMZ, pfSense 2 manages networks N1, N2 and Management.

Clients in LAN have the default gateway set to the LAN ip address of pfSense1. LAN clients are allowed to the Management network, firewall rules are set. In addition, pfSense1 has a routing table entry for the management network pointing to the LAN ip address of pfSense2. That all works well, no issue.

Now I would like to add a management link on a separate interface to the pfSense1 box. But as pfSense1 has this route for management network pointing to pfSense2,...I guess that this will be an issue? Because pfSense1 will not directly connect over the it's management interface "to the management network", but always through pfSense2 as this is the route set.

So,...is there a way around that?oO

Regards

heper

please draw a better schematic.

one that includes interfaces & ip's & subnets.

why do you have 2 pfsense's for this? seems like over-complicating things?

inzanez

It was a requirement to have that separated using two physical stacks,…never mind that, I know it's always a point one could argue,...:-)
Schematic is below, it's a bit more complicated now as it's not two single boxes but two CARP clusters.

What I would like to do now is have a dedicated interface on Firewalls B and D into the MGMT net (so from those FWs onto the switch).

heper

so you have a route configured ON 10.0.0.1: ```
dst:10.0.2.0/24 GW:10.0.0.4


and now you want to add a new interface (eg. 10.0.2.123/24) on  B/D and connect it to the mgmt switch directly ?

just remove the static route then? what is the point in keeping the route, when you have a direct connection?
or
you might be able to get both by using a gateway & policy routing:
-remove the static route /  use policy routing for dst:10.0.2.0/24 and set the GW to FW-AC_gw
-use the directly connected interface for other traffic that is not policy-routed ?

inzanez

so you have a route configured ON 10.0.0.1:
Code: [Select]

dst:10.0.2.0/24 GW:10.0.0.4

and now you want to add a new interface (eg. 10.0.2.123/24) on B/D and connect it to the mgmt switch directly ?

Yes, thats right.

just remove the static route then? what is the point in keeping the route, when you have a direct connection?

If I do that, I would need to make '10.0.2.123' (the new address in mgmt from b/d) the default gateway for devices in the management network, wouldn't I?
Otherwise all connections from LAN to Management are going through the new interface on B/D, but will try to get back to the host through A/C (as this is default GW in the management network). And as states would not be present,…everything would be dropped?

-remove the static route / use policy routing for dst:10.0.2.0/24 and set the GW to FW-AC_gw
-use the directly connected interface for other traffic that is not policy-routed ?

I'm sorry, I don't quite understand that approach. Having a policy route that sais: DST:10.0.2.0/24 –> GW == FW-AC_gw,...would not be different than what I got with the static route now? Or would it?

inzanez

Ok, just out of interest I added policy based routing with a rule:

LAN –> Management, Gateway FW-A/C IP Address.
I also added the NIC to the MGMT Interface on FW-B/D which are "10.0.2.4" and "10.0.2.5". Funny thing is: When I connect from a LAN client to 10.0.2.4, it's not routed over FW-A/C, it's accessed directly...

heper

the difference is that you can specify a gateway for each firewall-rule independently. you can't do that with a static route.

so you could say eg.:
src:host_A dst: mgmt-subnet ==> gw=GW_mgmt
src: host_B dst: mgmt-subnet ==> gw=*

inzanez

Yep, I just realized that. But apparently the traffic for the "management network member interfaces" on FW-B/D is not routed through that Gateway I specified.
For all other hosts on the management network, it DOES work indeed.

inzanez

I seem to have difficulties with policy based routing in general.

Given the diagram above, as is (so Firewalls B&D have NO connection into Management network through a dedicated interface), I did the two following tests:

On Firewall B/D, I set a static route that sais: 10.0.2.0/24 has Gateway 10.0.0.4
I connect via SSH session from my client 10.0.0.123 (LAN) to 10.0.2.xxx into MGMT network.

–> This works, connection stable, tcpdump looks very clean!

On Firewall B/D, disabled the static route, created a RULE for: LAN network --> MGMT network and set the Gateway in the rule to 10.0.0.4
I connect via SSH session from my client 10.0.0.123 (LAN) to 10.0.2.xxx into MGMT network.

--> Connection initially seems to be stable, but ssh client loses connection (freezes) after about 1 minute. When I look at tcpdump on the client, I see many many TCP retransmissions, DUP ACK and so on.

This only happens with the policy based routing...any idea what might cause that?