From static routing to policy routing with gateway groups

[errfld]

Hi everyone!

I am trying a few days now to modify my current setup from static routing to policy based routing. I want to add some fallback gateways with gateway groups so my first step I reckend would be to switch from static to policy based routing.

Here is a little schematic abstraction of my current working setup:

My real configuration entails some more LANs but for the moment I try to focus on the two provided LANs and ensure connectivity between them.

PFSenseHQ:
no NAT

10.254.202.2 Interface:

192.168.6.1 Interface:
(omitted picture because it's straight forward - no Gateway)

Default GW: 10.254.202.1

HQLan firewall rules: contains allow any rule and anti blockout
HQWan (10.254.202.2) firewall rules: only contains allow any rule (without gateway policy)

PFSenseRS:
is pretty much a mirror of PFSenseHQ (of course with adapted gateway and such)

PFSenseDC:
Outgoing NAT only on WAN Interface with hybrid rule generation.

Interface DC2HQ (10.254.202.1) has no upstream gateway
Interface DC2RS (10.254.203.1) has no upstream gateway

Routing is done via static routes like:

The WAN-Gateway is the default route.

This simple setups works fine and with no major problems so far. But sometimes my network connection between the DataCenter and HQ or RS is disturbed by some working crew hitting a cable.

I created gateway groups on all 3 firewalls which use the direkt connection on Tier1 and a openvpn tunnel on Tier2. In addition I added rules with the appropriate gateways:

• PFSenseDC-Interface HQ_DC IPv4* src 192.168.2.0/23 dst 192.168.6.0/24 route-to 10.254.203.2
• PFSenseDC-Interface RS_DC IPv4* src 192.168.6.0/24 dst 192.168.2.0/23
  route-to 10.254.202.2
• PFSenseDC-Interface RS_DC IPv4* src 192.168.6.0/24 dst * (no route-to) for WAN
• OpenVPN is disabled for testing purposes

Furthermore I have 2 test machines (192.168.2.19) and (192.168.6.1) which i use for simple ICMP tests.

When I now deactivate one of the static routes on PFSenseDC the outgoing ping on the target network stops working. (in my test case the route to 192.168.6.0/24)

To debug this behaviour I took a look at /tmp/rules.debug on all machines, and I saw, that reply-to is omitted on rules with route-to and Gatewaygroups. I guess this should not be a problem, as there are routes provided which should encompass the target networks. I could track down the ICMP was sent all the way to the target firewall (I will investigate the receiving client with tcpdump today evening) but no answer reaches the sender. From the sender Network is the Internet also unreachable.

Any suggestions where I should start looking for or what I should double (triple) check?