Policy based routing not overriding static routing
I am running pfsense in a Data Centre and using Gateway groups with Tiers to specify the next-hop for layer 2 leased lines to various sites. On each site we have a primary and secondary leased line hence the need to use Tiers.
I am using policy-based routing and gateway groups matching traffic using a destination network subnet i.e. 192.168.2.0/24 on the WAN Firewall and specifying a Gateway group for this network in the firewall rule. As indicated in these documents: -
Even though I get a match on the wan firewall rule and can see active sessions. I still have to specify a static route as no route appears for the downstream subnet in the routing table of the Data Centre pfsense router, when I run netstat -r.
As I have two leased lines to each site, I have to specify the static route.
In this example, I have a downstream subnet 192.168.2.0/24 and have added the subnet twice, once for each gateway. The secondary route has the larger /24 prefix and the smaller /25’s from the same subnet prefer the route via the primary gateway. You can not specify the same /24 twice using the GUI in pfsense as there appears to be no support for secondary routes.
192.168.2.0/24 gateway-2 10.10.10.2 gw2vlan200
192.168.2.0/25 gateway-1 10.10.11.2 gw1vlan100
192.168.2.128/25 gateway-1 10.10.11.2 gw1vlan100
If the primary connection goes down the Gateway group switches over to the second-Tier gateway specified in the Gateway group and traffic that is related/ i.e. originated from the downstream subnet appears to route without issues. However, unrelated traffic, i.e. new inbound traffic appears to follow the static routes specified in the routing table, even though it is matched in the WAN firewall rule and a Gateway group is specified. So, in this case, I have to disable the static route(s) for the affected route.
If I remove the static routes altogether inbound traffic that is NOT related, will bounce on the WAN interface of the Data Centre router because there is no route to the downstream subnet in the routing table even though I have specified it using policy-based routing.
One final observation, I would expect a static route to be removed if a gateway goes down. This also does not appear to happen, according to netstat -r.
Does anyone know if this is a bug or have a workaround?