Existing connections ignore route changes

adam65535

EDIT: I forgot to mention that I am using CARP on the interfaces for firewall failover (not wan gateway failover) in case that affects the behavior below.

I have a need for routing to change for existing connections. I have a perl script that monitors an IP and adds and drops a route to a remote subnet depending on a site to site T1 connection. The problem is that existing connections do not obey or seem to see the routing table changes. New connections do. This is with the default LAN rule allowing everything outbound for testing.

Is there a way to make sure existing connections in the state table(no gateway defined in the LAN rule that passes this traffic) to see the route changes? Basically I want traffic to a subnet to go either out the internet or out an OPT interface through a site to site T1 connection.

The transition must not interrupt existing connections to the remote subnet.

Example:
LAN 10.1.1.1/24
WAN 1.2.2.1/24 –---- routes through the internet to remote site 1.4.4.1/24
OPT1 1.3.3.1/24 <-> router local site (1.3.3.2/24) <-> router remote site <-> remote external subnet 1.4.4.1/24

I monitor the remote router and add and drop the route below.
route add -net 1.4.4.0/24 1.3.3.2

For an existing connection from 10.1.1.5 to 1.4.4.4 the static route change is ignored so the packets are still sent through OPT1 when the static route for OPT1 gets deleted by my script. This would cause issues with existing connections. If I establish a new connection it will correctly go out the WAN interface.

I am looking to replace a homegrown Linux firewall/router with pfsense. The behavior of pf with routing changes is a sticking point though.

I also tried to use gateway groups only for the specific rule (not for default gateway failover) but it had the same problem. Existing connections did not fail over.

adam65535

I wonder if my question is not clear, too complex, or really nobody knows.

From my testing it seems like NATed connections (only ones I tested) do not obey any routing changes for existing connections. I wonder if that is one of the reasons for the option below is defaulted to disabled. To make sure any existing connections are terminated for the wan when routing changes. WIthout doing that it would appear that packets for states that were already connected before a routing change would still be going to a dead gateway until the TCP times out.

There must be a way to make sure packets are routed according to the routing table for already established connections when a route changes.

Advanced->Gateway Monitoring
- States
By default the monitoring process will flush states for a gateway that goes down. This option overrides that behavior by not clearing states for existing connections.