Option to use routing table for all packet routing including gateway

adam65535

I see that existing connections seem to stick to a certain gateway. The only way to failover is to let them time out or let pfsense kill the current connections which interrupts routed traffic. This also prevents me from using a perl script to adjust routing for a specific purpose I have because existing connections do not change to the new routing table rules when I add a route.

Why not just let the routing table send the packets to the correct gateway without forcing them to a specific one? This would allow switching the gateway and existing connections would just route over the new gateway without needing to kill them. I assume this same issue exists with running dynamic routing protocols on pfsense but I haven't used them on pfsense to know.

Is this a behavior that pfsense enforces with pf rules somehow or is it a limitation in pf that existing connections lock onto the routing they initially have once entering the state table and any changes in routing table are ignored for that state from then on?

eri--

ITs how the current used packet filter works.
There will be improvement to this on coming versions.

But leave or take this is a mixed subject since some people prefer on solution some prefer the other.

For now you can enable bypass firewall rules for statically routed subnets to avoid this.

adam65535

Thanks for the response. The 'Static route filtering' option in the advanced settings wouldn't be a secure solution for my needs because the static route that I toggle in a perl script is for a public IP space that only a few subnets get routed over during Internet outages. I wouldn't want it to be unfiltered.

Do you mean there will be enhancements to the upstream pf filter to change this behavior or do you mean there might be enhancements to pfsense to use an alternative filter that does allow routing to change for existing connections?

I wonder if monowall uses a filter that allows route changes for existing connections. They don't have firewall standby redundancy like pfsense though so I couldn't use it. Currently the location that I need this behavior I have a custom built firewall using a bunch of tools on Linux that complies with all the requirements. Using a GUI would be preferred though for sure. I can still use pfsense in some areas… just not all of them because of this issue. I can't have any existing connections getting killed during a failover to a backup firewall.

Thanks again. I searched and could not locate a document explaining the pf and sticky type routing behavior. Can you point me to another forum or documentation that might give more information about that?

cmb

You can use floating rules to override the "pass out" policy routing, then as long as you don't specify a gateway in any rules, traffic follows the routing table including routing table changes.

adam65535

Thanks for the response. I tested this a few months ago but now I am thinking my testing was at fault when I determined how pfsense behaved. I can only simulate the same behavior for local connections generated from the firewall itself. I am not testing the gateway changing but only a single route being added to the route table for a small IP range to force that traffic through another route. The perl script I created drops the route on the local and remote firewall when the MPLS connection goes down between sites(simple ping every second between each side).

I just did another test and with the default allow all LAN rule I was able to get the route to change to another router by simply manually adding a static route through the pfsense shell. The packets on an existing TCP connection then immediately started to route to the second router on the WAN subnet (route add -net ExternalTcpServerIP/32 PrivateMplsGatewayIP). A could see the traffic trying to be sent to the PrivateMplsGatewayIP. Keep in mind I am using a cluster so NATed outgoing rules are sent from a different external IP than the firewall IP.

Default pass rule:
pass in quick on $LAN from 192.168.1.0/24 to any keep state label "USER_RULE: Default allow LAN to any rule"

Connections matching that rule seem to not be forced to a specific gateway and follow the routing table. Is that the behavior you would expect?

I gather that policy routing only applies to a packet that matches a route-to.

Pfsense generates a single hidden rule below (viewable from /tmp/rules.debug) that has route-to in it. It seems to only try and force a gateway for local client connections originating on the firewall IP to the internet. I did verify with another test that an existing TCP client connection initiated from the firewall does not change to a different route if I manually add a route to the routing table.

The hidden rules generated are below:

I replaced the IPs with names (GatewayIP, WanIP, WanSubnet
pass out all keep state allow-opts label "let out anything from firewall host itself"
pass out route-to ( em0 GatewayIP ) from WanIP to !WanSubnet keep state allow-opts label "let out anything from firewall host itself"

I see that the route-to would match traffic generated from the firewall. I don't want that behavior either though actually.

To get all traffic (local firewall generated traffic) to not get policy routed would it be as simple as an option to not use route-to for even the hidden rules so that firewall initiated connections follow the routing table too?

I am still digging for as much documentation on these subjects as I can. I really want to understand how pf and the routing table mix. Things are really starting to click as I read more and do more testing.

adam65535

I know why I could not repeat my initial test. I did not have a secondary firewall setup at the time for clustering/failover (Not WAN failover… I don't use that). There is a bug where policy routing does not get applied to the virtual Carp IPs (Wan in the instance). My first test was without Carp. My second test was with Carp. I found a bug report that matches the scenario I am in.

http://redmine.pfsense.org/issues/1823

Under a cluster/failover situation you want to NAT from the virtual cluster/CARP IP for outgoing WAN traffic instead of the real firewall IP for failover purposes. I setup advanced outbound NAT so that all outgoing traffic from internal nets get NATed to the virtual WAN Carp IP. Outgoing traffic from the internal nets does not match the rule below because the rule has only the real IP of the gateway (WanIP) and does not cover the Virtual CARP IP. This is the behavior I want actually :).

pass out route-to ( em0 GatewayIP ) from WanIP to !WanSubnet keep state allow-opts label "let out anything from firewall host itself"

If this bug gets fixed my dynamic routing will not work anymore for the internal networks.

I can only hope that the route-to policy routing stuff can be made an option for the hidden rules. I do not believe there is any way to override it without specifying another gateway which would still use policy routing. It would have been great to create a floating rule saying basically 'remove route-to' for all traffic going out wan but I don't think that is possible.

Please please please have the option to strip out route-to for the hidden rules as a feature. Pretty please? I don't want any policy routing. Without policy routing I would be able to manually add routes with a custom script (or a dynamic routing protocol) and packets in an existing connection would immediately get sent to the new router I specify.

adam65535

RESOLVED:

The route-to behavior can be disabled with a simple floating rule and leaving route-to set to default. This solves my routing issues by disabling policy routing. Packets go by the routing table by default with the rule below at the top of the floating rule base. They can still be specified in your other rules if you want specific rules to use policy routing of course.

Action: Pass
Interface: (none selected)
Direction: out
Protocol: any
Source: any
Destination: any

That simple rule at the top of your floating rules will disable the route-to behavior and a rule gets generated in the raw pf rules like this:
pass out from any to any keep state label "USER_RULE: Disable route-to by overriding the hidden rule"

Just an FYI if someone new to pf firewalling reads this and thinks this will allow all outgoing traffic even traffic from your LAN interfaces going through the firewall then that would be incorrect. There is a block 'in' enforced by pfsense at the bottom of the rule base. The in rules is what controls traffic through the firewall in pfsense. Queue (match) out rules can still be used in floating to enforce QoS queue settings though without affecting pass behavior of other rules in LAN section.