Policy routing / gateway rules ignored when "Any" rule is present at the end.

kathampy · Jul 5, 2016, 6:20 PM

Under the Default Network section, the policy routing rules Steam and Steam Clientport are completely ignored whenever the final Default > Any rule is enabled - traffic matches this rule instead. If I disable the Default > Any rule at the end (or disconnect the default gateway), the above policy routing rules start working!

I have 1 LAN and 2 WAN interfaces. The rules are defined on the LAN interface.

An L3 switch does inter-VLAN routing and forwards remaining traffic to pfSense.

All relevant information is in the attached network diagram.

Derelict · Jul 3, 2016, 7:45 PM

They are processed in order. You have done something incorrectly.

What interface is that on? What is that interface's IP network configuration?

What are the contents of all your aliases?

kathampy · Jul 3, 2016, 8:10 PM

I've added more details in the original post. Now that I think about it, will the presence of the LAN static route require the "Disable Negate Rules" box to be checked?

Derelict · Jul 3, 2016, 8:44 PM

For IPv4 source from Default the Reject rule at the end will do nothing because the rule above the Guest network separator will match everything.

Define "not working." is it DNS, connectivity or what?

Nobody can use anything but the firewall IP address itself as DNS servers the way you have it configured.

Do you have any floating rules defined?

No, I don't see that static routes to a LAN gateway are added to the negate_networks table.

kathampy · Jul 3, 2016, 9:15 PM

@Derelict:

For IPv4 source from Default the Reject rule at the end will do nothing because the rule above the Guest network separator will match everything.

Yes, it is meant to reject traffic only from the Guest network. I want connections to fail instantly with Reject, instead of timing out with the default Block action.

@Derelict:

Nobody can use anything but the firewall IP address itself as DNS servers the way you have it configured.

There is a dedicated DNS server on the Default network. All computers use this as their DNS server. Outbound DNS is allowed from the Default network, as shown in the rules.

@Derelict:

Define "not working." is it DNS, connectivity or what?

The rules Steam and Steam Clientport never match. There are no states created for them. Instead states are created for the Default > Any rule and traffic flows via the default gateway. The Diagnostics > States viewer clearly shows that the destination ports match Steam, but they are created on the default gateway interface and not the configured gateway.

If I disable the Default > Any rule, then new states are correctly created for the Steam and Steam Clientport rules and it uses policy routing to the configured gateway.

@Derelict:

Do you have any floating rules defined?

No.

Derelict · Jul 3, 2016, 9:07 PM

No idea then. Sorry. There is something somewhere else. PM me your /tmp/rules.debug.

kathampy · Jul 3, 2016, 9:24 PM

@Derelict:

No idea then. Sorry. There is something somewhere else. PM me your /tmp/rules.debug.

It appears to be working correctly after enabling "Disable Negate Rules". I need to test it for a while.

This says static routes are included in negate_networks?
https://redmine.pfsense.org/issues/1950

Derelict · Jul 3, 2016, 9:28 PM

That is very old.

My internal static routes are not included in negate_networks.

Sounds like you might have the same networks included in remote VPN networks or something.

kathampy · Jul 3, 2016, 9:35 PM

@Derelict:

Sounds like you might have the same networks included in remote VPN networks or something.

I don't have any other networks beyond what's shown here. Can something happen because the 192.168.0.0/16 static route overlaps the /24 LAN interface? I can try adding individual static routes for each /24 VLAN instead.

How do I view the negate_networks table on my box?

Derelict · Jul 3, 2016, 9:36 PM

That shouldn't matter but I wouldn't do it that way. My transport network would be outside the routed subnet.

You can Diagnostics > Command Prompt grep negate_networks /tmp/rules.debug That will also show everywhere it is used.

Or Diagnostics > Tables and look at negate_networks.

kathampy · Jul 3, 2016, 9:47 PM

@Derelict:

That shouldn't matter but I wouldn't do it that way. My transport network would be outside the routed subnet.

What's the correct way to configure this on pfSense? Just create separate static routes for each? These are the VLANs on the switch that are relevant to the LAN interface:

Default Network
Subnet: 192.168.1.0/24
L3 Switch: 192.168.1.1

Guest Network
Subnet: 192.168.5.0/24
L3 Switch: 192.168.5.1

Firewall Network
Subnet: 192.168.2.0/24
L3 Switch: 192.168.2.1
pfSense: 192.168.2.2

kathampy · Jul 3, 2016, 9:57 PM

I unchecked "Disable Negate Rules" and tried this.

@Derelict:

You can Diagnostics > Command Prompt grep negate_networks /tmp/rules.debug That will also show everywhere it is used.

table <negate_networks>
pass  in  quick  on $LAN inet proto { tcp udp }  from $Default  to <negate_networks>  port 53 tracker 10000001 keep state  label "NEGATE_ROUTE: Negate policy routing for destination"
pass  in  quick  on $LAN inet proto udp  from $Default port $Steam_Source  to <negate_networks> tracker 10000002 keep state  label "NEGATE_ROUTE: Negate policy routing for destination"
pass  in  quick  on $LAN inet proto udp  from $Default  to <negate_networks>  port $Steam tracker 10000003 keep state  label "NEGATE_ROUTE: Negate policy routing for destination"</negate_networks></negate_networks></negate_networks></negate_networks>

If I check "Disable Negate Rules", then it's empty.

@DerelictOr:

Diagnostics > Tables and look at negate_networks.

This is always empty.

kathampy · Jul 3, 2016, 10:02 PM

It looks like "Disable Negate Rules" should not have any effect in my case. I need to wait till it stops working again and debug it then.

kathampy · Jul 5, 2016, 3:57 PM

I've PMed you my rules.debug. "Disable Negate Rules" did not have any effect. The two gateway rules are still being bypassed.

johnpoz · Jul 5, 2016, 4:19 PM

So I am a bit confused on your setup. So you mention L3 switches on all your networks on pfsense.

Are these switches all doing routing? And those network 192.168.1/24 and 5.0/24 and 2.0/24 are transit networks?? Those seem quite large for transit networks, do you have hosts other than your routers on these networks?

This network you call your default that is the wan of pfsense? All your rules have default as the source?

I would suggest you draw up your connectivity.. And keep in mind any connection between pfsense and a downstream router should be via a transit network. Ie there should not be any hosts or devices on this network that connects pfsense to another router or you run into asymmetrical routing problems.

Your only showing 1 set of rules. Which is fine if this is on the transit interface to pfsense.. But your rules on that interface need to allow for the source networks unless your downstream routers are also natting. And your outbound nat need to allow for the natting of these downstream networks as well.

kathampy · Jul 5, 2016, 6:44 PM

I've attached a network diagram that explains the setup.

The diagram is the logical view. In reality pfSense in running on Hyper-V. The two WAN cables are also connected to the switch on dedicated VLANs 3 & 4. VLAN 2, 3 & 4 are connected to the pfSense VM as individual network interfaces.

johnpoz · Jul 5, 2016, 6:43 PM

yup that drawing makes it very clear.. That was not there before was it?

So that looks right from my quick look at it.. But your rules are very specific with source ports and source network for example. So your showing specific states that match that rule but going out your normal routing? Your default gateway seems to be the ficus connection.

kathampy · Jul 5, 2016, 6:49 PM

@johnpoz:

yup that drawing makes it very clear.. That was not there before was it?

So that looks right from my quick look at it.. But your rules are very specific with source ports and source network for example. So your showing specific states that match that rule but going out your normal routing? Your default gateway seems to be the ficus connection.

Yes, I have traffic that matches Steam (UDP Destination Port 27000:27030 in this case) that goes out the default gateway. I can see these states under the Ficus interface in the Diagnostic > States viewer with matching destination ports. As soon as I disconnect the default gateway PPPoE, the traffic immediately starts flowing over the Airtel PPPoE connection within milliseconds. Now the States viewer shows the states under the Airtel interface and they stay there until the traffic stops and the states are closed.

johnpoz · Jul 5, 2016, 6:56 PM

really makes no sense since if it doesn't trigger the rule that sends it out specific gateway without the allow rule at the end it should just fail.

Derelict · Jul 6, 2016, 2:13 AM

Yes, I have traffic that matches Steam (UDP Destination Port 27000:27030 in this case) that goes out the default gateway. I can see these states under the Ficus interface in the Diagnostic > States viewer with matching destination ports.

I would like to see outputs of pfctl -vvsr and pfctl -vvss when the firewall is in this mode.

That will show exactly which rule is passing the traffic in question. It would be especially helpful if you could clear all states, generate the traffic in question, then take these samples.

I realize it might be kind of large.

There's probably a simple explanation for what you're seeing. Just don't know what it is yet.