Policy routing / gateway rules ignored when "Any" rule is present at the end.

kathampy · Jul 3, 2016, 9:47 PM

That shouldn't matter but I wouldn't do it that way. My transport network would be outside the routed subnet.

What's the correct way to configure this on pfSense? Just create separate static routes for each? These are the VLANs on the switch that are relevant to the LAN interface:

Default Network
Subnet: 192.168.1.0/24
L3 Switch: 192.168.1.1

Guest Network
Subnet: 192.168.5.0/24
L3 Switch: 192.168.5.1

Firewall Network
Subnet: 192.168.2.0/24
L3 Switch: 192.168.2.1
pfSense: 192.168.2.2

kathampy · Jul 3, 2016, 9:57 PM

I unchecked "Disable Negate Rules" and tried this.

@Derelict:

You can Diagnostics > Command Prompt grep negate_networks /tmp/rules.debug That will also show everywhere it is used.

table <negate_networks>
pass  in  quick  on $LAN inet proto { tcp udp }  from $Default  to <negate_networks>  port 53 tracker 10000001 keep state  label "NEGATE_ROUTE: Negate policy routing for destination"
pass  in  quick  on $LAN inet proto udp  from $Default port $Steam_Source  to <negate_networks> tracker 10000002 keep state  label "NEGATE_ROUTE: Negate policy routing for destination"
pass  in  quick  on $LAN inet proto udp  from $Default  to <negate_networks>  port $Steam tracker 10000003 keep state  label "NEGATE_ROUTE: Negate policy routing for destination"</negate_networks></negate_networks></negate_networks></negate_networks>

If I check "Disable Negate Rules", then it's empty.

@DerelictOr:

Diagnostics > Tables and look at negate_networks.

This is always empty.

kathampy · Jul 3, 2016, 10:02 PM

It looks like "Disable Negate Rules" should not have any effect in my case. I need to wait till it stops working again and debug it then.

kathampy · Jul 5, 2016, 3:57 PM

I've PMed you my rules.debug. "Disable Negate Rules" did not have any effect. The two gateway rules are still being bypassed.

johnpoz · Jul 5, 2016, 4:19 PM

So I am a bit confused on your setup. So you mention L3 switches on all your networks on pfsense.

Are these switches all doing routing? And those network 192.168.1/24 and 5.0/24 and 2.0/24 are transit networks?? Those seem quite large for transit networks, do you have hosts other than your routers on these networks?

This network you call your default that is the wan of pfsense? All your rules have default as the source?

I would suggest you draw up your connectivity.. And keep in mind any connection between pfsense and a downstream router should be via a transit network. Ie there should not be any hosts or devices on this network that connects pfsense to another router or you run into asymmetrical routing problems.

Your only showing 1 set of rules. Which is fine if this is on the transit interface to pfsense.. But your rules on that interface need to allow for the source networks unless your downstream routers are also natting. And your outbound nat need to allow for the natting of these downstream networks as well.

kathampy · Jul 5, 2016, 6:44 PM

I've attached a network diagram that explains the setup.

The diagram is the logical view. In reality pfSense in running on Hyper-V. The two WAN cables are also connected to the switch on dedicated VLANs 3 & 4. VLAN 2, 3 & 4 are connected to the pfSense VM as individual network interfaces.

johnpoz · Jul 5, 2016, 6:43 PM

yup that drawing makes it very clear.. That was not there before was it?

So that looks right from my quick look at it.. But your rules are very specific with source ports and source network for example. So your showing specific states that match that rule but going out your normal routing? Your default gateway seems to be the ficus connection.

kathampy · Jul 5, 2016, 6:49 PM

@johnpoz:

yup that drawing makes it very clear.. That was not there before was it?

So that looks right from my quick look at it.. But your rules are very specific with source ports and source network for example. So your showing specific states that match that rule but going out your normal routing? Your default gateway seems to be the ficus connection.

Yes, I have traffic that matches Steam (UDP Destination Port 27000:27030 in this case) that goes out the default gateway. I can see these states under the Ficus interface in the Diagnostic > States viewer with matching destination ports. As soon as I disconnect the default gateway PPPoE, the traffic immediately starts flowing over the Airtel PPPoE connection within milliseconds. Now the States viewer shows the states under the Airtel interface and they stay there until the traffic stops and the states are closed.

johnpoz · Jul 5, 2016, 6:56 PM

really makes no sense since if it doesn't trigger the rule that sends it out specific gateway without the allow rule at the end it should just fail.

Derelict · Jul 6, 2016, 2:13 AM

Yes, I have traffic that matches Steam (UDP Destination Port 27000:27030 in this case) that goes out the default gateway. I can see these states under the Ficus interface in the Diagnostic > States viewer with matching destination ports.

I would like to see outputs of pfctl -vvsr and pfctl -vvss when the firewall is in this mode.

That will show exactly which rule is passing the traffic in question. It would be especially helpful if you could clear all states, generate the traffic in question, then take these samples.

I realize it might be kind of large.

There's probably a simple explanation for what you're seeing. Just don't know what it is yet.