Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Policy routing / gateway rules ignored when "Any" rule is present at the end.

    Routing and Multi WAN
    3
    20
    4.0k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • K
      kathampy
      last edited by

      Under the Default Network section, the policy routing rules Steam and Steam Clientport are completely ignored whenever the final Default > Any rule is enabled - traffic matches this rule instead. If I disable the Default > Any rule at the end (or disconnect the default gateway), the above policy routing rules start working!

      I have 1 LAN and 2 WAN interfaces. The rules are defined on the LAN interface.

      An L3 switch does inter-VLAN routing and forwards remaining traffic to pfSense.

      All relevant information is in the attached network diagram.
      Rules.PNG
      Rules.PNG_thumb
      Aliases.PNG
      Aliases.PNG_thumb
      Gateways.PNG
      Gateways.PNG_thumb
      Routes.PNG
      Routes.PNG_thumb
      Network.PNG
      Network.PNG_thumb

      1 Reply Last reply Reply Quote 0
      • DerelictD
        Derelict LAYER 8 Netgate
        last edited by

        They are processed in order. You have done something incorrectly.

        What interface is that on? What is that interface's IP network configuration?

        What are the contents of all your aliases?

        Chattanooga, Tennessee, USA
        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
        Do Not Chat For Help! NO_WAN_EGRESS(TM)

        1 Reply Last reply Reply Quote 0
        • K
          kathampy
          last edited by

          I've added more details in the original post. Now that I think about it, will the presence of the LAN static route require the "Disable Negate Rules" box to be checked?

          1 Reply Last reply Reply Quote 0
          • DerelictD
            Derelict LAYER 8 Netgate
            last edited by

            For IPv4 source from Default the Reject rule at the end will do nothing because the rule above the Guest network separator will match everything.

            Define "not working." is it DNS, connectivity or what?

            Nobody can use anything but the firewall IP address itself as DNS servers the way you have it configured.

            Do you have any floating rules defined?

            No, I don't see that static routes to a LAN gateway are added to the negate_networks table.

            Chattanooga, Tennessee, USA
            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
            Do Not Chat For Help! NO_WAN_EGRESS(TM)

            1 Reply Last reply Reply Quote 0
            • K
              kathampy
              last edited by

              @Derelict:

              For IPv4 source from Default the Reject rule at the end will do nothing because the rule above the Guest network separator will match everything.

              Yes, it is meant to reject traffic only from the Guest network. I want connections to fail instantly with Reject, instead of timing out with the default Block action.

              @Derelict:

              Nobody can use anything but the firewall IP address itself as DNS servers the way you have it configured.

              There is a dedicated DNS server on the Default network. All computers use this as their DNS server. Outbound DNS is allowed from the Default network, as shown in the rules.

              @Derelict:

              Define "not working." is it DNS, connectivity or what?

              The rules Steam and Steam Clientport never match. There are no states created for them. Instead states are created for the Default > Any rule and traffic flows via the default gateway. The Diagnostics > States viewer clearly shows that the destination ports match Steam, but they are created on the default gateway interface and not the configured gateway.

              If I disable the Default > Any rule, then new states are correctly created for the Steam and Steam Clientport rules and it uses policy routing to the configured gateway.

              @Derelict:

              Do you have any floating rules defined?

              No.

              1 Reply Last reply Reply Quote 0
              • DerelictD
                Derelict LAYER 8 Netgate
                last edited by

                No idea then. Sorry. There is something somewhere else. PM me your /tmp/rules.debug.

                Chattanooga, Tennessee, USA
                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                1 Reply Last reply Reply Quote 0
                • K
                  kathampy
                  last edited by

                  @Derelict:

                  No idea then. Sorry. There is something somewhere else. PM me your /tmp/rules.debug.

                  It appears to be working correctly after enabling "Disable Negate Rules". I need to test it for a while.

                  This says static routes are included in negate_networks?
                  https://redmine.pfsense.org/issues/1950

                  1 Reply Last reply Reply Quote 0
                  • DerelictD
                    Derelict LAYER 8 Netgate
                    last edited by

                    That is very old.

                    My internal static routes are not included in negate_networks.

                    Sounds like you might have the same networks included in remote VPN networks or something.

                    Chattanooga, Tennessee, USA
                    A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                    DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                    Do Not Chat For Help! NO_WAN_EGRESS(TM)

                    1 Reply Last reply Reply Quote 0
                    • K
                      kathampy
                      last edited by

                      @Derelict:

                      Sounds like you might have the same networks included in remote VPN networks or something.

                      I don't have any other networks beyond what's shown here. Can something happen because the 192.168.0.0/16 static route overlaps the /24 LAN interface? I can try adding individual static routes for each /24 VLAN instead.

                      How do I view the negate_networks table on my box?

                      1 Reply Last reply Reply Quote 0
                      • DerelictD
                        Derelict LAYER 8 Netgate
                        last edited by

                        That shouldn't matter but I wouldn't do it that way. My transport network would be outside the routed subnet.

                        You can Diagnostics > Command Prompt grep negate_networks /tmp/rules.debug That will also show everywhere it is used.

                        Or Diagnostics > Tables and look at negate_networks.

                        Chattanooga, Tennessee, USA
                        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                        1 Reply Last reply Reply Quote 0
                        • K
                          kathampy
                          last edited by

                          @Derelict:

                          That shouldn't matter but I wouldn't do it that way. My transport network would be outside the routed subnet.

                          What's the correct way to configure this on pfSense? Just create separate static routes for each? These are the VLANs on the switch that are relevant to the LAN interface:

                          Default Network
                          Subnet: 192.168.1.0/24
                          L3 Switch: 192.168.1.1

                          Guest Network
                          Subnet: 192.168.5.0/24
                          L3 Switch: 192.168.5.1

                          Firewall Network
                          Subnet: 192.168.2.0/24
                          L3 Switch: 192.168.2.1
                          pfSense: 192.168.2.2

                          1 Reply Last reply Reply Quote 0
                          • K
                            kathampy
                            last edited by

                            I unchecked "Disable Negate Rules" and tried this.

                            @Derelict:

                            You can Diagnostics > Command Prompt grep negate_networks /tmp/rules.debug That will also show everywhere it is used.

                            table <negate_networks>
                            pass  in  quick  on $LAN inet proto { tcp udp }  from $Default  to <negate_networks>  port 53 tracker 10000001 keep state  label "NEGATE_ROUTE: Negate policy routing for destination"
                            pass  in  quick  on $LAN inet proto udp  from $Default port $Steam_Source  to <negate_networks> tracker 10000002 keep state  label "NEGATE_ROUTE: Negate policy routing for destination"
                            pass  in  quick  on $LAN inet proto udp  from $Default  to <negate_networks>  port $Steam tracker 10000003 keep state  label "NEGATE_ROUTE: Negate policy routing for destination"</negate_networks></negate_networks></negate_networks></negate_networks>
                            

                            If I check "Disable Negate Rules", then it's empty.

                            @DerelictOr:

                            Diagnostics > Tables and look at negate_networks.

                            This is always empty.

                            1 Reply Last reply Reply Quote 0
                            • K
                              kathampy
                              last edited by

                              It looks like "Disable Negate Rules" should not have any effect in my case. I need to wait till it stops working again and debug it then.

                              1 Reply Last reply Reply Quote 0
                              • K
                                kathampy
                                last edited by

                                I've PMed you my rules.debug. "Disable Negate Rules" did not have any effect. The two gateway rules are still being bypassed.

                                1 Reply Last reply Reply Quote 0
                                • johnpozJ
                                  johnpoz LAYER 8 Global Moderator
                                  last edited by

                                  So I am a bit confused on your setup.  So you mention L3 switches on all your networks on pfsense.

                                  Are these switches all doing routing?  And those network 192.168.1/24 and 5.0/24 and 2.0/24 are transit networks??  Those seem quite large for transit networks, do you have hosts other than your routers on these networks?

                                  This network you call your default that is the wan of pfsense?  All your rules have default as the source?

                                  I would suggest you draw up your connectivity..  And keep in mind any connection between pfsense and a downstream router should be via a transit network.  Ie there should not be any hosts or devices on this network that connects pfsense to another router or you run into asymmetrical routing problems.

                                  Your only showing 1 set of rules.  Which is fine if this is on the transit interface to pfsense.. But your rules on that interface need to allow for the source networks unless your downstream routers are also natting.  And your outbound nat need to allow for the natting of these downstream networks as well.

                                  An intelligent man is sometimes forced to be drunk to spend time with his fools
                                  If you get confused: Listen to the Music Play
                                  Please don't Chat/PM me for help, unless mod related
                                  SG-4860 24.11 | Lab VMs 2.7.2, 24.11

                                  1 Reply Last reply Reply Quote 0
                                  • K
                                    kathampy
                                    last edited by

                                    I've attached a network diagram that explains the setup.

                                    The diagram is the logical view. In reality pfSense in running on Hyper-V. The two WAN cables are also connected to the switch on dedicated VLANs 3 & 4. VLAN 2, 3 & 4 are connected to the pfSense VM as individual network interfaces.

                                    1 Reply Last reply Reply Quote 0
                                    • johnpozJ
                                      johnpoz LAYER 8 Global Moderator
                                      last edited by

                                      yup that drawing makes it very clear.. That was not there before was it?

                                      So that looks right from my quick look at it..  But your rules are very specific with source ports and source network for example.  So your showing specific states that match that rule but going out your normal routing?  Your default gateway seems to be the ficus connection.

                                      An intelligent man is sometimes forced to be drunk to spend time with his fools
                                      If you get confused: Listen to the Music Play
                                      Please don't Chat/PM me for help, unless mod related
                                      SG-4860 24.11 | Lab VMs 2.7.2, 24.11

                                      1 Reply Last reply Reply Quote 0
                                      • K
                                        kathampy
                                        last edited by

                                        @johnpoz:

                                        yup that drawing makes it very clear.. That was not there before was it?

                                        So that looks right from my quick look at it..  But your rules are very specific with source ports and source network for example.  So your showing specific states that match that rule but going out your normal routing?  Your default gateway seems to be the ficus connection.

                                        Yes, I have traffic that matches Steam (UDP Destination Port 27000:27030 in this case) that goes out the default gateway. I can see these states under the Ficus interface in the Diagnostic > States viewer with matching destination ports. As soon as I disconnect the default gateway PPPoE, the traffic immediately starts flowing over the Airtel PPPoE connection within milliseconds. Now the States viewer shows the states under the Airtel interface and they stay there until the traffic stops and the states are closed.

                                        1 Reply Last reply Reply Quote 0
                                        • johnpozJ
                                          johnpoz LAYER 8 Global Moderator
                                          last edited by

                                          really makes no sense since if it doesn't trigger the rule that sends it out specific gateway without the allow rule at the end it should just fail.

                                          An intelligent man is sometimes forced to be drunk to spend time with his fools
                                          If you get confused: Listen to the Music Play
                                          Please don't Chat/PM me for help, unless mod related
                                          SG-4860 24.11 | Lab VMs 2.7.2, 24.11

                                          1 Reply Last reply Reply Quote 0
                                          • DerelictD
                                            Derelict LAYER 8 Netgate
                                            last edited by

                                            Yes, I have traffic that matches Steam (UDP Destination Port 27000:27030 in this case) that goes out the default gateway. I can see these states under the Ficus interface in the Diagnostic > States viewer with matching destination ports.

                                            I would like to see outputs of pfctl -vvsr and pfctl -vvss when the firewall is in this mode.

                                            That will show exactly which rule is passing the traffic in question. It would be especially helpful if you could clear all states, generate the traffic in question, then take these samples.

                                            I realize it might be kind of large.

                                            There's probably a simple explanation for what you're seeing. Just don't know what it is yet.

                                            Chattanooga, Tennessee, USA
                                            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                            Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.