Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    [SOLVED] Policy-Based Routing Not Consistently Going Out the Specified Gateway

    Scheduled Pinned Locked Moved OpenVPN
    42 Posts 4 Posters 9.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DerelictD
      Derelict LAYER 8 Netgate
      last edited by

      Doubtful. There is some other reason the traffic is not matching that policy routing rule - else it would be policy routed accordingly.

      Chattanooga, Tennessee, USA
      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
      Do Not Chat For Help! NO_WAN_EGRESS(TM)

      1 Reply Last reply Reply Quote 0
      • F
        Finger79
        last edited by

        @Derelict:

        Doubtful. There is some other reason the traffic is not matching that policy routing rule - else it would be policy routed accordingly.

        Then why does the rule match in the Firewall Logs?

        As you can see, the rule is very simple.  If the source is .103 IPv4, then policy route it through WANGW.

        This works perfectly after a fresh reboot of pfSense.  Then like I said, after some time or some event, it no longer goes out the WANGW and goes out the VPN.  Something is overriding the routing portion of the firewall rule.

        1 Reply Last reply Reply Quote 0
        • F
          Finger79
          last edited by

          Just did some more, all from Firefox on the tablet:

          Shows real WAN IP
          Google "What is my IP"
          iplocation.com
          whatismyip.net
          privateinternetaccess.com
          whatismyip.org
          ExpressVPN.com
          MXtoolbox.com
          ip-address.org
          iplocation.net
          findipinfo.com
          myipaddress.com

          Shows VPN IP
          TorGuard.net
          DuckDuckGo "What is my IP"
          whatismyipaddress.com
          BearsMyIP.com
          ipchicken.com
          ipaddress.pro

          1 Reply Last reply Reply Quote 0
          • luckman212L
            luckman212 LAYER 8
            last edited by

            @Finger79:

            Then why does the rule match in the Firewall Logs?

            As you can see, the rule is very simple.  If the source is .103 IPv4, then policy route it through WANGW.

            Do you have a kill switch rule below the policy route for .103 to block all traffic? You need this in case WANGW is down, because rules will be skipped if the GW is flapping, could explain your inconsistent results…

            1 Reply Last reply Reply Quote 0
            • DerelictD
              Derelict LAYER 8 Netgate
              last edited by

              When it stops working, run this:

              pfctl -vvsr | grep -A3 XX.XX.XX.103

              Here: I'll show you one of mine. I'm not afraid of leaking inside addresses:

              $ pfctl -vvsr | grep -A3 192.168.223.6

              @307(1493852191) pass in quick on igb1.223 route-to (ovpnc3 172.29.114.130) inet from 192.168.223.6 to <openvpn_lan:2>flags S/SA keep state label "USER_RULE: Route OpenVPN Addresses Through OpenVPN"
                [ Evaluations: 2386      Packets: 0        Bytes: 0          States: 0    ]
                [ Inserted: pid 21796 State Creations: 0    ]

              Anyway, that will show the EXACT rules in the active rule set that have anything to do with that address at that specific point in time.</openvpn_lan:2>

              Chattanooga, Tennessee, USA
              A comprehensive network diagram is worth 10,000 words and 15 conference calls.
              DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
              Do Not Chat For Help! NO_WAN_EGRESS(TM)

              1 Reply Last reply Reply Quote 0
              • F
                Finger79
                last edited by

                @124(10000001) pass in log quick on igb1 inet from 192.168.100.103 to <negate_net   ="" works:0="">flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for de                                                                                                                              stination"
                  [ Evaluations: 2572      Packets: 0        Bytes: 0          States: 0    ]
                  [ Inserted: pid 54887 State Creations: 0    ]
                @125(1505701172) pass in log quick on igb1 route-to (igb0 xxx.xxx.xxx.xxx public IP) inet from                                                                                                                                192.168.100.103 to any flags S/SA keep state label "USER_RULE: Tablet Out Naked WAN"
                  [ Evaluations: 865      Packets: 55591    Bytes: 19139183    States: 5    ]
                  [ Inserted: pid 54887 State Creations: 807  ]
                @126(1458032398) block return in log quick on igb1 inet from any to <pfb_africa_   ="" v4:6176="">label "USER_RULE: pfb_Africa"</pfb_africa_ ></negate_net >

                1 Reply Last reply Reply Quote 0
                • DerelictD
                  Derelict LAYER 8 Netgate
                  last edited by

                  Looks good to me unless negate_networks includes the wrong destinations, which is pretty unlikely.

                  Or if there is a rule that matches that source address that won't be shown there.

                  I'd be happy to look at /tmp/rules.debug if you want to PM it.

                  Chattanooga, Tennessee, USA
                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                  1 Reply Last reply Reply Quote 0
                  • F
                    Finger79
                    last edited by

                    Somewhat redacted and edited:

                    < /tmp/rules.debug removed >

                    1 Reply Last reply Reply Quote 0
                    • F
                      Finger79
                      last edited by

                      Wonder if this table should be empty:

                      Diagnostics -> Tables
                      negate_networks

                      No entries exist in this table.

                      1 Reply Last reply Reply Quote 0
                      • DerelictD
                        Derelict LAYER 8 Netgate
                        last edited by

                        pass  in  quick  on $LAN  $GWPIA_TX_CHI inet proto tcp  from any to $Facebook port 443  tag "NO_WAN_EGRESS" tracker 1422073736 flags S/SA keep state  label "USER_RULE: Allow Facebook"
                        pass  in  quick  on $LAN inet proto tcp  from any to $CloudFlare port $HTTP_HTTPS tracker 1422073738 flags S/SA keep state  label "USER_RULE: CloudFlare"
                        pass  in log  quick  on $LAN inet from 192.168.100.103  to <negate_networks>tracker 10000001 keep state  label "NEGATE_ROUTE: Negate policy routing for destination"
                        pass  in log  quick  on $LAN  $GWWANGW inet from 192.168.100.103 to any tracker 1505701172 keep state  label "USER_RULE: Asus Tablet Out Naked WAN"

                        Anything at $Facebook/443 or $CloudFlare/$HTTP_HTTPS will not match your source 192.168.100.103 rules. That is probably your problem.

                        Put the most specific rules at the top.</negate_networks>

                        Chattanooga, Tennessee, USA
                        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                        1 Reply Last reply Reply Quote 0
                        • F
                          Finger79
                          last edited by

                          It was that damned CloudFlare rule.
                          I re-ran the list of places that previously showed the VPN IP and they all reported the real WAN IP as expected.

                          I really hope this consistently fixes it.  I'll update the thread if it doesn't fix it after I've pulled some hair out.

                          (BTW, those Facebook IPs are straight from Facebook so only include their CIDR blocks and nobody else.  Back when that info was public.)

                          Shows VPN IP
                          TorGuard.net –> Shows real IP :)
                          DuckDuckGo "What is my IP" --> Shows real IP :)
                          whatismyipaddress.com --> Shows real IP :)
                          BearsMyIP.com --> Shows real IP :)
                          ipchicken.com --> Shows real IP :)
                          ipaddress.pro --> Shows real IP :)

                          Anecdotally, this also tells me just how many sites are CloudFlare customers (at least the free account).  Holy crap it's a lot.

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post
                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.