Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Static routes...not as expected?

    Scheduled Pinned Locked Moved Routing and Multi WAN
    14 Posts 4 Posters 1.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      mmiller7
      last edited by

      I just noticed when my primary WAN is down, I can still access/ping stuff that is set "static route" thru the primary WAN.

      Is this normal?

      Static routes:
      2304c6f4-4f1d-4f0c-93fd-3b3c30f54a87-image.png

      When "WAN" (the primary) is down and it's failed over to WAN2, should clients be able to reach 1.0.0.1 even though I have a static route to force that thru the WAN interface?

      DerelictD 1 Reply Last reply Reply Quote 0
      • DerelictD
        Derelict LAYER 8 Netgate @mmiller7
        last edited by

        @mmiller7 What does the actual routing table show? Diagnostics > Routes

        Chattanooga, Tennessee, USA
        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
        Do Not Chat For Help! NO_WAN_EGRESS(TM)

        M 1 Reply Last reply Reply Quote 0
        • M
          mmiller7 @Derelict
          last edited by mmiller7

          @derelict

          Simulating a failure unplugging the coax, here's what the routes table shows. Annoyingly, I was not able to reproduce the problem by pulling the cable as it happened when the conneciton failed on its own.

          0c2f9702-bfa4-46d7-a273-a3b45c90cb03-image.png

          DerelictD 1 Reply Last reply Reply Quote 0
          • DerelictD
            Derelict LAYER 8 Netgate @mmiller7
            last edited by

            @mmiller7 Are you 100% certain the route actually has the 24.153.32.1 destination when that WAN is up?

            What does executing this in Diagnostics > Command Prompt output?

            grep -C10 1.0.0.1 /cf/conf/config.xml

            While you're at it you might as well dump the full routing table:

            netstat -rnfinet

            Chattanooga, Tennessee, USA
            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
            Do Not Chat For Help! NO_WAN_EGRESS(TM)

            M 2 Replies Last reply Reply Quote 0
            • viktor_gV
              viktor_g Netgate
              last edited by

              I can confirm that
              Redmine issue created: https://redmine.pfsense.org/issues/11296

              1 Reply Last reply Reply Quote 0
              • M
                mmiller7 @Derelict
                last edited by

                In case I forgot to say my full pfSense version before...2.4.5-RELEASE-p1 (amd64) / built on Tue Jun 02 17:51:17 EDT 2020 / FreeBSD 11.3-STABLE

                @derelict Would you prefer the tests with the WAN1 up, attempting to simulate it down (pull a cable), or both? I'm assuming both may be helpful.

                24.153.32.1 was the upstream gateway my ISP pushed via DHCP when I took that screenshot. I learned some time ago the hard way that it can change (as can my IP address, albeit infrequent) which is why I tried to not hard code that as the monitor IP (and I want to allow the modem self-generated 192.168.100.0/24 lease so I can see its status when it is down and use that to help complain to my ISP) so letting it pick the gateway monitor IP dynamically did not work well.

                Other IPs in the screenshots:
                192.168.100.1 - WAN1 Arris modem's diagnostic/status page
                1.0.0.1 - WAN1 Monitor-IP - Cloudfire DNS (also WAN1 dedicated DNS)
                192.168.10.1 - my WAN2 backup cellular hotspot gateway

                Possibly relevant, I do have these checked in advanced settings, if that somehow affects this behavior:
                f5415da6-0fe1-4891-ba51-e571f76f3761-image.png

                From my config:

                [2.4.5-RELEASE][root@pfSense.apt]/root: grep -C10 1.0.0.1 /cf/conf/config.xml
                		<ssh>
                			<enable>enabled</enable>
                		</ssh>
                		<ipv6allow></ipv6allow>
                		<enableserial></enableserial>
                		<afterfilterchangeshellcmd></afterfilterchangeshellcmd>
                		<mds_disable>0</mds_disable>
                		<gw_down_kill_states></gw_down_kill_states>
                		<skip_rules_gw_down></skip_rules_gw_down>
                		<dnsserver>1.1.1.1</dnsserver>
                		<dnsserver>1.0.0.1</dnsserver>
                		<dnsserver>8.8.8.8</dnsserver>
                		<dnsserver>8.8.4.4</dnsserver>
                	</system>
                	<interfaces>
                		<wan>
                			<enable></enable>
                			<if>igb0</if>
                			<blockbogons></blockbogons>
                			<descr><![CDATA[WAN]]></descr>
                			<alias-address></alias-address>
                --
                			<network>192.168.100.1/32</network>
                			<gateway>WAN_DHCP</gateway>
                			<descr><![CDATA[Modem Diagnostics]]></descr>
                		</route>
                		<route>
                			<network>192.168.10.0/24</network>
                			<gateway>WAN2_DHCP</gateway>
                			<descr><![CDATA[Access MiFi Settings]]></descr>
                		</route>
                		<route>
                			<network>1.0.0.1/32</network>
                			<gateway>WAN_DHCP</gateway>
                			<descr><![CDATA[Monitoring IP and DNS for WAN]]></descr>
                		</route>
                		<route>
                			<network>8.8.4.4/32</network>
                			<gateway>WAN2_DHCP</gateway>
                			<descr><![CDATA[Monitoring IP and DNS for WAN2]]></descr>
                		</route>
                	</staticroutes>
                	<dhcpd>
                --
                				<username><![CDATA[admin@192.168.1.194 (Local Database)]]></username>
                			</updated>
                			<created>
                				<time>1607182884</time>
                				<username><![CDATA[admin@192.168.1.194 (Local Database)]]></username>
                			</created>
                			<disabled></disabled>
                		</rule>
                		<rule>
                			<id></id>
                			<tracker>0100000101</tracker>
                			<type>pass</type>
                			<interface>lan</interface>
                			<ipprotocol>inet</ipprotocol>
                			<tag></tag>
                			<tagged></tagged>
                			<max></max>
                			<max-src-nodes></max-src-nodes>
                			<max-src-conn></max-src-conn>
                			<max-src-states></max-src-states>
                			<statetimeout></statetimeout>
                --
                				<time>1607181413</time>
                				<username><![CDATA[admin@192.168.1.194 (Local Database)]]></username>
                			</updated>
                			<disabled></disabled>
                		</rule>
                		<rule>
                			<type>pass</type>
                			<ipprotocol>inet6</ipprotocol>
                			<descr><![CDATA[Default allow LAN IPv6 to any rule]]></descr>
                			<interface>lan</interface>
                			<tracker>0100000102</tracker>
                			<source>
                				<network>lan</network>
                			</source>
                			<destination>
                				<any></any>
                			</destination>
                		</rule>
                		<rule>
                			<descr><![CDATA[OpenVPN  wizard]]></descr>
                			<source>
                --
                			<losshigh>75</losshigh>
                		</gateway_item>
                		<gateway_item>
                			<interface>wan</interface>
                			<gateway>dynamic</gateway>
                			<name>WAN_DHCP</name>
                			<weight>1</weight>
                			<ipprotocol>inet</ipprotocol>
                			<time_period>15000</time_period>
                			<descr><![CDATA[Interface WAN_DHCP Gateway]]></descr>
                			<monitor>1.0.0.1</monitor>
                			<latencyhigh>1000</latencyhigh>
                			<losshigh>80</losshigh>
                		</gateway_item>
                		<gateway_item>
                			<interface>opt3</interface>
                			<gateway>dynamic</gateway>
                			<name>WAN2_DHCP6</name>
                			<weight>1</weight>
                			<ipprotocol>inet6</ipprotocol>
                			<descr><![CDATA[Interface WAN2_DHCP6 Gateway]]></descr>
                [2.4.5-RELEASE][root@pfSense.apt]/root: 
                

                I will collect the additional route information you requested.

                I presume my original expectation is correct, that clients should fail to reach 1.0.0.1 if WAN1 is down, given my static route; even without fancy LAN firewall rules explicitly blocking 1.0.0.1 destination on WAN2?

                1 Reply Last reply Reply Quote 0
                • M
                  mmiller7 @Derelict
                  last edited by

                  @derelict said in Static routes...not as expected?:

                  @mmiller7 Are you 100% certain the route actually has the 24.153.32.1 destination when that WAN is up?

                  What does executing this in Diagnostics > Command Prompt output?

                  grep -C10 1.0.0.1 /cf/conf/config.xml

                  While you're at it you might as well dump the full routing table:

                  netstat -rnfinet

                  FYI I'm having a bit of difficulty posting the information, its telling me my post was flagged for spam by some monitoring bot...hopefully the retry worked.

                  DerelictD 1 Reply Last reply Reply Quote 0
                  • DerelictD
                    Derelict LAYER 8 Netgate @mmiller7
                    last edited by

                    @mmiller7 You don't have to do any of that and you are probably confusing something by doing so.

                    Setting 1.0.0.1 as the gateway monitoring address automatically creates the host route out that interface that you are trying to create manually.

                    Setting 1.0.0.1 as a DNS server with a gateway set (I can't see if you have a gateway set or not) does the same thing.

                    There is zero need to try to make a static route like that.

                    Chattanooga, Tennessee, USA
                    A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                    DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                    Do Not Chat For Help! NO_WAN_EGRESS(TM)

                    M 1 Reply Last reply Reply Quote 0
                    • M
                      mmiller7
                      last edited by mmiller7

                      Good news! I've reproduced my issue while capturing the requests routing info.

                      Routing table when WAN1 is operational:

                      [2.4.5-RELEASE][root@pfSense.apt]/root: netstat -rnfinet
                      Routing tables
                      
                      Internet:
                      Destination        Gateway            Flags     Netif Expire
                      default            24.153.32.1        UGS        igb0
                      1.0.0.1            24.153.32.1        UGHS       igb0
                      1.0.0.1/32         24.153.32.1        UGS        igb0
                      8.8.4.4            192.168.10.1       UGHS       igb1
                      8.8.4.4/32         192.168.10.1       UGS        igb1
                      10.0.8.0/24        10.0.8.2           UGS      ovpns1
                      10.0.8.1           link#13            UHS         lo0
                      10.0.8.2           link#13            UH       ovpns1
                      10.0.9.0/24        10.0.9.2           UGS      ovpns2
                      10.0.9.1           link#14            UHS         lo0
                      10.0.9.2           link#14            UH       ovpns2
                      24.153.32.0/24     link#1             U          igb0
                      24.153.32.24       link#1             UHS         lo0
                      127.0.0.1          link#6             UH          lo0
                      192.168.1.0/24     link#10            U         lagg1
                      192.168.1.1        link#10            UHS         lo0
                      192.168.2.0/24     link#11            U       lagg1.2
                      192.168.2.1        link#11            UHS         lo0
                      192.168.3.0/24     link#12            U       lagg1.3
                      192.168.3.1        link#12            UHS         lo0
                      192.168.10.0/24    192.168.10.1       UGS        igb1
                      192.168.10.23      link#2             UHS         lo0
                      192.168.100.1/32   24.153.32.1        UGS        igb0
                      [2.4.5-RELEASE][root@pfSense.apt]/root: 
                      
                      

                      INITIALLY Routing table when WAN1 is down (pull coax from modem simulate ISP failure):

                      [2.4.5-RELEASE][root@pfSense.apt]/root: netstat -rnfinet
                      Routing tables
                      
                      Internet:
                      Destination        Gateway            Flags     Netif Expire
                      default            192.168.10.1       UGS        igb1
                      1.0.0.1            24.153.32.1        UGHS       igb0
                      1.0.0.1/32         24.153.32.1        UGS        igb0
                      8.8.4.4            192.168.10.1       UGHS       igb1
                      8.8.4.4/32         192.168.10.1       UGS        igb1
                      10.0.8.0/24        10.0.8.2           UGS      ovpns1
                      10.0.8.1           link#13            UHS         lo0
                      10.0.8.2           link#13            UH       ovpns1
                      10.0.9.0/24        10.0.9.2           UGS      ovpns2
                      10.0.9.1           link#14            UHS         lo0
                      10.0.9.2           link#14            UH       ovpns2
                      24.153.32.0/24     link#1             U          igb0
                      24.153.32.24       link#1             UHS         lo0
                      127.0.0.1          link#6             UH          lo0
                      192.168.1.0/24     link#10            U         lagg1
                      192.168.1.1        link#10            UHS         lo0
                      192.168.2.0/24     link#11            U       lagg1.2
                      192.168.2.1        link#11            UHS         lo0
                      192.168.3.0/24     link#12            U       lagg1.3
                      192.168.3.1        link#12            UHS         lo0
                      192.168.10.0/24    192.168.10.1       UGS        igb1
                      192.168.10.23      link#2             UHS         lo0
                      192.168.100.1/32   24.153.32.1        UGS        igb0
                      

                      After ~10 minutes, Routing table when WAN1 is down (pull coax from modem simulate ISP failure):

                      [2.4.5-RELEASE][root@pfSense.apt]/root: netstat -rnfinet
                      Routing tables
                      
                      Internet:
                      Destination        Gateway            Flags     Netif Expire
                      default            192.168.10.1       UGS        igb1
                      1.0.0.1            192.168.100.1      UGHS       igb0
                      1.0.0.1/32         192.168.100.1      UGS        igb0
                      8.8.4.4            192.168.10.1       UGHS       igb1
                      8.8.4.4/32         192.168.10.1       UGS        igb1
                      10.0.8.0/24        10.0.8.2           UGS      ovpns1
                      10.0.8.1           link#13            UHS         lo0
                      10.0.8.2           link#13            UH       ovpns1
                      10.0.9.0/24        10.0.9.2           UGS      ovpns2
                      10.0.9.1           link#14            UHS         lo0
                      10.0.9.2           link#14            UH       ovpns2
                      127.0.0.1          link#6             UH          lo0
                      192.168.1.0/24     link#10            U         lagg1
                      192.168.1.1        link#10            UHS         lo0
                      192.168.2.0/24     link#11            U       lagg1.2
                      192.168.2.1        link#11            UHS         lo0
                      192.168.3.0/24     link#12            U       lagg1.3
                      192.168.3.1        link#12            UHS         lo0
                      192.168.10.0/24    192.168.10.1       UGS        igb1
                      192.168.10.23      link#2             UHS         lo0
                      192.168.100.0/24   link#1             U          igb0
                      192.168.100.1/32   192.168.100.1      UGS        igb0
                      192.168.100.11     link#1             UHS         lo0
                      

                      **** AT THIS POINT I CAN NOW PING 1.0.0.1 FROM MY CLIENS when only WAN2 is operational, althtough the behavior seems to "flap" accessible/inaccessible likely as the modem reboots ****

                      matthew@Inspiron13-7378 $ ping 1.0.0.1
                      PING 1.0.0.1 (1.0.0.1) 56(84) bytes of data.
                      64 bytes from 1.0.0.1: icmp_seq=1 ttl=52 time=37.1 ms
                      64 bytes from 1.0.0.1: icmp_seq=2 ttl=52 time=46.5 ms
                      64 bytes from 1.0.0.1: icmp_seq=3 ttl=52 time=37.4 ms
                      64 bytes from 1.0.0.1: icmp_seq=4 ttl=52 time=130 ms
                      64 bytes from 1.0.0.1: icmp_seq=5 ttl=52 time=354 ms
                      64 bytes from 1.0.0.1: icmp_seq=6 ttl=52 time=34.2 ms
                      64 bytes from 1.0.0.1: icmp_seq=7 ttl=52 time=40.1 ms
                      

                      Routing table when WAN1 is down no-carrier (pull Ethernet between modem/router):

                      [2.4.5-RELEASE][root@pfSense.apt]/root: netstat -rnfinet
                      Routing tables
                      
                      Internet:
                      Destination        Gateway            Flags     Netif Expire
                      default            192.168.10.1       UGS        igb1
                      8.8.4.4            192.168.10.1       UGHS       igb1
                      8.8.4.4/32         192.168.10.1       UGS        igb1
                      10.0.8.0/24        10.0.8.2           UGS      ovpns1
                      10.0.8.1           link#13            UHS         lo0
                      10.0.8.2           link#13            UH       ovpns1
                      10.0.9.0/24        10.0.9.2           UGS      ovpns2
                      10.0.9.1           link#14            UHS         lo0
                      10.0.9.2           link#14            UH       ovpns2
                      127.0.0.1          link#6             UH          lo0
                      192.168.1.0/24     link#10            U         lagg1
                      192.168.1.1        link#10            UHS         lo0
                      192.168.2.0/24     link#11            U       lagg1.2
                      192.168.2.1        link#11            UHS         lo0
                      192.168.3.0/24     link#12            U       lagg1.3
                      192.168.3.1        link#12            UHS         lo0
                      192.168.10.0/24    192.168.10.1       UGS        igb1
                      192.168.10.23      link#2             UHS         lo0
                      

                      **** AT THIS POINT I CAN NOW PING 1.0.0.1 FROM MY CLIENS when only WAN2 is operational, and it seems very stable reachable ****

                      If it helps, here is a visual from my home automation server (which I have PING a bunch of stuff to get a quick high level picture of my entire network status)
                      da587860-e1d3-48ad-a9db-8190296feb44-image.png

                      1 Reply Last reply Reply Quote 0
                      • M
                        mmiller7 @Derelict
                        last edited by

                        @derelict said in Static routes...not as expected?:

                        @mmiller7 You don't have to do any of that and you are probably confusing something by doing so.

                        Setting 1.0.0.1 as the gateway monitoring address automatically creates the host route out that interface that you are trying to create manually.

                        Setting 1.0.0.1 as a DNS server with a gateway set (I can't see if you have a gateway set or not) does the same thing.

                        There is zero need to try to make a static route like that.

                        I'll try removing the static route entries and see if it changes anything in my test.

                        M 1 Reply Last reply Reply Quote 0
                        • M
                          mmiller7 @mmiller7
                          last edited by

                          @derelict I've deleted my manually-entered static rules (but left the monitor IP and DNS); no change.

                          P 1 Reply Last reply Reply Quote 0
                          • P
                            philippe richard @mmiller7
                            last edited by

                            @mmiller7 said in Static routes...not as expected?:

                            @derelict I've deleted my manually-entered static rules (but left the monitor IP and DNS); no change.

                            Hello, did you solve the problem because I have the same.

                            thank you

                            M 1 Reply Last reply Reply Quote 0
                            • M
                              mmiller7 @philippe richard
                              last edited by mmiller7

                              @philippe-richard Not solved, there is a bug report mentioned earlier in the thread on it that from what I gather is not yet rolled into a release (states targeting CE-Next, which I gather is an upcoming release not yet finished).

                              As a workaround, I added firewall rules to "pass" the outbound traffic that should go on my static route thru the gateway I want, followed by rules to "block" the outbound traffic matching that for every other way it might get out. This is a messy kluge but at least kinda works around it.

                              I very much hope that they implement a full fix in an upcoming release, I'd love to get rid of all my extra convoluted rules.

                              P 1 Reply Last reply Reply Quote 1
                              • P
                                philippe richard @mmiller7
                                last edited by

                                @mmiller7 Hello, thank you for your answer. I think I'll do like you while waiting for an update that will fix this problem. I wish you a good day.

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post
                                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.