Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    Static routes...not as expected?

    Routing and Multi WAN
    3
    11
    138
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      mmiller7 last edited by

      I just noticed when my primary WAN is down, I can still access/ping stuff that is set "static route" thru the primary WAN.

      Is this normal?

      Static routes:
      2304c6f4-4f1d-4f0c-93fd-3b3c30f54a87-image.png

      When "WAN" (the primary) is down and it's failed over to WAN2, should clients be able to reach 1.0.0.1 even though I have a static route to force that thru the WAN interface?

      Derelict 1 Reply Last reply Reply Quote 0
      • Derelict
        Derelict LAYER 8 Netgate @mmiller7 last edited by

        @mmiller7 What does the actual routing table show? Diagnostics > Routes

        M 1 Reply Last reply Reply Quote 0
        • M
          mmiller7 @Derelict last edited by mmiller7

          @derelict

          Simulating a failure unplugging the coax, here's what the routes table shows. Annoyingly, I was not able to reproduce the problem by pulling the cable as it happened when the conneciton failed on its own.

          0c2f9702-bfa4-46d7-a273-a3b45c90cb03-image.png

          Derelict 1 Reply Last reply Reply Quote 0
          • Derelict
            Derelict LAYER 8 Netgate @mmiller7 last edited by

            @mmiller7 Are you 100% certain the route actually has the 24.153.32.1 destination when that WAN is up?

            What does executing this in Diagnostics > Command Prompt output?

            grep -C10 1.0.0.1 /cf/conf/config.xml

            While you're at it you might as well dump the full routing table:

            netstat -rnfinet

            M 2 Replies Last reply Reply Quote 0
            • viktor_g
              viktor_g Netgate last edited by

              I can confirm that
              Redmine issue created: https://redmine.pfsense.org/issues/11296

              1 Reply Last reply Reply Quote 0
              • M
                mmiller7 @Derelict last edited by

                In case I forgot to say my full pfSense version before...2.4.5-RELEASE-p1 (amd64) / built on Tue Jun 02 17:51:17 EDT 2020 / FreeBSD 11.3-STABLE

                @derelict Would you prefer the tests with the WAN1 up, attempting to simulate it down (pull a cable), or both? I'm assuming both may be helpful.

                24.153.32.1 was the upstream gateway my ISP pushed via DHCP when I took that screenshot. I learned some time ago the hard way that it can change (as can my IP address, albeit infrequent) which is why I tried to not hard code that as the monitor IP (and I want to allow the modem self-generated 192.168.100.0/24 lease so I can see its status when it is down and use that to help complain to my ISP) so letting it pick the gateway monitor IP dynamically did not work well.

                Other IPs in the screenshots:
                192.168.100.1 - WAN1 Arris modem's diagnostic/status page
                1.0.0.1 - WAN1 Monitor-IP - Cloudfire DNS (also WAN1 dedicated DNS)
                192.168.10.1 - my WAN2 backup cellular hotspot gateway

                Possibly relevant, I do have these checked in advanced settings, if that somehow affects this behavior:
                f5415da6-0fe1-4891-ba51-e571f76f3761-image.png

                From my config:

                [2.4.5-RELEASE][root@pfSense.apt]/root: grep -C10 1.0.0.1 /cf/conf/config.xml
                		<ssh>
                			<enable>enabled</enable>
                		</ssh>
                		<ipv6allow></ipv6allow>
                		<enableserial></enableserial>
                		<afterfilterchangeshellcmd></afterfilterchangeshellcmd>
                		<mds_disable>0</mds_disable>
                		<gw_down_kill_states></gw_down_kill_states>
                		<skip_rules_gw_down></skip_rules_gw_down>
                		<dnsserver>1.1.1.1</dnsserver>
                		<dnsserver>1.0.0.1</dnsserver>
                		<dnsserver>8.8.8.8</dnsserver>
                		<dnsserver>8.8.4.4</dnsserver>
                	</system>
                	<interfaces>
                		<wan>
                			<enable></enable>
                			<if>igb0</if>
                			<blockbogons></blockbogons>
                			<descr><![CDATA[WAN]]></descr>
                			<alias-address></alias-address>
                --
                			<network>192.168.100.1/32</network>
                			<gateway>WAN_DHCP</gateway>
                			<descr><![CDATA[Modem Diagnostics]]></descr>
                		</route>
                		<route>
                			<network>192.168.10.0/24</network>
                			<gateway>WAN2_DHCP</gateway>
                			<descr><![CDATA[Access MiFi Settings]]></descr>
                		</route>
                		<route>
                			<network>1.0.0.1/32</network>
                			<gateway>WAN_DHCP</gateway>
                			<descr><![CDATA[Monitoring IP and DNS for WAN]]></descr>
                		</route>
                		<route>
                			<network>8.8.4.4/32</network>
                			<gateway>WAN2_DHCP</gateway>
                			<descr><![CDATA[Monitoring IP and DNS for WAN2]]></descr>
                		</route>
                	</staticroutes>
                	<dhcpd>
                --
                				<username><![CDATA[admin@192.168.1.194 (Local Database)]]></username>
                			</updated>
                			<created>
                				<time>1607182884</time>
                				<username><![CDATA[admin@192.168.1.194 (Local Database)]]></username>
                			</created>
                			<disabled></disabled>
                		</rule>
                		<rule>
                			<id></id>
                			<tracker>0100000101</tracker>
                			<type>pass</type>
                			<interface>lan</interface>
                			<ipprotocol>inet</ipprotocol>
                			<tag></tag>
                			<tagged></tagged>
                			<max></max>
                			<max-src-nodes></max-src-nodes>
                			<max-src-conn></max-src-conn>
                			<max-src-states></max-src-states>
                			<statetimeout></statetimeout>
                --
                				<time>1607181413</time>
                				<username><![CDATA[admin@192.168.1.194 (Local Database)]]></username>
                			</updated>
                			<disabled></disabled>
                		</rule>
                		<rule>
                			<type>pass</type>
                			<ipprotocol>inet6</ipprotocol>
                			<descr><![CDATA[Default allow LAN IPv6 to any rule]]></descr>
                			<interface>lan</interface>
                			<tracker>0100000102</tracker>
                			<source>
                				<network>lan</network>
                			</source>
                			<destination>
                				<any></any>
                			</destination>
                		</rule>
                		<rule>
                			<descr><![CDATA[OpenVPN  wizard]]></descr>
                			<source>
                --
                			<losshigh>75</losshigh>
                		</gateway_item>
                		<gateway_item>
                			<interface>wan</interface>
                			<gateway>dynamic</gateway>
                			<name>WAN_DHCP</name>
                			<weight>1</weight>
                			<ipprotocol>inet</ipprotocol>
                			<time_period>15000</time_period>
                			<descr><![CDATA[Interface WAN_DHCP Gateway]]></descr>
                			<monitor>1.0.0.1</monitor>
                			<latencyhigh>1000</latencyhigh>
                			<losshigh>80</losshigh>
                		</gateway_item>
                		<gateway_item>
                			<interface>opt3</interface>
                			<gateway>dynamic</gateway>
                			<name>WAN2_DHCP6</name>
                			<weight>1</weight>
                			<ipprotocol>inet6</ipprotocol>
                			<descr><![CDATA[Interface WAN2_DHCP6 Gateway]]></descr>
                [2.4.5-RELEASE][root@pfSense.apt]/root: 
                

                I will collect the additional route information you requested.

                I presume my original expectation is correct, that clients should fail to reach 1.0.0.1 if WAN1 is down, given my static route; even without fancy LAN firewall rules explicitly blocking 1.0.0.1 destination on WAN2?

                1 Reply Last reply Reply Quote 0
                • M
                  mmiller7 @Derelict last edited by

                  @derelict said in Static routes...not as expected?:

                  @mmiller7 Are you 100% certain the route actually has the 24.153.32.1 destination when that WAN is up?

                  What does executing this in Diagnostics > Command Prompt output?

                  grep -C10 1.0.0.1 /cf/conf/config.xml

                  While you're at it you might as well dump the full routing table:

                  netstat -rnfinet

                  FYI I'm having a bit of difficulty posting the information, its telling me my post was flagged for spam by some monitoring bot...hopefully the retry worked.

                  Derelict 1 Reply Last reply Reply Quote 0
                  • Derelict
                    Derelict LAYER 8 Netgate @mmiller7 last edited by

                    @mmiller7 You don't have to do any of that and you are probably confusing something by doing so.

                    Setting 1.0.0.1 as the gateway monitoring address automatically creates the host route out that interface that you are trying to create manually.

                    Setting 1.0.0.1 as a DNS server with a gateway set (I can't see if you have a gateway set or not) does the same thing.

                    There is zero need to try to make a static route like that.

                    M 1 Reply Last reply Reply Quote 0
                    • M
                      mmiller7 last edited by mmiller7

                      Good news! I've reproduced my issue while capturing the requests routing info.

                      Routing table when WAN1 is operational:

                      [2.4.5-RELEASE][root@pfSense.apt]/root: netstat -rnfinet
                      Routing tables
                      
                      Internet:
                      Destination        Gateway            Flags     Netif Expire
                      default            24.153.32.1        UGS        igb0
                      1.0.0.1            24.153.32.1        UGHS       igb0
                      1.0.0.1/32         24.153.32.1        UGS        igb0
                      8.8.4.4            192.168.10.1       UGHS       igb1
                      8.8.4.4/32         192.168.10.1       UGS        igb1
                      10.0.8.0/24        10.0.8.2           UGS      ovpns1
                      10.0.8.1           link#13            UHS         lo0
                      10.0.8.2           link#13            UH       ovpns1
                      10.0.9.0/24        10.0.9.2           UGS      ovpns2
                      10.0.9.1           link#14            UHS         lo0
                      10.0.9.2           link#14            UH       ovpns2
                      24.153.32.0/24     link#1             U          igb0
                      24.153.32.24       link#1             UHS         lo0
                      127.0.0.1          link#6             UH          lo0
                      192.168.1.0/24     link#10            U         lagg1
                      192.168.1.1        link#10            UHS         lo0
                      192.168.2.0/24     link#11            U       lagg1.2
                      192.168.2.1        link#11            UHS         lo0
                      192.168.3.0/24     link#12            U       lagg1.3
                      192.168.3.1        link#12            UHS         lo0
                      192.168.10.0/24    192.168.10.1       UGS        igb1
                      192.168.10.23      link#2             UHS         lo0
                      192.168.100.1/32   24.153.32.1        UGS        igb0
                      [2.4.5-RELEASE][root@pfSense.apt]/root: 
                      
                      

                      INITIALLY Routing table when WAN1 is down (pull coax from modem simulate ISP failure):

                      [2.4.5-RELEASE][root@pfSense.apt]/root: netstat -rnfinet
                      Routing tables
                      
                      Internet:
                      Destination        Gateway            Flags     Netif Expire
                      default            192.168.10.1       UGS        igb1
                      1.0.0.1            24.153.32.1        UGHS       igb0
                      1.0.0.1/32         24.153.32.1        UGS        igb0
                      8.8.4.4            192.168.10.1       UGHS       igb1
                      8.8.4.4/32         192.168.10.1       UGS        igb1
                      10.0.8.0/24        10.0.8.2           UGS      ovpns1
                      10.0.8.1           link#13            UHS         lo0
                      10.0.8.2           link#13            UH       ovpns1
                      10.0.9.0/24        10.0.9.2           UGS      ovpns2
                      10.0.9.1           link#14            UHS         lo0
                      10.0.9.2           link#14            UH       ovpns2
                      24.153.32.0/24     link#1             U          igb0
                      24.153.32.24       link#1             UHS         lo0
                      127.0.0.1          link#6             UH          lo0
                      192.168.1.0/24     link#10            U         lagg1
                      192.168.1.1        link#10            UHS         lo0
                      192.168.2.0/24     link#11            U       lagg1.2
                      192.168.2.1        link#11            UHS         lo0
                      192.168.3.0/24     link#12            U       lagg1.3
                      192.168.3.1        link#12            UHS         lo0
                      192.168.10.0/24    192.168.10.1       UGS        igb1
                      192.168.10.23      link#2             UHS         lo0
                      192.168.100.1/32   24.153.32.1        UGS        igb0
                      

                      After ~10 minutes, Routing table when WAN1 is down (pull coax from modem simulate ISP failure):

                      [2.4.5-RELEASE][root@pfSense.apt]/root: netstat -rnfinet
                      Routing tables
                      
                      Internet:
                      Destination        Gateway            Flags     Netif Expire
                      default            192.168.10.1       UGS        igb1
                      1.0.0.1            192.168.100.1      UGHS       igb0
                      1.0.0.1/32         192.168.100.1      UGS        igb0
                      8.8.4.4            192.168.10.1       UGHS       igb1
                      8.8.4.4/32         192.168.10.1       UGS        igb1
                      10.0.8.0/24        10.0.8.2           UGS      ovpns1
                      10.0.8.1           link#13            UHS         lo0
                      10.0.8.2           link#13            UH       ovpns1
                      10.0.9.0/24        10.0.9.2           UGS      ovpns2
                      10.0.9.1           link#14            UHS         lo0
                      10.0.9.2           link#14            UH       ovpns2
                      127.0.0.1          link#6             UH          lo0
                      192.168.1.0/24     link#10            U         lagg1
                      192.168.1.1        link#10            UHS         lo0
                      192.168.2.0/24     link#11            U       lagg1.2
                      192.168.2.1        link#11            UHS         lo0
                      192.168.3.0/24     link#12            U       lagg1.3
                      192.168.3.1        link#12            UHS         lo0
                      192.168.10.0/24    192.168.10.1       UGS        igb1
                      192.168.10.23      link#2             UHS         lo0
                      192.168.100.0/24   link#1             U          igb0
                      192.168.100.1/32   192.168.100.1      UGS        igb0
                      192.168.100.11     link#1             UHS         lo0
                      

                      **** AT THIS POINT I CAN NOW PING 1.0.0.1 FROM MY CLIENS when only WAN2 is operational, althtough the behavior seems to "flap" accessible/inaccessible likely as the modem reboots ****

                      matthew@Inspiron13-7378 $ ping 1.0.0.1
                      PING 1.0.0.1 (1.0.0.1) 56(84) bytes of data.
                      64 bytes from 1.0.0.1: icmp_seq=1 ttl=52 time=37.1 ms
                      64 bytes from 1.0.0.1: icmp_seq=2 ttl=52 time=46.5 ms
                      64 bytes from 1.0.0.1: icmp_seq=3 ttl=52 time=37.4 ms
                      64 bytes from 1.0.0.1: icmp_seq=4 ttl=52 time=130 ms
                      64 bytes from 1.0.0.1: icmp_seq=5 ttl=52 time=354 ms
                      64 bytes from 1.0.0.1: icmp_seq=6 ttl=52 time=34.2 ms
                      64 bytes from 1.0.0.1: icmp_seq=7 ttl=52 time=40.1 ms
                      

                      Routing table when WAN1 is down no-carrier (pull Ethernet between modem/router):

                      [2.4.5-RELEASE][root@pfSense.apt]/root: netstat -rnfinet
                      Routing tables
                      
                      Internet:
                      Destination        Gateway            Flags     Netif Expire
                      default            192.168.10.1       UGS        igb1
                      8.8.4.4            192.168.10.1       UGHS       igb1
                      8.8.4.4/32         192.168.10.1       UGS        igb1
                      10.0.8.0/24        10.0.8.2           UGS      ovpns1
                      10.0.8.1           link#13            UHS         lo0
                      10.0.8.2           link#13            UH       ovpns1
                      10.0.9.0/24        10.0.9.2           UGS      ovpns2
                      10.0.9.1           link#14            UHS         lo0
                      10.0.9.2           link#14            UH       ovpns2
                      127.0.0.1          link#6             UH          lo0
                      192.168.1.0/24     link#10            U         lagg1
                      192.168.1.1        link#10            UHS         lo0
                      192.168.2.0/24     link#11            U       lagg1.2
                      192.168.2.1        link#11            UHS         lo0
                      192.168.3.0/24     link#12            U       lagg1.3
                      192.168.3.1        link#12            UHS         lo0
                      192.168.10.0/24    192.168.10.1       UGS        igb1
                      192.168.10.23      link#2             UHS         lo0
                      

                      **** AT THIS POINT I CAN NOW PING 1.0.0.1 FROM MY CLIENS when only WAN2 is operational, and it seems very stable reachable ****

                      If it helps, here is a visual from my home automation server (which I have PING a bunch of stuff to get a quick high level picture of my entire network status)
                      da587860-e1d3-48ad-a9db-8190296feb44-image.png

                      1 Reply Last reply Reply Quote 0
                      • M
                        mmiller7 @Derelict last edited by

                        @derelict said in Static routes...not as expected?:

                        @mmiller7 You don't have to do any of that and you are probably confusing something by doing so.

                        Setting 1.0.0.1 as the gateway monitoring address automatically creates the host route out that interface that you are trying to create manually.

                        Setting 1.0.0.1 as a DNS server with a gateway set (I can't see if you have a gateway set or not) does the same thing.

                        There is zero need to try to make a static route like that.

                        I'll try removing the static route entries and see if it changes anything in my test.

                        M 1 Reply Last reply Reply Quote 0
                        • M
                          mmiller7 @mmiller7 last edited by

                          @derelict I've deleted my manually-entered static rules (but left the monitor IP and DNS); no change.

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post

                          Products

                          • Platform Overview
                          • TNSR
                          • pfSense
                          • Appliances

                          Services

                          • Training
                          • Professional Services

                          Support

                          • Subscription Plans
                          • Contact Support
                          • Product Lifecycle
                          • Documentation

                          News

                          • Media Coverage
                          • Press
                          • Events

                          Resources

                          • Blog
                          • FAQ
                          • Find a Partner
                          • Resource Library
                          • Security Information

                          Company

                          • About Us
                          • Careers
                          • Partners
                          • Contact Us
                          • Legal
                          Our Mission

                          We provide leading-edge network security at a fair price - regardless of organizational size or network sophistication. We believe that an open-source security model offers disruptive pricing along with the agility required to quickly address emerging threats.

                          Subscribe to our Newsletter

                          Product information, software announcements, and special offers. See our newsletter archive to sign up for future newsletters and to read past announcements.

                          © 2021 Rubicon Communications, LLC | Privacy Policy